Skip to content

Model Authentication

For Kubernetes jobs, EvalHub routes model traffic through a sidecar proxy running alongside the adapter. The sidecar handles credential resolution and upstream TLS; the adapter receives credential references instead of real model API keys or SA tokens. Local jobs can read real credentials from a file:/// auth directory.

Model connection URLs and API keys must not be placed in benchmark parameters. Submit the primary endpoint in model.url, reference the model-auth Secret with model.auth.secret_ref, and put additional model connections in that Secret as <role>_api-key / <role>_url pairs. Model identifiers and evaluation settings remain parameters. See migration and adapter availability.

Note: The diagram illustrates the api-key / *_api-key / *_url pattern. Other patterns exist — see the scenarios below for details.

All model traffic is routed through the EvalHub sidecar proxy running in the same pod as the adapter. You provide credentials in a Kubernetes secret — EvalHub ensures API keys and SA tokens are only visible to the sidecar; hf-token and ca_cert may be projected to the adapter as passthrough keys. The sidecar resolves and injects credentials into outbound requests on behalf of the adapter.

EvalHub recognizes the following secret keys: api-key, *_api-key, *_sa_token, and *_url. When model.auth.secret_ref is set and the user secret contains any of these keys, EvalHub generates an ephemeral internal secret where credentials become :ref placeholders (e.g. api-key → api-key:ref). A *_url becomes the sidecar address with the original URL path preserved: https://judge.example/api/v1 becomes http://localhost:8080/api/v1. The internal secret is created per evaluation job and removed when the Job resource is deleted (Kubernetes GC via owner reference). For ca_cert-only or hf-token-only secrets no internal secret is created — those keys are passthrough.

The adapter receives a projected volume combining the internal ref secret (when present) with passthrough keys (hf-token, ca_cert) from the user secret. The user-provided secret with real credential values is mounted only into the sidecar.

You submit the real model URL in model.url when creating a job. EvalHub rewrites its scheme and host to http://localhost:8080 in the adapter’s JobSpec.model.url, preserving the original path, query, and fragment. For example, https://my-model.example.com/v1 becomes http://localhost:8080/v1. Auxiliary URLs are read from the projected Secret files, not inserted into parameters.

Both containers use /var/run/secrets/model, but receive different volumes:

FileUser Secret / sidecar seesAdapter sees in Kubernetes
api-keyReal primary-model credentialapi-key:ref
judge_api-keyReal judge credentialjudge_api-key:ref
judge_urlhttps://judge.example/api/v1http://localhost:8080/api/v1
embedding_api-keyReal embedding credentialembedding_api-key:ref
embedding_urlhttps://embedding.example/v1http://localhost:8080/v1
hf-token, ca_certOriginal values, if suppliedOriginal values, passed through

The adapter’s model-auth-internal projected volume combines synthetic references and allowed passthrough files. The raw model-auth Secret volume is mounted only in the sidecar. Unrecognized Secret keys are not automatically exposed to the adapter. These files are not automatically converted into per-role environment variables; the adapter must read them using the SDK.

This means:

  • Real API keys and SA tokens are only ever seen by the sidecar, not the adapter
  • TLS to the model endpoint is handled by the sidecar using ca_cert from the secret
  • The adapter calls the rewritten sidecar URL, including its original API path, regardless of auth type

The sidecar and projected-volume layout above apply to the Kubernetes runtime. For direct local adapter execution, see Local auth directories.

For an end-to-end workflow covering judge, grader, and scorer model selection and credentials, see Using Judge and Scorer Models.

See also: ODH-ADR-EH-0004 — Eval Sidecar Credential Injection


Migration from inline connection parameters

Section titled “Migration from inline connection parameters”
  1. Move auxiliary model URLs and API keys from parameters into the matching <role>_url / <role>_api-key Secret entries. Use the exact hyphenated key suffix; judge_api_key is not a recognized Secret credential key.
  2. Keep the primary endpoint in model.url, reference the Secret with model.auth.secret_ref, and leave model IDs and generation settings in parameters. Use the role mapping to choose the prefix expected by the adapter.
  3. Remove auxiliary URL/key runtime environment variables. When a framework needs such variables, the adapter populates them from Secret resolution.
  4. Submit a new job after changing the Secret; an already-running sidecar caches its values at startup.

Inspect, DeepEval, FollowBench, ToolEval, Promptfoo, CLEAR, and ATIF ignore legacy connection parameters. Their presence alone does not fail a job, and they never override or fill missing Secret entries. The role fallback and validation rules still apply. Connection fields in nested framework model options are removed while generation settings and unrelated test data are retained.

Remove old credentials from submitted requests and fixture generators even though they are ignored: job metadata can still persist those inputs. Adapter references list their specific legacy fields and exceptions, including Ragas’s standalone local overrides.


Use resolve_model_credentials() — it reads api-key from the mounted secret and returns a ModelCredentials object. api_key is the ref token (e.g. api-key:ref), not the real secret value. If no secret is mounted (open model), api_key is None.

# Inside run_benchmark_job(self, config: JobSpec, ...)
from evalhub.adapter.auth import resolve_model_credentials
creds = resolve_model_credentials()
headers = {}
if creds.api_key:
headers["Authorization"] = f"Bearer {creds.api_key}"
resp = requests.post(
f"{str(config.model.url).rstrip('/')}/chat/completions",
headers=headers,
json={...},
)

This example uses an OpenAI-compatible API base URL ending in /v1, without a query or fragment. Append /chat/completions after removing any trailing slash. The sidecar resolves api-key:ref to the real API key and forwards the request. If api_key is None, an HTTP client can omit Authorization. The sidecar injects its pod SA token for requests without a credential reference. OpenAI SDK clients require a non-empty api_key; local can satisfy that client requirement when no Secret key exists, but it does not select an auxiliary model route.

Use read_model_auth_key(key_name) to read any key from the secret mount by name. Returns None if the key is absent or the secret is not mounted.

from evalhub.adapter.auth import read_model_auth_key
# KFP SA token (Scenario 3)
kfp_token = read_model_auth_key("kfp_sa_token") # "kfp_sa_token:ref" or None
kfp_url = read_model_auth_key("kfp_url") # sidecar address or None
# Multi-model (Scenario 4)
judge_token = read_model_auth_key("judge_api-key") # "judge_api-key:ref" or None
judge_url = read_model_auth_key("judge_url") # sidecar address or None
# HuggingFace token (projected directly — real value, not a ref)
hf_token = read_model_auth_key("hf-token")

Handle None explicitly — it means the key was not in the secret or no secret was mounted:

if kfp_token is None:
raise RuntimeError("kfp_sa_token not found in secret mount — check secret_ref")

OpenAI-compatible target, judge and embedding clients

Section titled “OpenAI-compatible target, judge and embedding clients”

Configure each client with the URL and credential returned by the SDK. The same code reads synthetic references in Kubernetes and real files in local mode:

from openai import OpenAI
from evalhub.adapter.auth import read_model_auth_key
def model_client(config, role=None):
key = read_model_auth_key(f"{role}_api-key") if role else None
url = read_model_auth_key(f"{role}_url") if role else None
if url and not key:
raise ValueError(f"{role}_url in model.auth.secret_ref requires {role}_api-key")
return OpenAI(
base_url=(url or config.model.url).rstrip("/"),
api_key=key or read_model_auth_key("api-key") or "local",
)
target = model_client(config)
judge = model_client(config, "judge")
embedding = model_client(config, "embedding")
judgment = judge.chat.completions.create(
model=config.parameters["judge_model"],
messages=[{"role": "user", "content": "Score the supplied answer."}],
)
vectors = embedding.embeddings.create(
model=config.parameters["embedding_model"],
input=["The answer to evaluate"],
)

Supply OpenAI-compatible base URLs with the correct API path, such as /v1 or /api/v1; the SDK appends chat/completions or embeddings. Do not append a second /v1, reconstruct the upstream URL, or change a returned reference token. The SDK sends the supplied key as Authorization: Bearer <key>.

Two role URLs can both read as http://localhost:8080/v1. The Bearer reference selects the upstream and its credential, so judge_api-key:ref and embedding_api-key:ref must remain distinct even when their projected URLs match.

Adding a new <role>_api-key / <role>_url pair does not require a sidecar change for this protocol. The adapter still needs to read that role’s files and configure the appropriate client; the sidecar does not infer roles from model names.

Construct auxiliary connections only from the model-auth Secret. Do not read connection URLs or API keys from config.parameters or auxiliary runtime env variables. If the framework uses environment variables, set them in a subprocess environment from the resolved Secret values, independently for each role.

Framework passthrough settings can override a correctly configured client. Use the Inspect adapter’s without_connection_options filtering rule: recursively copy dictionaries and lists, removing keys that are api_key, api-key, api_base, base_url, or url, or end in _api_key, _api-key, _base_url, or _url. Apply this rule before forwarding model-option dictionaries, and extend it for any additional connection fields accepted by your framework. Retain generation options such as temperature and preserve unrelated test data. Ignore legacy connection fields rather than failing solely because they are present. Secret validation, including a role URL without its matching key, still applies.

The model proxy handles OpenAI-compatible Bearer-authenticated paths including /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, and /v1/responses; the upstream must implement the requested API. Adding a role does not need a new path registration. This does not add native Anthropic x-api-key, Azure api-key, or other provider-specific authentication. Use an OpenAI-compatible gateway when a model’s native client needs an unsupported authentication scheme. Inspect’s pinned Ollama client uses OpenAI-compatible /v1 requests with Bearer auth, so it can carry the reference; do not infer Ollama from the upstream port after EvalHub has rewritten the URL to the sidecar.

  • Do not expect a real API key from the mount. In Kubernetes jobs, api-key contains api-key:ref — a placeholder. The sidecar resolves the real value at request time.
  • Use the role’s reference for role routing. A dummy SDK key or an absent Authorization header cannot select a separate judge/embedding upstream; the sidecar instead uses the primary model route and its SA-token behavior.
  • Use config.model.url for the target and read_model_auth_key("<role>_url") for other roles. Both are rewritten to the sidecar inside a Kubernetes job. Do not recover upstream URLs from parameters.

A secret (model.auth.secret_ref) is required if:

  • The model requires an API key (api-key)
  • The endpoint uses a custom CA certificate (ca_cert)
  • You need a HuggingFace token for gated datasets (hf-token)
  • You need to call an in-cluster SA-token-authenticated service like KFP (_sa_token / _url)

If none of the above apply, omit model.auth. The sidecar still proxies all model traffic. For SA token-authenticated, RBAC-protected models (Scenario 2), the sidecar injects the pod SA token when the adapter sends no Authorization header.


Scenario 1: API key (and optional CA certificate)

Section titled “Scenario 1: API key (and optional CA certificate)”

Use this when the model requires an API key, for example a secured vLLM endpoint.

KeyDescription
api-keyAPI key — sidecar injects as Authorization: Bearer <api-key>
ca_certPEM CA certificate — sidecar uses for upstream TLS to the model endpoint
hf-tokenHuggingFace token — projected directly to the adapter (HF Hub calls bypass the sidecar)

At least one of api-key or ca_cert must be present. A secret containing only hf-token is not enough — the HF token is still available to the adapter, but the sidecar will not inject an API key.

apiVersion: v1
kind: Secret
metadata:
name: vllm-api-key
namespace: team-a
type: Opaque
stringData:
api-key: "your-api-key-here"
ca_cert: |
-----BEGIN CERTIFICATE-----
... your CA certificate PEM ...
-----END CERTIFICATE-----

Or using kubectl:

Terminal window
kubectl create secret generic vllm-api-key -n team-a \
--from-literal=api-key="your-api-key-here" \
--from-file=ca_cert=/path/to/ca.pem
  1. Reference the secret in the job

    Set model.auth.secret_ref to the secret name when submitting the evaluation job.

    Terminal window
    curl -k -X POST "$HOST/api/v1/evaluations/jobs" \
    -H "Authorization: Bearer $token" \
    -H "Content-Type: application/json" \
    -H "X-Tenant: team-a" \
    -d '{
    "name": "model api key/cert test",
    "model": {
    "url": "https://vllm-route-prabhu.apps.rosa.prabhu-comhub.xqmp.p3.openshiftapps.com/v1",
    "name": "gpt2",
    "auth": {
    "secret_ref": "vllm-api-key"
    }
    },
    "benchmarks": [
    {
    "id": "arc_easy",
    "provider_id": "lm_evaluation_harness",
    "parameters": {
    "limit": 5,
    "num_examples": 10,
    "tokenizer": "google/flan-t5-small"
    }
    }
    ]
    }'

    Adapters that use resolve_model_credentials() get api-key:ref automatically — no extra code needed for single-model API key auth.


Scenario 2: SA token-authenticated, RBAC-protected model (KServe / kube-rbac-proxy)

Section titled “Scenario 2: SA token-authenticated, RBAC-protected model (KServe / kube-rbac-proxy)”

Use this when the model endpoint is protected by Kubernetes RBAC (e.g. a KServe service behind kube-rbac-proxy).

The sidecar automatically injects the pod’s ServiceAccount token as Authorization: Bearer <token> when the adapter sends no Authorization header. The adapter does not have direct access to the SA token — the sidecar handles injection transparently.

If the endpoint uses a custom CA for TLS only, create a secret with just ca_cert and reference it with model.auth.secret_ref.

  1. Grant the ServiceAccount access

    The model namespace typically already has a Role (e.g. gpt2-view-role) from model deployment. Create a RoleBinding that binds the EvalHub job ServiceAccount to that role.

    Terminal window
    oc create rolebinding gpt2-view-evalhub-prabhu-job \
    --role=gpt2-view-role \
    --serviceaccount=team-a:evalhub-prabhu-job \
    -n team-a

    Or apply a RoleBinding YAML:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
    name: gpt2-view-evalhub-prabhu-job
    namespace: team-a
    subjects:
    - kind: ServiceAccount
    name: evalhub-prabhu-job
    namespace: team-a
    roleRef:
    kind: Role
    name: gpt2-view-role
    apiGroup: rbac.authorization.k8s.io

    Verify access:

    Terminal window
    kubectl auth can-i get inferenceservices.serving.kserve.io/gpt2 \
    -n team-a \
    --as=system:serviceaccount:team-a:evalhub-prabhu-job
  2. Submit the job

    Omit model.auth unless the endpoint uses a custom CA certificate.

    Terminal window
    curl -k -X POST "$HOST/api/v1/evaluations/jobs" \
    -H "Authorization: Bearer $token" \
    -H "Content-Type: application/json" \
    -H "X-Tenant: team-a" \
    -d '{
    "name": "SA token model test",
    "model": {
    "url": "https://gpt2-team-a.apps.rosa.prabhu-comhub.xqmp.p3.openshiftapps.com/v1",
    "name": "gpt2"
    },
    "benchmarks": [
    {
    "id": "arc_easy",
    "provider_id": "lm_evaluation_harness",
    "parameters": {
    "limit": 5,
    "num_examples": 10,
    "tokenizer": "google/flan-t5-small"
    }
    }
    ]
    }'

Scenario 3: In-cluster SA-token-authenticated services (KFP and others)

Section titled “Scenario 3: In-cluster SA-token-authenticated services (KFP and others)”

Use this when the adapter needs to call an in-cluster service (such as Kubeflow Pipelines) that requires the pod SA token. Use the _sa_token / _url key convention — the sidecar injects the SA token and routes to the real service URL.

KeyValueEffect
<prefix>_sa_token"" (empty)Sidecar injects pod SA token
<prefix>_sa_token<jwt>Sidecar forwards value as-is
<prefix>_urlReal service URLSidecar routes to this upstream

The prefix must match between the _sa_token and _url keys (e.g. kfp_sa_token pairs with kfp_url).

Terminal window
kubectl create secret generic kfp-creds \
--from-literal=kfp_sa_token="" \
--from-literal=kfp_url="http://ml-pipeline.kubeflow.svc.cluster.local:8888" \
-n team-a
  1. Reference the secret in the job

    Terminal window
    curl -k -X POST "$HOST/api/v1/evaluations/jobs" \
    -H "Authorization: Bearer $token" \
    -H "Content-Type: application/json" \
    -H "X-Tenant: team-a" \
    -d '{
    "name": "kfp sa token test",
    "model": {
    "url": "https://your-model-endpoint/v1",
    "name": "gpt2",
    "auth": {
    "secret_ref": "kfp-creds"
    }
    },
    "benchmarks": [
    {
    "id": "arc_easy",
    "provider_id": "lm_evaluation_harness",
    "parameters": {
    "limit": 5,
    "num_examples": 10,
    "tokenizer": "google/flan-t5-small"
    }
    }
    ]
    }'
  2. Call KFP from your adapter

    The adapter reads ref tokens from the secret mount and calls the sidecar. The sidecar resolves kfp_sa_token:ref → injects pod SA token → forwards to kfp_url.

    from evalhub.adapter.auth import read_model_auth_key
    kfp_token = read_model_auth_key("kfp_sa_token") # returns "kfp_sa_token:ref"
    kfp_url = read_model_auth_key("kfp_url") # returns sidecar address
    resp = requests.get(
    f"{kfp_url}/apis/v1beta1/runs",
    headers={"Authorization": f"Bearer {kfp_token}"},
    )

Use one Secret for the primary model and any judge, embedding, auditor, or other model clients. Create it in the tenant namespace where the evaluation Job runs. The values below are placeholders:

apiVersion: v1
kind: Secret
metadata:
name: multi-model-creds
namespace: team-a
type: Opaque
stringData:
api-key: "<target-credential>"
judge_api-key: "<judge-credential>"
judge_url: https://judge.example/v1
embedding_api-key: "<embedding-credential>"
embedding_url: https://embedding.example/v1

Submit only the Secret reference, primary endpoint and model identifiers:

{
"name": "ragas-with-separate-models",
"model": {
"url": "https://target.example/v1",
"name": "target-model",
"auth": {"secret_ref": "multi-model-creds"}
},
"benchmarks": [{
"provider_id": "ragas",
"id": "ragas_rag_default",
"parameters": {
"judge_model": "judge-model",
"embedding_model": "embedding-model"
}
}]
}

The adapter must use the role files as shown in the OpenAI-compatible client example. For built-in adapters, see the role-to-parameter table.

Role entriesAdapter behavior
Neither key nor URLReuse model.url and the primary Secret’s api-key
Key onlyReuse model.url with that role’s credential
Key and URLUse the role’s projected URL and credential reference
URL onlyReject: a separate route requires the matching role key

The suffix is exactly _api-key, including the hyphen. A plain judge_api_key Secret entry is unrecognized and is not projected to the adapter. If roles share a real credential but use different upstream URLs, copy that credential into each role’s key; projected URL equality is not a reason to reuse another role’s reference. Real role credentials must be non-empty.

For an additional critic client, create critic_api-key / critic_url and implement the corresponding SDK reads in the adapter. The sidecar already understands arbitrary role prefixes for OpenAI-compatible Bearer requests; no role registration or sidecar code change is needed. New protocols with different authentication headers require separate support.

For direct local adapter execution, set EVALHUB_MODE=local and model.auth.secret_ref to an absolute file:/// directory URL. Create files with the same names (api-key, judge_api-key, judge_url, and so on). These files contain the real credentials and real upstream URLs; Kubernetes-style :ref values require a sidecar and must not be written into a direct local auth directory. The SDK resolves this directory from EVALHUB_JOB_SPEC_PATH.

ATIF additionally requires a regular auth directory with no symlink components and bounded credential-file reads. See ATIF configuration. When the local runtime itself is configured with a sidecar, follow its local-mode routing configuration rather than assuming the Kubernetes projected-volume layout.

Create or update the Secret before starting the job. The sidecar caches values at startup; changes require a new evaluation job.


If your model endpoint is fully public and requires no API key, CA cert, or SA token, omit model.auth entirely. The sidecar still proxies all model traffic. No credential secret is mounted on the adapter.


If the evaluation job fails with authentication or TLS errors, check both the adapter and sidecar container logs:

Terminal window
# Adapter logs
kubectl logs -n <tenant-namespace> job/<job-name> -c adapter
# Sidecar logs
kubectl logs -n <tenant-namespace> job/<job-name> -c sidecar

Typical failures include:

  • Missing CA certificate → TLS error on sidecar upstream connection
  • Missing or invalid API key → 401 Unauthorized from the model endpoint
  • An api-key key present in the secret but with an empty string value → 400 Bad Request from the sidecar
  • RBAC misconfiguration (Scenario 2) → 403 Forbidden from the upstream model endpoint; check the RoleBinding and SA permissions
  • _url key used as a ref token → 400 Bad Request from the sidecar (only api-key, *_api-key, *_sa_token are valid ref keys)

ScenarioSecret keysmodel.authHow the sidecar handles the request
API keyapi-key, optionally ca_cert, hf-tokensecret_ref requiredSidecar resolves :ref token → real key
SA token-authenticated, RBAC-protected modelNone, or ca_cert onlyOmit unless TLSSidecar injects pod SA token when no Authorization from adapter
KFP / in-cluster service<prefix>_sa_token, <prefix>_urlsecret_ref requiredSidecar injects pod SA token via _sa_token:ref
Multi-model<prefix>_api-key, <prefix>_url per modelsecret_ref requiredAdapter sends per-prefix ref; sidecar resolves and routes to matching URL
Open modelNoneOmitSidecar proxies; no credentials needed