Model Authentication

EvalHub routes all model traffic through a sidecar proxy running alongside the adapter. The sidecar handles credential resolution and upstream TLS — the adapter never sees real API keys or SA tokens. A secret is required only when the model or an in-cluster service needs authentication.

How It Works

Note: The diagram illustrates the api-key / *_api-key / *_url pattern. Other patterns exist — see the scenarios below for details.

All model traffic is routed through the EvalHub sidecar proxy running in the same pod as the adapter. You provide credentials in a Kubernetes secret — EvalHub ensures API keys and SA tokens are only visible to the sidecar; hf-token and ca_cert may be projected to the adapter as passthrough keys. The sidecar resolves and injects credentials into outbound requests on behalf of the adapter.

EvalHub recognizes the following secret keys: api-key, *_api-key, *_sa_token, and *_url. When model.auth.secret_ref is set and the user secret contains any of these keys, EvalHub generates an ephemeral internal secret where values are replaced with :ref placeholders (e.g. api-key → api-key:ref) and *_url values are replaced with http://localhost:8080 (the sidecar address). This internal secret is created per evaluation job and removed when the Job resource is deleted (Kubernetes GC via owner reference). For ca_cert-only or hf-token-only secrets no internal secret is created — those keys are passthrough.

The adapter receives a projected volume combining the internal ref secret (when present) with passthrough keys (hf-token, ca_cert) from the user secret. The user-provided secret with real credential values is mounted only into the sidecar.

You submit the real model URL in model.url when creating a job. EvalHub rewrites the in-pod configuration so the adapter always calls http://localhost:8080 (the sidecar); the sidecar forwards to the real upstream URL.

This means:

• Real API keys and SA tokens are only ever seen by the sidecar, not the adapter
• TLS to the model endpoint is handled by the sidecar using ca_cert from the secret
• The adapter always calls http://localhost:8080, regardless of auth type

This guide applies to evaluation jobs created by the Kubernetes runtime. Local runtime behavior may differ.

See also: ODH-ADR-EH-0004 — Eval Sidecar Credential Injection

---

Writing an adapter

Common case: single model API key

Use resolve_model_credentials() — it reads api-key from the mounted secret and returns a ModelCredentials object. api_key is the ref token (e.g. api-key:ref), not the real secret value. If no secret is mounted (open model), api_key is None.

# Inside run_benchmark_job(self, config: JobSpec, ...)
from evalhub.adapter.auth import resolve_model_credentials

creds = resolve_model_credentials()

headers = {}
if creds.api_key:
    headers["Authorization"] = f"Bearer {creds.api_key}"

resp = requests.post(
    f"{config.model.url}/v1/chat/completions",
    headers=headers,
    json={...},
)

The sidecar intercepts the request, resolves api-key:ref → real API key, and forwards to the model endpoint. If api_key is None (SA token-authenticated, RBAC-protected or open model), omit the Authorization header entirely — do not send a placeholder. The sidecar injects the pod SA token when no Authorization header is present or when an empty Bearer value is sent.

Custom keys (KFP, multi-model, hf-token)

Use read_model_auth_key(key_name) to read any key from the secret mount by name. Returns None if the key is absent or the secret is not mounted.

from evalhub.adapter.auth import read_model_auth_key

# KFP SA token (Scenario 3)
kfp_token = read_model_auth_key("kfp_sa_token")  # "kfp_sa_token:ref" or None
kfp_url   = read_model_auth_key("kfp_url")        # sidecar address or None

# Multi-model (Scenario 4)
judge_token = read_model_auth_key("judge_api-key")  # "judge_api-key:ref" or None
judge_url   = read_model_auth_key("judge_url")       # sidecar address or None

# HuggingFace token (projected directly — real value, not a ref)
hf_token = read_model_auth_key("hf-token")

Handle None explicitly — it means the key was not in the secret or no secret was mounted:

if kfp_token is None:
    raise RuntimeError("kfp_sa_token not found in secret mount — check secret_ref")

Common mistakes

• Do not expect a real API key from the mount. In Kubernetes jobs, api-key contains api-key:ref — a placeholder. The sidecar resolves the real value at request time.
• Do not send a placeholder Authorization header for SA token-authenticated, RBAC-protected or open models. Omit the header (or send an empty Bearer) so the sidecar can inject the pod SA token. A non-empty, non-ref Authorization value is forwarded as-is and will bypass SA injection.
• Always use config.model.url from JobSpec; never hardcode the submitted URL. Although you submit the real endpoint URL when creating a job, config.model.url is automatically rewritten to http://localhost:8080 (the sidecar) inside the pod.

---

When to use a secret

A secret (model.auth.secret_ref) is required if:

• The model requires an API key (api-key)
• The endpoint uses a custom CA certificate (ca_cert)
• You need a HuggingFace token for gated datasets (hf-token)
• You need to call an in-cluster SA-token-authenticated service like KFP (_sa_token / _url)

If none of the above apply, omit model.auth. The sidecar still proxies all model traffic. For SA token-authenticated, RBAC-protected models (Scenario 2), the sidecar injects the pod SA token when the adapter sends no Authorization header.

---

Scenario 1: API key (and optional CA certificate)

Use this when the model requires an API key, for example a secured vLLM endpoint.

Secret structure

Key	Description
api-key	API key — sidecar injects as Authorization: Bearer <api-key>
ca_cert	PEM CA certificate — sidecar uses for upstream TLS to the model endpoint
hf-token	HuggingFace token — projected directly to the adapter (HF Hub calls bypass the sidecar)

At least one of api-key or ca_cert must be present. A secret containing only hf-token is not enough — the HF token is still available to the adapter, but the sidecar will not inject an API key.

apiVersion: v1
kind: Secret
metadata:
  name: vllm-api-key
  namespace: team-a
type: Opaque
stringData:
  api-key: "your-api-key-here"
  ca_cert: |
    -----BEGIN CERTIFICATE-----
    ... your CA certificate PEM ...
    -----END CERTIFICATE-----

Or using kubectl:

kubectl create secret generic vllm-api-key -n team-a \
  --from-literal=api-key="your-api-key-here" \
  --from-file=ca_cert=/path/to/ca.pem

1. Reference the secret in the job

   Set model.auth.secret_ref to the secret name when submitting the evaluation job.

   curl

   curl -k -X POST "$HOST/api/v1/evaluations/jobs" \
     -H "Authorization: Bearer $token" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: team-a" \
     -d '{
     "name": "model api key/cert test",
     "model": {
       "url": "https://vllm-route-prabhu.apps.rosa.prabhu-comhub.xqmp.p3.openshiftapps.com/v1",
       "name": "gpt2",
       "auth": {
         "secret_ref": "vllm-api-key"
       }
     },
     "benchmarks": [
       {
         "id": "arc_easy",
         "provider_id": "lm_evaluation_harness",
         "parameters": {
           "limit": 5,
           "num_examples": 10,
           "tokenizer": "google/flan-t5-small"
         }
       }
     ]
   }'

   Python SDK

   import os
   from evalhub import SyncEvalHubClient
   from evalhub.models.api import ModelConfig, BenchmarkConfig, JobSubmissionRequest
   
   client = SyncEvalHubClient(base_url=os.environ.get("HOST", "http://localhost:8080"))
   
   job = client.jobs.submit(
       JobSubmissionRequest(
           name="model api key/cert test",
           model=ModelConfig(
               url="https://vllm-route-prabhu.apps.rosa.prabhu-comhub.xqmp.p3.openshiftapps.com/v1",
               name="gpt2",
               auth={"secret_ref": "vllm-api-key"},
           ),
           benchmarks=[
               BenchmarkConfig(
                   id="arc_easy",
                   provider_id="lm_evaluation_harness",
                   parameters={"limit": 5, "num_examples": 10, "tokenizer": "google/flan-t5-small"},
               )
           ],
       ),
       headers={"X-Tenant": "team-a"},
   )

   Adapters that use resolve_model_credentials() get api-key:ref automatically — no extra code needed for single-model API key auth.

---

Scenario 2: SA token-authenticated, RBAC-protected model (KServe / kube-rbac-proxy)

Use this when the model endpoint is protected by Kubernetes RBAC (e.g. a KServe service behind kube-rbac-proxy).

The sidecar automatically injects the pod’s ServiceAccount token as Authorization: Bearer <token> when the adapter sends no Authorization header. The adapter does not have direct access to the SA token — the sidecar handles injection transparently.

If the endpoint uses a custom CA for TLS only, create a secret with just ca_cert and reference it with model.auth.secret_ref.

1. Grant the ServiceAccount access

   The model namespace typically already has a Role (e.g. gpt2-view-role) from model deployment. Create a RoleBinding that binds the EvalHub job ServiceAccount to that role.

   oc create rolebinding gpt2-view-evalhub-prabhu-job \
     --role=gpt2-view-role \
     --serviceaccount=team-a:evalhub-prabhu-job \
     -n team-a

   Or apply a RoleBinding YAML:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: gpt2-view-evalhub-prabhu-job
     namespace: team-a
   subjects:
     - kind: ServiceAccount
       name: evalhub-prabhu-job
       namespace: team-a
   roleRef:
     kind: Role
     name: gpt2-view-role
     apiGroup: rbac.authorization.k8s.io

   Verify access:

   kubectl auth can-i get inferenceservices.serving.kserve.io/gpt2 \
     -n team-a \
     --as=system:serviceaccount:team-a:evalhub-prabhu-job

2. Submit the job

   Omit model.auth unless the endpoint uses a custom CA certificate.

   curl

   curl -k -X POST "$HOST/api/v1/evaluations/jobs" \
     -H "Authorization: Bearer $token" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: team-a" \
     -d '{
     "name": "SA token model test",
     "model": {
       "url": "https://gpt2-team-a.apps.rosa.prabhu-comhub.xqmp.p3.openshiftapps.com/v1",
       "name": "gpt2"
     },
     "benchmarks": [
       {
         "id": "arc_easy",
         "provider_id": "lm_evaluation_harness",
         "parameters": {
           "limit": 5,
           "num_examples": 10,
           "tokenizer": "google/flan-t5-small"
         }
       }
     ]
   }'

   Python SDK

   import os
   from evalhub import SyncEvalHubClient
   from evalhub.models.api import ModelConfig, BenchmarkConfig, JobSubmissionRequest
   
   client = SyncEvalHubClient(base_url=os.environ.get("HOST", "http://localhost:8080"))
   
   job = client.jobs.submit(
       JobSubmissionRequest(
           name="SA token model test",
           model=ModelConfig(
               url="https://gpt2-team-a.apps.rosa.prabhu-comhub.xqmp.p3.openshiftapps.com/v1",
               name="gpt2",
           ),
           benchmarks=[
               BenchmarkConfig(
                   id="arc_easy",
                   provider_id="lm_evaluation_harness",
                   parameters={"limit": 5, "num_examples": 10, "tokenizer": "google/flan-t5-small"},
               )
           ],
       ),
       headers={"X-Tenant": "team-a"},
   )

---

Scenario 3: In-cluster SA-token-authenticated services (KFP and others)

Use this when the adapter needs to call an in-cluster service (such as Kubeflow Pipelines) that requires the pod SA token. Use the _sa_token / _url key convention — the sidecar injects the SA token and routes to the real service URL.

Secret structure

Key	Value	Effect
<prefix>_sa_token	"" (empty)	Sidecar injects pod SA token
<prefix>_sa_token	<jwt>	Sidecar forwards value as-is
<prefix>_url	Real service URL	Sidecar routes to this upstream

The prefix must match between the _sa_token and _url keys (e.g. kfp_sa_token pairs with kfp_url).

kubectl create secret generic kfp-creds \
  --from-literal=kfp_sa_token="" \
  --from-literal=kfp_url="http://ml-pipeline.kubeflow.svc.cluster.local:8888" \
  -n team-a

1. Reference the secret in the job

   curl

   curl -k -X POST "$HOST/api/v1/evaluations/jobs" \
     -H "Authorization: Bearer $token" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: team-a" \
     -d '{
     "name": "kfp sa token test",
     "model": {
       "url": "https://your-model-endpoint/v1",
       "name": "gpt2",
       "auth": {
         "secret_ref": "kfp-creds"
       }
     },
     "benchmarks": [
       {
         "id": "arc_easy",
         "provider_id": "lm_evaluation_harness",
         "parameters": {
           "limit": 5,
           "num_examples": 10,
           "tokenizer": "google/flan-t5-small"
         }
       }
     ]
   }'

   Python SDK

   import os
   from evalhub import SyncEvalHubClient
   from evalhub.models.api import ModelConfig, BenchmarkConfig, JobSubmissionRequest
   
   client = SyncEvalHubClient(base_url=os.environ.get("HOST", "http://localhost:8080"))
   
   job = client.jobs.submit(
       JobSubmissionRequest(
           name="kfp sa token test",
           model=ModelConfig(
               url="https://your-model-endpoint/v1",
               name="gpt2",
               auth={"secret_ref": "kfp-creds"},
           ),
           benchmarks=[
               BenchmarkConfig(
                   id="arc_easy",
                   provider_id="lm_evaluation_harness",
                   parameters={"limit": 5, "num_examples": 10, "tokenizer": "google/flan-t5-small"},
               )
           ],
       ),
       headers={"X-Tenant": "team-a"},
   )

2. Call KFP from your adapter

   The adapter reads ref tokens from the secret mount and calls the sidecar. The sidecar resolves kfp_sa_token:ref → injects pod SA token → forwards to kfp_url.

   from evalhub.adapter.auth import read_model_auth_key
   
   kfp_token = read_model_auth_key("kfp_sa_token")  # returns "kfp_sa_token:ref"
   kfp_url    = read_model_auth_key("kfp_url")       # returns sidecar address
   
   resp = requests.get(
       f"{kfp_url}/apis/v1beta1/runs",
       headers={"Authorization": f"Bearer {kfp_token}"},
   )

---

Scenario 4: Multi-model secrets

Use this when a single eval job needs to call multiple model endpoints, each with its own API key and URL.

Secret structure

Use a <prefix>_api-key / <prefix>_url pair per model:

kubectl create secret generic multi-model-creds \
  --from-literal=model-1_api-key="sk-model1-key" \
  --from-literal=model-1_url="https://model1.example.com" \
  --from-literal=model-2_api-key="sk-model2-key" \
  --from-literal=model-2_url="https://model2.example.com" \
  -n team-a

1. Reference the secret in the job

   curl

   curl -k -X POST "$HOST/api/v1/evaluations/jobs" \
     -H "Authorization: Bearer $token" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: team-a" \
     -d '{
     "name": "multi-model test",
     "model": {
       "url": "https://model1.example.com/v1",
       "name": "model-1",
       "auth": {
         "secret_ref": "multi-model-creds"
       }
     },
     "benchmarks": [
       {
         "id": "arc_easy",
         "provider_id": "lm_evaluation_harness",
         "parameters": {
           "limit": 5,
           "num_examples": 10,
           "tokenizer": "google/flan-t5-small"
         }
       }
     ]
   }'

   Python SDK

   import os
   from evalhub import SyncEvalHubClient
   from evalhub.models.api import ModelConfig, BenchmarkConfig, JobSubmissionRequest
   
   client = SyncEvalHubClient(base_url=os.environ.get("HOST", "http://localhost:8080"))
   
   job = client.jobs.submit(
       JobSubmissionRequest(
           name="multi-model test",
           model=ModelConfig(
               url="https://model1.example.com/v1",
               name="model-1",
               auth={"secret_ref": "multi-model-creds"},
           ),
           benchmarks=[
               BenchmarkConfig(
                   id="arc_easy",
                   provider_id="lm_evaluation_harness",
                   parameters={"limit": 5, "num_examples": 10, "tokenizer": "google/flan-t5-small"},
               )
           ],
       ),
       headers={"X-Tenant": "team-a"},
   )

2. Use ref tokens in your adapter

   For multi-model secrets, your adapter code must read each key by name and send it as the Bearer token — the sidecar resolves the real key and routes to the matching _url.

   from evalhub.adapter.auth import read_model_auth_key
   
   model1_token = read_model_auth_key("model-1_api-key")  # returns "model-1_api-key:ref"
   model1_url   = read_model_auth_key("model-1_url")       # returns sidecar address
   
   model2_token = read_model_auth_key("model-2_api-key")  # returns "model-2_api-key:ref"
   model2_url   = read_model_auth_key("model-2_url")       # returns sidecar address
   
   resp = requests.post(
       f"{model1_url}/v1/chat/completions",
       headers={"Authorization": f"Bearer {model1_token}"},
       json={...},
   )

   Each request is independently routed based on the prefix in the ref token.

---

Open models (no authentication)

If your model endpoint is fully public and requires no API key, CA cert, or SA token, omit model.auth entirely. The sidecar still proxies all model traffic. No credential secret is mounted on the adapter.

---

Troubleshooting

If the evaluation job fails with authentication or TLS errors, check both the adapter and sidecar container logs:

# Adapter logs
kubectl logs -n <tenant-namespace> job/<job-name> -c adapter

# Sidecar logs
kubectl logs -n <tenant-namespace> job/<job-name> -c sidecar

Typical failures include:

• Missing CA certificate → TLS error on sidecar upstream connection
• Missing or invalid API key → 401 Unauthorized from the model endpoint
• An api-key key present in the secret but with an empty string value → 400 Bad Request from the sidecar
• RBAC misconfiguration (Scenario 2) → 403 Forbidden from the upstream model endpoint; check the RoleBinding and SA permissions
• _url key used as a ref token → 400 Bad Request from the sidecar (only api-key, *_api-key, *_sa_token are valid ref keys)

---

Summary

Scenario	Secret keys	model.auth	How the sidecar handles the request
API key	api-key, optionally ca_cert, hf-token	secret_ref required	Sidecar resolves :ref token → real key
SA token-authenticated, RBAC-protected model	None, or ca_cert only	Omit unless TLS	Sidecar injects pod SA token when no Authorization from adapter
KFP / in-cluster service	<prefix>_sa_token, <prefix>_url	secret_ref required	Sidecar injects pod SA token via _sa_token:ref
Multi-model	<prefix>_api-key, <prefix>_url per model	secret_ref required	Adapter sends per-prefix ref; sidecar resolves and routes to matching URL
Open model	None	Omit	Sidecar proxies; no credentials needed