Examples
Examples use the evalhub eval run --config submission shape: name, model,
and benchmarks[]. See Configuration for the migration and
adapter image requirements, and Model Authentication
for the projected Secret files and sidecar routing contract.
Do not put model connection URLs or API keys in parameters, including nested
model_args or model_roles settings. Those legacy connection fields are
ignored; they cannot configure or override a Secret-backed connection.
MaaS (OpenAI-compatible API key Secret)
Section titled “MaaS (OpenAI-compatible API key Secret)”When every role shares one gateway and credential, only api-key is needed.
Replace the credential placeholder and create or update the Secret in team-a:
kubectl create secret generic maas-creds --namespace team-a \ --from-literal=api-key='<maas-credential>' \ --dry-run=client -o yaml | kubectl apply -f -name: inspect-petri-maas-smokemodel: url: https://maas.example.com/v1 name: target-model auth: secret_ref: maas-credsbenchmarks: - id: inspect/petri-sycophancy provider_id: inspect parameters: auditor_model: auditor-model judge_model: judge-model max_samples: 3 max_turns: 10Each model identifier must exist on the gateway. The adapter receives the
sidecar URL and api-key:ref; the sidecar supplies the real gateway key.
Petri: separate target, auditor and judge endpoints
Section titled “Petri: separate target, auditor and judge endpoints”Create one Secret in the evaluation’s tenant namespace. These are placeholders; populate it using your secret-management process:
apiVersion: v1kind: Secretmetadata: name: inspect-model-auth namespace: team-atype: OpaquestringData: api-key: "<target-credential>" auditor_api-key: "<auditor-credential>" auditor_url: https://auditor.example/api/v1 judge_api-key: "<judge-credential>" judge_url: https://judge.example/v1 realism_api-key: "<realism-credential>" realism_url: https://realism.example/v1The submission contains model identifiers and audit settings:
name: inspect-petri-independent-rolesmodel: url: https://target.example/v1 name: target-model auth: secret_ref: inspect-model-authbenchmarks: - id: inspect/petri-sycophancy provider_id: inspect parameters: auditor_model: auditor-model judge_model: judge-model realism_model: realism-model realism_filter: true max_samples: 3 max_turns: 10For example, the adapter sees auditor_api-key:ref and
http://localhost:8080/api/v1 for the auditor, and judge_api-key:ref and
http://localhost:8080/v1 for the judge. The reference chooses the upstream;
the visible hostname alone cannot identify the role. To share an endpoint with
a different credential, provide the role key and omit its URL. To reuse the
primary connection, omit both entries for that role.
Models served by different vendors can use this pattern through OpenAI-compatible
gateways. Native Anthropic x-api-key clients are not supported for auxiliary
Secret routing; setting ANTHROPIC_API_KEY does not configure an auditor or judge.
Ollama through its OpenAI-compatible API
Section titled “Ollama through its OpenAI-compatible API”Ollama exposes /v1. Use a model ID present on that endpoint:
name: inspect-petri-ollamamodel: url: http://ollama:11434/v1 name: granite3.3:8bbenchmarks: - id: inspect/petri-sycophancy provider_id: inspect parameters: auditor_model: granite3.3:8b judge_model: granite3.3:8b max_samples: 3This example reuses one unauthenticated endpoint; no Secret is needed. For a
separate authenticated Ollama role, add its role pair to the Secret as above.
An explicit ollama/ auxiliary model prefix selects Inspect’s Ollama wrapper,
which uses OpenAI-compatible /v1 requests and Bearer auth. Detecting the
upstream from port 11434 is unreliable after the sidecar rewrites the URL.
Standard inspect-evals
Section titled “Standard inspect-evals”name: inspect-gsm8k-smokemodel: url: https://maas.example.com/v1 name: target-model auth: secret_ref: maas-credsbenchmarks: - id: inspect/gsm8k provider_id: inspect parameters: max_samples: 3 log_level: infoFor gated Hugging Face datasets, add hf-token to the same model-auth Secret.
For a custom task use id: inspect/custom and put its import path in
parameters.task, for example inspect_evals/mmlu.
HLE, CyberSecEval and StrongREJECT graders
Section titled “HLE, CyberSecEval and StrongREJECT graders”For the HLE or CyberSecEval collection path, add judge_api-key / judge_url to
the Secret. The collection supplies its task/scorer configuration, and the job
selects the judge model:
name: inspect-hle-judgemodel: url: https://target.example/v1 name: target-model auth: secret_ref: inspect-model-authcollection: id: knowledge-reasoning-v1 benchmarks: - id: inspect/hle provider_id: inspect parameters: judge_model: judge-model num_examples: 1For CyberSecEval, use collection trustworthiness-v1 with benchmark override
id: inspect/cyberseceval-2-pi. For StrongREJECT, use an explicit benchmark
id: inspect/strong-reject and grader_model: judge-model;
it also reads the judge Secret pair.
For standalone HLE’s named grader or any arbitrary named role, the prefix is
the role name instead. Add grader_api-key / grader_url to the Secret and use:
parameters: model_roles: grader: grader-model model_args: temperature: 0.2Do not include URLs or keys in model_roles or model_args. Legacy role
dictionaries can retain a model name; their connection fields are ignored.
Bloom behavioural suites
Section titled “Bloom behavioural suites”When generation needs a separate model endpoint, add these entries under the
existing stringData key in the Petri example’s inspect-model-auth Secret.
Keep its target, auditor and judge entries when applying the updated Secret:
scenarios_api-key: "<scenario-credential>"scenarios_url: https://scenarios.example/v1name: inspect-bloom-custommodel: url: https://target.example/v1 name: target-model auth: secret_ref: inspect-model-authbenchmarks: - id: inspect/bloom-custom provider_id: inspect parameters: bloom_template: delusion_sycophancy auditor_model: auditor-model judge_model: judge-model scenarios_model: scenario-model max_samples: 3The bloom scenarios subprocess receives OPENAI_API_KEY and OPENAI_BASE_URL
from the scenarios Secret pair. These variables are adapter outputs; do not
supply a separate runtime generation credential. With no scenario entries, that
step reuses the primary connection, even if the auditor has a separate endpoint.
Use behavior_dir to select a pre-built behavior directory and skip generation.
Submit via SDK CLI
Section titled “Submit via SDK CLI”Save one of the complete submission examples above to a YAML file:
evalhub eval run --config inspect-petri.yaml --waitevalhub eval status <job-id>evalhub eval results <job-id> --format jsonCreate or update the Secret before submission. Start a new evaluation after changing a Secret; an already-running sidecar caches the original values.