Skip to content

Examples

Examples use the evalhub eval run --config submission shape: name, model, and benchmarks[]. See Configuration for the migration and adapter image requirements, and Model Authentication for the projected Secret files and sidecar routing contract.

Do not put model connection URLs or API keys in parameters, including nested model_args or model_roles settings. Those legacy connection fields are ignored; they cannot configure or override a Secret-backed connection.

When every role shares one gateway and credential, only api-key is needed. Replace the credential placeholder and create or update the Secret in team-a:

Terminal window
kubectl create secret generic maas-creds --namespace team-a \
--from-literal=api-key='<maas-credential>' \
--dry-run=client -o yaml | kubectl apply -f -
name: inspect-petri-maas-smoke
model:
url: https://maas.example.com/v1
name: target-model
auth:
secret_ref: maas-creds
benchmarks:
- id: inspect/petri-sycophancy
provider_id: inspect
parameters:
auditor_model: auditor-model
judge_model: judge-model
max_samples: 3
max_turns: 10

Each model identifier must exist on the gateway. The adapter receives the sidecar URL and api-key:ref; the sidecar supplies the real gateway key.

Petri: separate target, auditor and judge endpoints

Section titled “Petri: separate target, auditor and judge endpoints”

Create one Secret in the evaluation’s tenant namespace. These are placeholders; populate it using your secret-management process:

apiVersion: v1
kind: Secret
metadata:
name: inspect-model-auth
namespace: team-a
type: Opaque
stringData:
api-key: "<target-credential>"
auditor_api-key: "<auditor-credential>"
auditor_url: https://auditor.example/api/v1
judge_api-key: "<judge-credential>"
judge_url: https://judge.example/v1
realism_api-key: "<realism-credential>"
realism_url: https://realism.example/v1

The submission contains model identifiers and audit settings:

name: inspect-petri-independent-roles
model:
url: https://target.example/v1
name: target-model
auth:
secret_ref: inspect-model-auth
benchmarks:
- id: inspect/petri-sycophancy
provider_id: inspect
parameters:
auditor_model: auditor-model
judge_model: judge-model
realism_model: realism-model
realism_filter: true
max_samples: 3
max_turns: 10

For example, the adapter sees auditor_api-key:ref and http://localhost:8080/api/v1 for the auditor, and judge_api-key:ref and http://localhost:8080/v1 for the judge. The reference chooses the upstream; the visible hostname alone cannot identify the role. To share an endpoint with a different credential, provide the role key and omit its URL. To reuse the primary connection, omit both entries for that role.

Models served by different vendors can use this pattern through OpenAI-compatible gateways. Native Anthropic x-api-key clients are not supported for auxiliary Secret routing; setting ANTHROPIC_API_KEY does not configure an auditor or judge.

Ollama exposes /v1. Use a model ID present on that endpoint:

name: inspect-petri-ollama
model:
url: http://ollama:11434/v1
name: granite3.3:8b
benchmarks:
- id: inspect/petri-sycophancy
provider_id: inspect
parameters:
auditor_model: granite3.3:8b
judge_model: granite3.3:8b
max_samples: 3

This example reuses one unauthenticated endpoint; no Secret is needed. For a separate authenticated Ollama role, add its role pair to the Secret as above. An explicit ollama/ auxiliary model prefix selects Inspect’s Ollama wrapper, which uses OpenAI-compatible /v1 requests and Bearer auth. Detecting the upstream from port 11434 is unreliable after the sidecar rewrites the URL.

name: inspect-gsm8k-smoke
model:
url: https://maas.example.com/v1
name: target-model
auth:
secret_ref: maas-creds
benchmarks:
- id: inspect/gsm8k
provider_id: inspect
parameters:
max_samples: 3
log_level: info

For gated Hugging Face datasets, add hf-token to the same model-auth Secret. For a custom task use id: inspect/custom and put its import path in parameters.task, for example inspect_evals/mmlu.

HLE, CyberSecEval and StrongREJECT graders

Section titled “HLE, CyberSecEval and StrongREJECT graders”

For the HLE or CyberSecEval collection path, add judge_api-key / judge_url to the Secret. The collection supplies its task/scorer configuration, and the job selects the judge model:

name: inspect-hle-judge
model:
url: https://target.example/v1
name: target-model
auth:
secret_ref: inspect-model-auth
collection:
id: knowledge-reasoning-v1
benchmarks:
- id: inspect/hle
provider_id: inspect
parameters:
judge_model: judge-model
num_examples: 1

For CyberSecEval, use collection trustworthiness-v1 with benchmark override id: inspect/cyberseceval-2-pi. For StrongREJECT, use an explicit benchmark id: inspect/strong-reject and grader_model: judge-model; it also reads the judge Secret pair.

For standalone HLE’s named grader or any arbitrary named role, the prefix is the role name instead. Add grader_api-key / grader_url to the Secret and use:

parameters:
model_roles:
grader: grader-model
model_args:
temperature: 0.2

Do not include URLs or keys in model_roles or model_args. Legacy role dictionaries can retain a model name; their connection fields are ignored.

When generation needs a separate model endpoint, add these entries under the existing stringData key in the Petri example’s inspect-model-auth Secret. Keep its target, auditor and judge entries when applying the updated Secret:

scenarios_api-key: "<scenario-credential>"
scenarios_url: https://scenarios.example/v1
name: inspect-bloom-custom
model:
url: https://target.example/v1
name: target-model
auth:
secret_ref: inspect-model-auth
benchmarks:
- id: inspect/bloom-custom
provider_id: inspect
parameters:
bloom_template: delusion_sycophancy
auditor_model: auditor-model
judge_model: judge-model
scenarios_model: scenario-model
max_samples: 3

The bloom scenarios subprocess receives OPENAI_API_KEY and OPENAI_BASE_URL from the scenarios Secret pair. These variables are adapter outputs; do not supply a separate runtime generation credential. With no scenario entries, that step reuses the primary connection, even if the auditor has a separate endpoint. Use behavior_dir to select a pre-built behavior directory and skip generation.

Save one of the complete submission examples above to a YAML file:

Terminal window
evalhub eval run --config inspect-petri.yaml --wait
evalhub eval status <job-id>
evalhub eval results <job-id> --format json

Create or update the Secret before submission. Start a new evaluation after changing a Secret; an already-running sidecar caches the original values.