Skip to content

Configuration Reference

Complete reference for Inspect AI adapter configuration options.

Jobs are submitted through the EvalHub API / CLI using a name + model + benchmarks[] body (same shape as evalhub eval run --config):

name: inspect-petri-sycophancy-001
model:
url: https://maas.example.com/v1
name: microsoft-phi-4
auth:
secret_ref: maas-creds
benchmarks:
- id: inspect/petri-sycophancy
provider_id: inspect
parameters:
auditor_model: gpt-oss-20b
judge_model: deepseek-r1-distill-qwen-14b
max_samples: 5
max_turns: 20

Inside the job pod, EvalHub flattens each benchmark into an adapter JobSpec (provider_id, benchmark_id, model, parameters). The tables below describe those fields.

ParameterTypeDescriptionExample
name (submission) / id (adapter)stringJob identifier"inspect-petri-sycophancy-001"
benchmarks[].provider_id / provider_idstringMust be "inspect""inspect"
benchmarks[].id / benchmark_idstringBenchmark to run (see Benchmarks)"inspect/petri-sycophancy"
model.namestringName of the evaluated (target) model"microsoft-phi-4"
ParameterTypeDescriptionDefault
model.urlstringOpenAI-compatible endpoint for the target (sets global OPENAI_BASE_URL)null
model.auth.secret_refstringKubernetes Secret with model credentials (for example api-key for MaaS / LiteLLM)null
benchmark_indexintegerIndex when running multiple benchmarks0
experiment_namestringMLflow experiment name for result trackingnull
parametersobjectInspect-specific configuration (see below){}
callback_urlstringEvalHub service callback URLnull

Model identifiers remain parameters; endpoints and credentials come from the model-auth Secret. Use the identifier served by the selected OpenAI-compatible endpoint, either bare or org/model. An auxiliary ollama/ prefix explicitly selects Inspect’s Ollama wrapper, which also uses OpenAI-compatible Bearer auth.

Provider styleModel name examples
MaaS / LiteLLMmicrosoft-phi-4, gpt-oss-20b, deepseek-r1-distill-qwen-14b
vLLM / HuggingFaceibm-granite/granite-3.3-8b-instruct, meta-llama/Llama-3.3-70B-Instruct
Ollamagranite3.3:8b, llama3.3, qwen3:32b
OpenRoutermeta-llama/llama-3.3-70b-instruct
Models behind an OpenAI-compatible gatewayThe model ID exposed by that gateway

Model connection URLs and API keys are not supported in API payload parameters. Use the shared Secret setup and the Inspect role mapping below. The worked examples show shared MaaS credentials and separate Petri/Bloom role connections.

For existing deployments, see the image requirements and Inspect migration steps.

UseModel selectionSecret connection
Targetmodel.nameapi-key, with endpoint in model.url
Petri/Bloom auditorauditor_modelauditor_api-key / auditor_url
Petri/Bloom judgejudge_modeljudge_api-key / judge_url
Bloom generationscenarios_modelscenarios_api-key / scenarios_url
Optional realism rolerealism_modelrealism_api-key / realism_url
HLE / CyberSecEval collection judgejudge_modeljudge_api-key / judge_url
StrongREJECTgrader_modeljudge_api-key / judge_url
Named roles, including standalone HLEmodel_roles: {grader: model_id}grader_api-key / grader_url; use the matching prefix for any role

Remove api_key, target_base_url, target_api_key, role *_base_url / *_api_key, and *_anthropic_* connection fields from parameters. Put auxiliary connections in the Secret and the primary URL in model.url. Legacy connection fields are ignored under the shared migration and fallback rules.

Connection fields in model_args, including nested dictionaries and lists, are removed before forwarding to Inspect; generation settings are retained. Legacy model_roles dictionaries may contain a model name, but their connection settings are ignored. Auxiliary OPENAI_JUDGE_*, OPENAI_API_KEY, ANTHROPIC_API_KEY, and URL environment variables do not configure role connections. The adapter sets framework environment values from Secret resolution.

Native anthropic/ auxiliary clients are not supported by this Bearer-reference contract because they use x-api-key. Serve such models through an OpenAI-compatible gateway. Inspect’s Ollama wrapper uses /v1 and Bearer auth; its upstream port cannot be detected after the URL has been rewritten.

All configuration below is specified in the parameters object of each benchmark (CLI/API) or the flattened adapter JobSpec.parameters.

The target is configured by model.name, model.url, and model.auth.secret_ref, outside parameters. Do not add target URLs or API keys to this object.

ParameterTypeDescriptionDefault
auditor_modelstringAuditor model name (bare or org/model)"claude-sonnet-4-6"
ParameterTypeDescriptionDefault
judge_modelstringJudge model name. Use the strongest available model — judge quality has the largest impact on audit reliability."claude-opus-4-7"

These Claude defaults require OpenAI-compatible endpoints serving those model IDs. Override auditor_model and judge_model when your MaaS endpoint or gateway serves different IDs. Native Anthropic authentication has the limitations described above; the model name does not configure a separate endpoint or authentication scheme.

ParameterTypeDescriptionDefault
scenarios_modelstringModel for the bloom scenarios generation stepauditor_model
ParameterTypeDescriptionDefault
realism_modelstringOptional fourth model role for realism filteringauditor_model
ParameterTypeDescriptionDefault
behavior_dirstringPath to a pre-built behaviour directory (bloom init + bloom scenarios). Skips scenario generation when set.null
bloom_templatestringTemplate for bloom init --from <template>. Required for inspect/bloom-custom when behavior_dir is unset. Example: delusion_sycophancy.null
ParameterTypeDescriptionDefault
seed_instructionsstringOverride seed selection: tags:sycophancy, id:seed_name, id:seed1,seed2, inline text, or a path. When unset, the benchmark default seed tag is used.null
judge_dimensionsstringOverride judge dimensions: tags:safety, a dimensions directory path, or dimension names. Defaults to all 38 built-in dimensions.null
max_turnsintegerMaximum auditor turns per scenario30
enable_rollbackbooleanAllow the auditor to restart from a prior checkpointtrue
realism_filterboolean | numberFilter auditor outputs by realism score. A boolean enables or disables filtering; a numeric value sets the realism-score threshold (for example 0.6). Experimental.false
target_toolsstringTool-creation mode: synthetic (default), fixed, or none"synthetic"
epochsintegerRepeat each seed/scenario N times1
ParameterTypeDescriptionDefault
taskstringTask spec override. Required for inspect/custom. Accepts a Python import path (inspect_evals/mmlu), file path (./my_task.py@my_task), or Petri task (inspect_petri/audit).null
judge_modelstringHLE / CyberSecEval collection grader model; connection uses the judge Secret pairunset
grader_modelstringStrongREJECT grader model; connection uses the judge Secret pairunset
model_rolesobjectNamed role to model ID, e.g. {"grader": "deployed-grader"}; connection uses that role’s Secret pair{}
model_argsobjectFramework generation settings; connection overrides are ignored{}
sandboxstringExecution sandbox: none, docker, or k8s. Use docker for tasks that need code-execution isolation (for example swe-bench, humaneval) when Docker is available.none
ParameterTypeDescriptionDefault
max_tasksintegerNumber of tasks to run concurrently1
max_samplesintegerLimit samples per task. Strongly recommended for Petri/Bloom development runs to control cost.null
task_argsobjectPass-through task arguments forwarded as -T key=value. For Dish (research preview): {"dish_scaffold": "claude-code"}.{}
log_levelstringInspect AI log verbosity (debug, info, warning, error)"info"
languagesarrayISO 639 language codes for EvalCard metadata[en]
languages_countintegerNumber of languages in the evaluation dataset1

Standard inspect-evals benchmarks run without a Docker sandbox inside Kubernetes pods (code executes in the adapter container). This is the only sandbox available in typical K8s deployments. Override with parameters.sandbox if you have a different provider configured (for example "docker" for local development with Docker Engine):

{ "parameters": { "sandbox": "docker" } }

Petri and Bloom modes do not use a sandbox.

Some inspect-evals benchmarks (for example humaneval, mmlu) download datasets from the HuggingFace Hub. Add hf-token to the model-auth Secret referenced by model.auth.secret_ref. It is passed through to /var/run/secrets/model/hf-token, and the adapter configures HF_TOKEN for the framework. See Model Authentication for the mount layout.