LM Evaluation Harness Configuration
Model authentication
Section titled “Model authentication”For OpenAI-compatible API backends, use model.url and the model-auth Secret;
keep model URLs and API keys out of parameters. Follow the shared
Model Authentication guide for setup and client usage.
Evaluation job example
Section titled “Evaluation job example”name: lm-eval-examplemodel: name: meta-llama/Llama-3.1-8B-Instruct url: http://model-endpoint:8080/v1benchmarks: - id: arc_easy provider_id: lm_evaluation_harness parameters: num_examples: 10 num_few_shot: 5The adapter recognizes the common parameters below. Individual tasks may impose additional constraints, so use the task definition and its README as the source of truth.
Common parameters
Section titled “Common parameters”| Parameter | Description |
|---|---|
num_examples | Limit the number of evaluated samples; useful for smoke tests but not final metrics |
num_few_shot | Number of few-shot examples, when supported by the task |
batch_size | Request batch size; defaults to 1 |
num_concurrent | Number of concurrent requests, capped by the adapter at 128 |
timeout_seconds | Request timeout in seconds; defaults to 300 |
tokenizer | Hugging Face tokenizer ID or local path used with the OpenAI-compatible endpoint; normally defaults to model.name when omitted or blank, except that disconnected-mode detection can select staged /test_data/tokenizer when the dataset bundle is detected (see Disconnected Cluster Evaluation) |
random_seed | Random seed used by LM Evaluation Harness; defaults to 42 |
parameters | Nested scalar generation parameters serialized as LM Evaluation Harness generation kwargs |
Only the top-level keys listed above are interpreted by the adapter. Keys in
the nested parameters object are serialized as generation kwargs; other
top-level keys are not automatically translated into task configuration.
Tokenizer selection and authentication
Section titled “Tokenizer selection and authentication”The adapter uses a Hugging Face tokenizer even when evaluating a model through
an OpenAI-compatible endpoint. If benchmarks[].parameters.tokenizer is omitted or
blank, the adapter normally uses model.name as the Hugging Face tokenizer identifier.
When disconnected-mode detection finds a staged dataset bundle and the conventional
/test_data/tokenizer directory, it uses that local tokenizer instead. An explicit
parameters.tokenizer value takes precedence over this fallback. See
Disconnected Cluster Evaluation for the required
staged layout and detection conditions. The example above uses
meta-llama/Llama-3.1-8B-Instruct for both the served model name and the tokenizer.
If your endpoint uses a model alias that is not a Hugging Face repository ID,
set parameters.tokenizer explicitly to a tokenizer compatible with the served
model. For example, for an endpoint serving Llama 3.1 8B Instruct:
name: tokenizer-override-evalmodel: name: my-served-llama url: http://model-endpoint:8080/v1 auth: secret_ref: my-model-authbenchmarks: - id: arc_easy provider_id: lm_evaluation_harness parameters: tokenizer: meta-llama/Llama-3.1-8B-Instruct num_examples: 10With the CLI, pass the tokenizer using --param tokenizer=.... For example,
when the endpoint serves Qwen/Qwen3.6-35B-A3B under the name Qwen3.6-35B-A3B:
evalhub eval run \ --name my-eval \ --model-url https://my-model-url/v1 \ --model-name Qwen3.6-35B-A3B \ --provider lm_evaluation_harness \ --model-auth-secret my-model-auth \ --param tokenizer=Qwen/Qwen3.6-35B-A3B \ -b winogenderWithout the override, the adapter tries Qwen3.6-35B-A3B as the Hugging Face
repository ID. If that repository does not exist, the evaluation fails with a
repository-not-found error.
To download a tokenizer from a gated repository, provide HF_TOKEN for a
Hugging Face account with approved access, using a token with read permission
for that repository. This applies both to an explicitly
configured tokenizer and to the implicit model.name default. Authentication
to the model endpoint does not grant access to a gated Hugging Face repository.
Use the existing Hugging Face token Secret setup
to add an hf-token key to your model authentication Secret (my-model-auth
in both examples). Reference that Secret through model.auth.secret_ref in
the job configuration or --model-auth-secret in the CLI. This makes the token available as
HF_TOKEN to the built-in adapter. See
Model Authentication for combining the Hugging
Face token with endpoint credentials.
For air-gapped clusters, see Disconnected Cluster Evaluation.
Task-specific requirements
Section titled “Task-specific requirements”Some tasks require additional image-level configuration:
| Benchmark(s) | Requirement |
|---|---|
careqa_open_perplexity | evaluate and bert-score (the runner image also needs matplotlib for bert_score import) |
tinyTruthfulQA | tinyBenchmarks |
humaneval, humaneval_instruct, mbpp | HF_ALLOW_CODE_EVAL=1; generated code is executed and requires isolation |
| Gated Hugging Face datasets | HF_TOKEN, or an hf-token key in the model authentication Secret |
These requirements belong in the ta-lmes-job image or its runtime
configuration. Adding a benchmark to config/providers/lm_evaluation_harness.yaml
does not install missing packages.
Code execution safety
Section titled “Code execution safety”humaneval, humaneval_instruct, and mbpp use the code_eval metric, which executes
model-generated Python code. Run these benchmarks only in an appropriately
isolated evaluation environment. The runner must explicitly set
HF_ALLOW_CODE_EVAL=1 before the benchmark is executed.
In the current LM Evaluation Harness integration, this code runs inside the evaluation Job container; it is not delegated to a separate code-execution sandbox.
Do not enable this setting for an untrusted, shared host environment.
Dataset and token handling
Section titled “Dataset and token handling”Review the task’s dataset_path, dataset_name, and download code before
deployment. Tasks may download data at runtime, require a Hugging Face token,
or require data to be pre-staged for disconnected clusters. For air-gapped
deployments, see Disconnected Cluster Evaluation.