Disconnected Cluster Evaluation
This guide explains how to stage benchmark data for disconnected evaluation. EvalHub’s init container copies each benchmark’s test_data_ref bundle to /test_data; the adapter then reads files from the paths its framework expects. The instructions cover the nine curated collections and 24 benchmark entries in the matrix below.
Prerequisites
Section titled “Prerequisites”- Target and judge endpoints must be reachable from the evaluation namespace. Disconnected evaluation still sends inference requests to these services.
- Adapter, init-container, and sidecar images must be available from a registry reachable by the cluster.
- Staged assets must be reachable by the init container from the job namespace. Use an internal S3-compatible store such as MinIO, or a PVC that already contains the files.
- Stage assets while connected. The staging workstation or preparation environment may need Internet access and Hugging Face access. The evaluation pod itself does not need public Internet.
- A
test_data_ref.hfortest_data_ref.gitsource is fetched when the job starts. It is not an offline source unless the referenced service is reachable inside the cluster. Use S3/PVC for a disconnected cluster.
Curated benchmark asset matrix
Section titled “Curated benchmark asset matrix”Each S3 prefix below is an example of a narrow bundle for one benchmark entry. Place the listed paths at the root of that prefix; EvalHub mounts that root at /test_data. The table uses the current curated collection configs and the disconnected asset preparation used by the integration test.
| Collection | Benchmark / provider | Example test_data_ref.s3.key | Required files under /test_data |
|---|---|---|---|
| Coding v1 | lcb:codegeneration_v6 / LightEval | disconnected/coding-v1/lcb-codegeneration-v6/ | Task-specific Hugging Face datasets/ and hub/ caches; lighteval/tinyBenchmarks.pkl; NLTK nltk_data/tokenizers/punkt/ and punkt_tab/; stage a model tokenizer only if the selected model/task configuration uses one |
| Document Understanding v1 | common-words-extraction / RULER | disconnected/document-understanding-v1/common-words-extraction/ | english_words.json; complete HF tokenizer at tokenizer/ |
| Document Understanding v1 | frequency-words-extraction / RULER | disconnected/document-understanding-v1/frequency-words-extraction/ | Complete HF tokenizer at tokenizer/; no task dataset file for the curated defaults |
| Document Understanding v1 | qa-squad / RULER | disconnected/document-understanding-v1/qa-squad/ | squad.json; complete HF tokenizer at tokenizer/ |
| Document Understanding v1 | qa-hotpotqa / RULER | disconnected/document-understanding-v1/qa-hotpotqa/ | hotpotqa.json; complete HF tokenizer at tokenizer/ |
| Document Understanding v1 | niah-single-essay / RULER | disconnected/document-understanding-v1/niah-single-essay/ | PaulGrahamEssays.json; complete HF tokenizer at tokenizer/ |
| Document Understanding v1 | variable-tracking / RULER | disconnected/document-understanding-v1/variable-tracking/ | Complete HF tokenizer at tokenizer/; no task dataset file for the curated defaults |
| Fairness and Bias v1 | winogender / LM Evaluation Harness | disconnected/fairness-bias-v1/winogender/ | oskarvanderwal--winogender--all/ Arrow bundle and tokenizer/ |
| Fairness and Bias v1 | crows_pairs_english / LM Evaluation Harness | disconnected/fairness-bias-v1/crows-pairs-english/ | jannalu--crows_pairs_multilingual--english/ Arrow bundle and tokenizer/ |
| Fairness and Bias v1 | bbq / LM Evaluation Harness | disconnected/fairness-bias-v1/bbq/ | oskarvanderwal--bbq--All/ Arrow bundle and tokenizer/ |
| Instruction and Output Reliability v1 | ifbench / IFBench | No data prefix required | Dataset is included in the supported adapter image; target endpoint must be reachable |
| Instruction and Output Reliability v1 | jsonschema_bench_easy / LM Evaluation Harness | disconnected/instruction-output-v1/jsonschema-bench-easy/ | epfl-dlab--JSONSchemaBench--Github_easy/ Arrow bundle and tokenizer/ |
| Instruction and Output Reliability v1 | followbench / FollowBench | No data prefix required | Dataset is included in the supported adapter image; target and configured judge endpoints must be reachable |
| Knowledge and Reasoning v1 | gpqa:diamond / LightEval | disconnected/knowledge-reasoning-v1/gpqa-diamond/ | Task-specific Hugging Face datasets/ and hub/ caches; lighteval/tinyBenchmarks.pkl; NLTK nltk_data/tokenizers/punkt/ and punkt_tab/; stage a model tokenizer only if the selected model/task configuration uses one |
| Knowledge and Reasoning v1 | inspect/hle / Inspect | disconnected/knowledge-reasoning-v1/inspect-hle/ | Pinned HLE Hugging Face cache, including dataset and referenced Hub/Xet files; preserve cache layout |
| Knowledge and Reasoning v1 | math_500 / LightEval | disconnected/knowledge-reasoning-v1/math-500/ | Task-specific Hugging Face datasets/ and hub/ caches; lighteval/tinyBenchmarks.pkl; NLTK nltk_data/tokenizers/punkt/ and punkt_tab/; stage a model tokenizer only if the selected model/task configuration uses one |
| Multimodal v1 | inspect/docvqa / Inspect | disconnected/multimodal-v1/inspect-docvqa/ | Pinned DocVQA Hugging Face cache, including the image data and all referenced Hub/Xet files; preserve cache layout |
| Safety and Ethical Principles v1 | toxigen / LM Evaluation Harness | disconnected/safety-ethical-principles-v1/toxigen/ | skg--toxigen-data--annotated/ Arrow bundle and tokenizer/ |
| Safety and Ethical Principles v1 | ethics_cm / LM Evaluation Harness | disconnected/safety-ethical-principles-v1/ethics-cm/ | EleutherAI--hendrycks_ethics--commonsense/ Arrow bundle and tokenizer/ |
| Safety and Ethical Principles v1 | bigbench_hhh_alignment_multiple_choice / LM Evaluation Harness | disconnected/safety-ethical-principles-v1/bigbench-hhh/ | hails--bigbench--hhh_alignment_zero_shot/ Arrow bundle and tokenizer/ |
| Tool Use v1 | First inspect/bfcl entry / Inspect | disconnected/tool-use-v1/bfcl/ | Complete BFCL/ cache for the selected categories, including the simple_python sentinel and category support files |
| Tool Use v1 | Second inspect/bfcl entry / Inspect | disconnected/tool-use-v1/bfcl/ | Same complete BFCL/ cache; this repeated entry still needs its own test_data_ref |
| Trustworthiness v1 | inspect/cyberseceval-2-pi / Inspect | disconnected/trustworthiness-v1/cyberseceval-2-pi/ | Pinned dataset at cyberseceval_2/prompt_injection_9b05e90e7a57333d006555420f7289cb.json |
| Trustworthiness v1 | truthfulqa_mc1 / LM Evaluation Harness | disconnected/trustworthiness-v1/truthfulqa-mc1/ | truthful_qa--multiple_choice/ Arrow bundle and tokenizer/ |
The matrix’s collection membership and category lists can change. Confirm them against the collection configuration deployed in your cluster. For example, Tool Use v1 has two separate BFCL entries; both require their own test_data_ref, even when they point to the same S3 prefix.
Per-benchmark references and narrow prefixes
Section titled “Per-benchmark references and narrow prefixes”test_data_ref belongs on each benchmark object, not on the collection or job as a whole. Upload a benchmark’s bundle under its own prefix and reference that prefix from that benchmark. Avoid attaching a single large prefix containing every collection’s assets to every benchmark: the init container would copy unrelated files into each job’s /test_data volume.
Example reference:
{ "test_data_ref": { "s3": { "bucket": "mlpipeline", "key": "disconnected/fairness-bias-v1/winogender/", "secret_ref": "minio-credentials" } }}A bundle with the LM Evaluation Harness layout above would contain:
s3://mlpipeline/disconnected/fairness-bias-v1/winogender/ oskarvanderwal--winogender--all/ dataset_dict.json ... tokenizer/ tokenizer.json tokenizer_config.json ...For each benchmark, include only its dataset/cache and the tokenizer or other support files that benchmark requires. The path after the S3 prefix becomes the same path under /test_data. Uploading files to object storage is not sufficient: each data-backed benchmark entry must reference its own staged source.
Connected staging versus disconnected evaluation
Section titled “Connected staging versus disconnected evaluation”Prepare the assets from a connected staging machine or an approved connected preparation environment. Gated Hugging Face credentials are needed only while downloading gated files such as HLE or GPQA; do not put access tokens in job parameters, the uploaded bundle, or MinIO credentials. Once the assets are staged, the evaluation pod must run without public Internet access. It must still be able to reach the configured target model, judge model (if any), and internal asset store used by the init container.
Target and judge connectivity
Section titled “Target and judge connectivity”Asset staging does not provide an inference endpoint. The target configured by model.url must resolve and accept requests from the job through the sidecar. For HLE, CyberSecEval 2 Prompt Injection, and FollowBench, configure the judge model name in the benchmark’s parameters.judge_model. The target and judge can share the target connection; when a distinct judge connection is required, configure its credential and optional endpoint in the Secret referenced by model.auth.secret_ref (judge_api-key and, when needed, judge_url). Do not put credentials in parameters or the dataset bundle. See Judge and scorer models for the Secret format and routing behavior.
LM Evaluation Harness
Section titled “LM Evaluation Harness”How it works
Section titled “How it works”lm_eval normally downloads datasets from Hugging Face at evaluation time. On a disconnected cluster, pre-stage the datasets and tokenizer on a connected machine and make them available to each evaluation pod under /test_data.
The init container syncs the configured test_data_ref source to /test_data. The adapter enables Hugging Face offline mode when it finds both a usable local tokenizer under /test_data and a staged dataset bundle containing dataset_dict.json. An explicit parameters.tokenizer path under /test_data is used when supplied. If parameters.tokenizer is omitted or blank, the adapter can use the conventional /test_data/tokenizer directory when a sibling dataset bundle is detected. An explicit tokenizer value takes precedence over that fallback.
For example, /test_data/allenai--ai2_arc--ARC-Easy/dataset_dict.json and /test_data/tokenizer/ are siblings. Keep the uploaded dataset and tokenizer files in this layout unless you set an explicit local tokenizer path.
Step 1: Pre-download Datasets
Section titled “Step 1: Pre-download Datasets”Install the dataset libraries at the versions pinned by the deployed LM Evaluation Harness adapter. The adapter pins both datasets==3.1.0 and pyarrow==25.0.1; matching both avoids dataset serialization and Arrow compatibility mismatches:
pip install "datasets==3.1.0" "pyarrow==25.0.1"Download the dataset and save it as an Arrow bundle. The directory name must follow the slug format used by the offline loader:
from datasets import load_dataset, get_dataset_config_names
# Single subset (e.g. arc_easy)dataset = load_dataset("allenai/ai2_arc", "ARC-Easy")dataset.save_to_disk("staging/arc_easy/allenai--ai2_arc--ARC-Easy")# ^^^^^^^^^^ one bundle directory per benchmark# ^^^ slug: dataset_path.replace("/","--") + "--" + subset
# No subset (e.g. hellaswag)dataset = load_dataset("hellaswag")dataset.save_to_disk("staging/hellaswag/hellaswag")# ^^^^^^^^^ one bundle directory per benchmark# ^^^ slug: dataset_path.replace("/","--")
# Multiple subsets (e.g. blimp has 67 subsets — download all at once)for subset in get_dataset_config_names("blimp"): load_dataset("blimp", subset).save_to_disk(f"staging/blimp/blimp--{subset}")The slug rule in one line:
| Has subset? | Slug formula | Example |
|---|---|---|
| Yes | dataset_path.replace("/","--") + "--" + subset | allenai--ai2_arc--ARC-Easy |
| No | dataset_path.replace("/","--") | hellaswag |
Examples:
| Benchmark | Dataset Path | Subset | Slug (directory name) |
|---|---|---|---|
arc_easy | allenai/ai2_arc | ARC-Easy | allenai--ai2_arc--ARC-Easy |
hellaswag | hellaswag | - | hellaswag |
blimp | blimp | all subsets (67) | blimp--<subset> per subtask |
crows_pairs_english | jannalu/crows_pairs_multilingual | english | jannalu--crows_pairs_multilingual--english |
blimpruns as 67 separate sub-tasks, each backed by a different subset of theblimpHuggingFace dataset. Use the multi-subset snippet above (get_dataset_config_names) to download them all in one go.
The full list of benchmark → dataset mappings is in
docs/dataset-mapping.md. This follows the movingincubationbranch, so verify each mapping against the task configuration in the exact adapter revision deployed in your cluster. For example, the RHOAI adapter configuration forcrows_pairs_englishusesjannalu/crows_pairs_multilingualwith theenglishsubset.
To stage that dataset, save it under
jannalu--crows_pairs_multilingual--english(the dataset path with/replaced by--, followed by--english). If MinIO already containsBigScienceBiasEval--crows_pairs_multilingual--english, that is the old source mapping; restage it under the slug above. The adapter uses the Parquet-backed dataset and does not requiretrust_remote_code.
Step 2: Pre-download Tokenizer
Section titled “Step 2: Pre-download Tokenizer”On a connected machine, download the tokenizer files:
pip install huggingface_hubSome Hugging Face repositories are gated. Accept the repository’s terms and authenticate on this staging machine with an account that has read access (for example, use hf auth login or provide HF_TOKEN to the download process). The token is only needed to download the files: do not put it in the staging directory or upload it with the assets. A disconnected evaluation should consume the staged files without fetching them from Hugging Face.
import osfrom huggingface_hub import hf_hub_download
repo_id = "meta-llama/Llama-3.2-1B-Instruct"revision = "main" # Replace with the exact tokenizer revision compatible with the served model.out_dir = "./staging/arc_easy/tokenizer"os.makedirs(out_dir, exist_ok=True)
for filename in ["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json"]: hf_hub_download(repo_id=repo_id, revision=revision, filename=filename, local_dir=out_dir)Set revision to the tokenizer repository revision paired with the served model; use its full commit SHA for reproducible staging. Do not assume that the model endpoint name identifies this tokenizer revision. This example saves files to ./staging/arc_easy/tokenizer/. In the pod this maps to /test_data/tokenizer. Put the compatible tokenizer in each benchmark bundle that requires it; you can copy the same directory into multiple bundles when those benchmarks use the same tokenizer revision.
The list above is illustrative, not exhaustive. Stage every tokenizer, vocabulary, merge, and configuration file required by the exact tokenizer and revision you evaluate. Check the tokenizer repository and include files such as
tokenizer.model(SentencePiece) orvocab.jsonandmerges.txt(BPE) when applicable; the three common files above are not sufficient for every model.
Step 3: Upload a narrow bundle to MinIO
Section titled “Step 3: Upload a narrow bundle to MinIO”For each benchmark, create a separate staging directory that contains only its dataset bundle and tokenizer. For example, arrange the ARC Easy files as:
staging/arc_easy/ allenai--ai2_arc--ARC-Easy/ dataset_dict.json ... tokenizer/ tokenizer.json tokenizer_config.json ...Use the AWS CLI to upload this one benchmark bundle under its own prefix:
# Export MinIO credentials in the connected staging shell.export AWS_ACCESS_KEY_ID=<minio-access-key>export AWS_SECRET_ACCESS_KEY=<minio-secret-key>export AWS_DEFAULT_REGION=us-east-1
# Port-forward MinIO (replace namespace and pod name as needed).kubectl port-forward -n minio pod/<minio-pod-name> 9000:9000 &
aws --endpoint-url http://127.0.0.1:9000 s3 sync \ ./staging/arc_easy/ \ s3://mlpipeline/disconnected/lm-evaluation-harness/arc-easy/The contents of that prefix appear at the root of /test_data. Repeat with a different local bundle directory and S3 prefix for each benchmark. For a benchmark such as BLiMP that uses many subsets, put all required subset bundles for that benchmark beneath its one prefix; do not include unrelated benchmarks’ data.
Step 4: Create the MinIO Credentials Secret
Section titled “Step 4: Create the MinIO Credentials Secret”The init container uses a Kubernetes Secret to authenticate with MinIO. Create the Secret in the evaluation job namespace before submitting the job:
kubectl create secret generic minio-credentials \ --namespace <job-namespace> \ --from-literal=AWS_ACCESS_KEY_ID=<minio-access-key> \ --from-literal=AWS_SECRET_ACCESS_KEY=<minio-secret-key> \ --from-literal=AWS_DEFAULT_REGION=us-east-1 \ --from-literal=AWS_S3_ENDPOINT=http://<minio-host>:<port>The secret_ref value in test_data_ref.s3 must match this Secret name (minio-credentials in the example above). Without this Secret the init container fails before the evaluation runs. See Using Custom Data for the full list of required Secret keys.
Step 5: Submit an Evaluation Job
Section titled “Step 5: Submit an Evaluation Job”Upload the tokenizer directory as tokenizer/ beside this benchmark’s dataset bundle, then reference that benchmark-specific MinIO prefix with test_data_ref. With this conventional layout, parameters.tokenizer may be omitted or left blank; the adapter uses /test_data/tokenizer when it detects the staged dataset. Set parameters.tokenizer explicitly to a local path under /test_data only when you need a different tokenizer location:
curl -X POST https://<eval-hub-host>/api/v1/evaluations/jobs \ -H "Authorization: Bearer <token>" \ -H "X-Tenant: <namespace>" \ -H "Content-Type: application/json" \ -d '{ "name": "arc_easy evaluation", "model": { "url": "https://my-model.apps.example.com/v1", "name": "meta-llama/Llama-3.2-1B-Instruct" }, "benchmarks": [ { "id": "arc_easy", "provider_id": "lm_evaluation_harness", "test_data_ref": { "s3": { "bucket": "mlpipeline", "key": "disconnected/lm-evaluation-harness/arc-easy/", "secret_ref": "minio-credentials" } } } ] }'The init container syncs only the referenced benchmark prefix to /test_data/; the dataset bundle and tokenizer are then available at their expected paths. Once the layout is detected, the adapter sets its Hugging Face cache paths and offline variables automatically, including HF_HOME, HF_HUB_CACHE, HF_DATASETS_CACHE, HF_HUB_OFFLINE, HF_DATASETS_OFFLINE, HF_EVALUATE_OFFLINE, and TRANSFORMERS_OFFLINE.
Multiple benchmarks
Section titled “Multiple benchmarks”EvalHub runs one Kubernetes Job per benchmark. Give each benchmark its own test_data_ref and narrow MinIO prefix containing only that benchmark’s dataset/cache and tokenizer. If all benchmarks use the conventional staged tokenizer, omit parameters.tokenizer; set an explicit local path only for a benchmark that uses a different tokenizer:
{ "name": "offline evaluation", "model": { "url": "https://my-model.apps.example.com/v1", "name": "meta-llama/Llama-3.2-1B-Instruct" }, "benchmarks": [ { "id": "arc_easy", "provider_id": "lm_evaluation_harness", "test_data_ref": { "s3": { "bucket": "mlpipeline", "key": "disconnected/lm-evaluation-harness/arc-easy/", "secret_ref": "minio-credentials" } } }, { "id": "hellaswag", "provider_id": "lm_evaluation_harness", "test_data_ref": { "s3": { "bucket": "mlpipeline", "key": "disconnected/lm-evaluation-harness/hellaswag/", "secret_ref": "minio-credentials" } } } ]}Troubleshooting
Section titled “Troubleshooting”Init container fails before evaluation starts
The MinIO credentials Secret is missing or incorrect. Verify it exists in the job namespace and contains AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, and AWS_S3_ENDPOINT.
Evaluation fails with a network error
The dataset bundle was not found in MinIO or the slug does not match. The slug is case-sensitive — verify it matches exactly what was uploaded.
Wrong slug format
Use docs/dataset-mapping.md as a starting point, then verify dataset_path and dataset_name against the task configuration in the exact adapter revision deployed in your cluster. The linked mapping follows a moving branch and can differ from the downstream adapter. For crows_pairs_english in the RHOAI adapter revision linked above, the values are jannalu/crows_pairs_multilingual and english, producing the slug jannalu--crows_pairs_multilingual--english.
- With subset:
dataset_path.replace("/", "--") + "--" + dataset_name→allenai--ai2_arc--ARC-Easy - Without a subset configured by the deployed task:
dataset_path.replace("/", "--")only →hellaswag
Omit the subset suffix only when the deployed task configuration has no dataset subset. The linked mapping is a reference and may differ from the downstream adapter; do not use a - in that moving mapping to override the deployed task configuration. If the task config defines a subset (for example, arc_easy uses ARC-Easy), include it in the slug or the bundle will not be found.
Offline mode not triggered
Check that the init container populated /test_data with both the tokenizer files under /test_data/tokenizer and a dataset bundle beside it containing dataset_dict.json. If the tokenizer is stored elsewhere under /test_data, set parameters.tokenizer to that local path. A Hugging Face tokenizer ID is not a local staged path and does not enable offline mode.
Tokenizer not found
For the default tokenizer, verify that the uploaded prefix contains the tokenizer files in tokenizer/; they should appear in the pod at /test_data/tokenizer. If you set parameters.tokenizer, verify that it names the corresponding local path under /test_data.
LightEval
Section titled “LightEval”The disconnected LightEval adapter uses the Hugging Face cache and dataset cache staged by the selected benchmark’s test_data_ref. It also imports tinyBenchmarks.pkl eagerly while loading LightEval task definitions, so this file must be present even when the selected task does not use TinyBenchmarks.
Use a LightEval adapter image that provides ensure_nltk_offline_resources. The staging example below imports this function before loading task definitions. If the import fails, the image does not contain staged NLTK support; update the adapter image and repeat the staging step.
For each LightEval benchmark in the matrix, prepare a separate bundle using the same LightEval adapter image and task configuration as the deployed job. The image does not download NLTK data during its build, and a disconnected job does not download it at runtime. Every LightEval bundle must therefore include NLTK punkt and punkt_tab, even when the benchmark itself does not tokenize sentences. The task registry imports IFEval, which uses those resources. Preserve the cache layout produced by Hugging Face:
/test_data/ datasets/ # selected task's prepared dataset cache hub/ # selected task's Hub snapshots and metadata lighteval/ tinyBenchmarks.pkl # pinned registry asset nltk_data/ tokenizers/ punkt/ english.pickle # complete punkt package, not an empty directory punkt_tab/ english/ # complete English punkt_tab files tokenizer/ # optional model tokenizer; distinct from NLTK dataThe adapter sets its Hugging Face cache variables to the staged paths and disables Hub downloads. It also selects the staged NLTK path and validates it before importing the task registry, using only the staged files rather than an image-level fallback. It mirrors the read-only datasets cache metadata into a writable job cache; keep the staged source intact. Do not include caches for unrelated tasks in the benchmark prefix. The LCB v6, GPQA Diamond, and MATH-500 endpoint jobs in this matrix use the task-specific datasets, tinyBenchmarks.pkl, and NLTK data; they do not require a separate model tokenizer unless the selected model/task configuration uses one. GPQA Diamond may require accepted gated-repository terms and an authenticated token while staging. The token must not be placed in the bundle or the evaluation pod.
To prepare one task cache, use a connected environment with the same LightEval adapter image and task registry as the deployed job. Download NLTK data into the bundle before importing task definitions. Put the pinned tinyBenchmarks.pkl file at the package’s task-registry location before loading task definitions, then run the following from the adapter’s application directory. Repeat with a fresh bundle directory and the exact task ID for each LightEval benchmark:
TASK_NAME=gpqa:diamondBUNDLE="$PWD/staging/gpqa-diamond"mkdir -p "$BUNDLE/lighteval"curl -fsSL \ https://raw.githubusercontent.com/felipemaiapolo/tinyBenchmarks/e9a8b1031b0340571beb6c9ca3a27891be09a8fd/tinyBenchmarks/tinyBenchmarks.pkl \ -o "$BUNDLE/lighteval/tinyBenchmarks.pkl"
HF_HOME="$BUNDLE" \HF_HUB_CACHE="$BUNDLE/hub" \HF_DATASETS_CACHE="$BUNDLE/datasets" \NLTK_DATA="$BUNDLE/nltk_data" \TASK_NAME="$TASK_NAME" \python - <<'PY'import os
import nltkfrom _hf_offline import ensure_lighteval_registry_assets, ensure_nltk_offline_resources
root = os.environ["HF_HOME"]for resource in ("punkt", "punkt_tab"): nltk.download(resource, download_dir=os.environ["NLTK_DATA"], raise_on_error=True)ensure_nltk_offline_resources(root)ensure_lighteval_registry_assets(root)
from datasets import load_datasetfrom main import _task_configs
task_name = os.environ["TASK_NAME"]task = _task_configs().get(task_name)if task is None or not task.hf_repo: raise RuntimeError(f"No Hugging Face dataset is configured for {task_name}")kwargs = {"cache_dir": os.environ["HF_DATASETS_CACHE"]}if task.hf_subset: kwargs["name"] = task.hf_subsetif getattr(task, "hf_revision", None): kwargs["revision"] = task.hf_revisionif task_name == "gpqa:diamond": kwargs["token"] = os.environ["HF_TOKEN"]loaded = load_dataset(task.hf_repo, **kwargs)if not len(loaded): raise RuntimeError(f"The dataset for {task_name} has no splits")print(f"Staged {task_name}: {task.hf_repo}/{task.hf_subset or 'default'}")PYAuthenticate to gated Hugging Face repositories before this step. Do not print HF_TOKEN; keep it in the connected staging shell or a protected staging Secret, and never upload it with the bundle. Upload the resulting hub/, datasets/, lighteval/tinyBenchmarks.pkl, and complete nltk_data/tokenizers/punkt/ and punkt_tab/ directories under each LightEval benchmark’s prefix. Include tokenizer/ only if the model/task configuration requires a local model tokenizer. The adapter sets NLTK_DATA automatically when it finds the staged directory in /test_data.
Use the test_data_ref prefix from the matrix on each LightEval benchmark entry. For example:
{ "id": "gpqa:diamond", "provider_id": "lighteval", "test_data_ref": { "s3": { "bucket": "mlpipeline", "key": "disconnected/knowledge-reasoning-v1/gpqa-diamond/", "secret_ref": "minio-credentials" } }}tinyBenchmarks.pkl is an auxiliary framework asset, not a dataset that can be omitted based on the selected benchmark. Include the exact file required by the LightEval version in the deployed adapter image. If it is missing, LightEval may try to fetch it during import despite Hugging Face offline flags.
If punkt or punkt_tab is missing, empty, or incomplete, the adapter fails before loading the LightEval task registry with an error naming /test_data/nltk_data/tokenizers. Verify that punkt/english.pickle is non-empty and that punkt_tab/english/ contains the files from the same NLTK version used in the deployed image. An empty directory does not satisfy this check, even if NLTK data happens to exist elsewhere in the image. Re-stage the affected benchmark bundle and rerun its job; the disconnected evaluation pod will not attempt a download.
RULER’s staged bundle is mounted at /test_data. Each benchmark needs its own prefix and its own test_data_ref; the adapter copies validated input files into a writable per-job cache. The curated Document Understanding v1 tasks use these inputs:
| Benchmark | Required files at bundle root |
|---|---|
common-words-extraction | english_words.json |
frequency-words-extraction | No task dataset file |
qa-squad | squad.json |
qa-hotpotqa | hotpotqa.json |
niah-single-essay | PaulGrahamEssays.json |
variable-tracking | No task dataset file for the curated defaults |
Every RULER benchmark also needs a complete local Hugging Face tokenizer under /test_data/tokenizer/, including all files required by the tokenizer and its exact revision. If parameters.tokenizer_path is omitted or equals the evaluated model ID, the adapter uses that directory; otherwise set it to a local path and use tokenizer_type: "hf". The OpenAI/tiktoken tokenizer mode is not supported for disconnected RULER jobs. The RULER image includes its NLTK sentence tokenizer data, so the pod does not need to download those resources.
For example, a SQuAD benchmark entry can use:
{ "id": "qa-squad", "provider_id": "ruler", "parameters": { "tokenizer_type": "hf", "tokenizer_path": "/test_data/tokenizer" }, "test_data_ref": { "s3": { "bucket": "mlpipeline", "key": "disconnected/document-understanding-v1/qa-squad/", "secret_ref": "minio-credentials" } }}The root of that S3 prefix must contain squad.json and tokenizer/. For a benchmark with no task dataset file, stage only its tokenizer. For essay-backed tasks, stage the non-empty PaulGrahamEssays.json file as well. The adapter validates required files and checksums and fails before inference if an asset is missing or invalid; it does not fall back to downloading data in disconnected mode.
Inspect AI: Hugging Face datasets and CyberSecEval 2
Section titled “Inspect AI: Hugging Face datasets and CyberSecEval 2”Inspect HLE and DocVQA use Hugging Face-backed datasets; CyberSecEval 2 Prompt Injection uses a pinned JSON file fetched from GitHub in connected mode. For offline runs, the corresponding Inspect adapter image must include the staged-cache support and the benchmark’s own test_data_ref.
Stage the pinned cais/hle dataset revision using an authenticated Hugging Face account that has access to the gated repository. Keep the generated Hugging Face cache layout together (including datasets/, hub/, and any referenced Xet files) under the benchmark’s S3 prefix. Access credentials are needed only during connected staging. The evaluation job still needs a reachable target and a reachable judge. Configure the judge model with the benchmark’s judge_model parameter and put any judge endpoint/key settings in the Secret referenced by model.auth.secret_ref; see Judge and scorer models.
DocVQA
Section titled “DocVQA”Stage the pinned lmms-lab-encoder/DocVQA DocVQA configuration and its validation split. Preserve the complete cache, including the image bytes/blobs referenced by the dataset records; a metadata-only dataset snapshot is insufficient. Keep the original Hugging Face cache layout under the benchmark prefix and include any Xet-backed objects created by the staging client. The target endpoint must be reachable inside the cluster and must accept image inputs.
CyberSecEval 2 Prompt Injection
Section titled “CyberSecEval 2 Prompt Injection”The adapter expects the pinned prompt-injection dataset at:
/test_data/cyberseceval_2/prompt_injection_9b05e90e7a57333d006555420f7289cb.jsonPlace that file at the root-relative path shown above under the S3 prefix for inspect/cyberseceval-2-pi. The evaluation job also calls a judge model. Set parameters.judge_model; configure a separate judge endpoint or credential through the Secret referenced by model.auth.secret_ref when needed. The sidecar routes the model requests, so both target and judge services must be reachable from the job through their configured routes. See Judge and scorer models.
Inspect AI: BFCL
Section titled “Inspect AI: BFCL”BFCL (inspect/bfcl) downloads its dataset and, for some categories, function implementations and documentation from GitHub. The Hugging Face dataset and tokenizer layout in the previous section does not stage these files. For other Inspect benchmarks that use Hugging Face datasets, follow the Inspect adapter’s Hugging Face asset instructions.
Stage BFCL assets
Section titled “Stage BFCL assets”Prepare the cache on a connected staging machine using the same Inspect package versions as the deployed adapter. The current adapter pins inspect-ai==0.3.278 and inspect-evals==0.26.0; verify the constraints.txt for the exact adapter image you will run, because these versions can change. For the current pins, create an isolated environment and install the matching packages:
python3 -m venv .venv-inspect-stagingsource .venv-inspect-staging/bin/activatepython -m pip install "inspect-ai==0.3.278" "inspect-evals==0.26.0"Set INSPECT_EVALS_CACHE_DIR to an absolute writable path before starting Inspect; the package reads this variable at import time. Run the BFCL task with the same categories that the EvalHub job will use. If the job omits task_args.categories, use the task’s defaults when warming the cache. For example:
export INSPECT_EVALS_CACHE_DIR="$PWD/staging/inspect-evals"inspect eval inspect_evals/bfcl --model mockllm/model --limit 1
# Tool Use v1 has two inspect/bfcl entries; warm the union of their 13 categories:inspect eval inspect_evals/bfcl --model mockllm/model --limit 1 -T "categories=['simple_python', 'simple_java', 'simple_javascript', 'multiple', 'parallel', 'parallel_multiple', 'live_simple', 'live_multiple', 'live_parallel', 'live_parallel_multiple', 'irrelevance', 'live_irrelevance', 'live_relevance']"--limit 1 limits the samples evaluated by the warm-up run; it does not limit the BFCL cache download. BFCL downloads its data and loads records for the requested categories during task setup, before Inspect applies the evaluation limit. Still stage and verify all assets required by every selected category.
The resulting cache must contain the complete BFCL/ tree required by the selected categories. Always preserve the simple_python data file, even when simple_python is not selected: Inspect-Evals uses it as a download sentinel, and omitting it causes a runtime GitHub download attempt. Also include every selected category’s question file and applicable ground-truth file. Multi-turn and agentic categories need their corresponding files under func_source_code/ and multi_turn_func_doc/. Copy the entire BFCL/ directory rather than selecting files manually; the required paths are determined by the category manifest in the pinned inspect-evals release.
For the MinIO workflow above, upload that directory under a prefix so the pod receives it at /test_data/BFCL:
aws --endpoint-url http://127.0.0.1:9000 s3 sync "$INSPECT_EVALS_CACHE_DIR/BFCL" s3://mlpipeline/disconnected/tool-use-v1/bfcl/BFCLUse that prefix in the BFCL benchmark’s test_data_ref; create the MinIO Secret as described above. For example, when selecting categories explicitly:
{ "id": "inspect/bfcl", "provider_id": "inspect", "parameters": { "num_examples": 1, "task_args": { "categories": ["simple_python", "multiple"] } }, "test_data_ref": { "s3": { "bucket": "mlpipeline", "key": "disconnected/tool-use-v1/bfcl/", "secret_ref": "minio-credentials" } }}For a curated Tool Use v1 job, attach test_data_ref to both repeated inspect/bfcl entries in the collection override. The Tool Use v1 collection config supplies two category lists (10 call-generation categories and 3 relevance categories); these overrides only point both entries at the staged cache:
{ "name": "Tool Use v1 offline evaluation", "model": { "url": "https://my-model.apps.example.com/v1", "name": "my-model" }, "collection": { "id": "tool-use-v1", "benchmarks": [ { "id": "inspect/bfcl", "provider_id": "inspect", "test_data_ref": { "s3": { "bucket": "mlpipeline", "key": "disconnected/tool-use-v1/bfcl/", "secret_ref": "minio-credentials" } } }, { "id": "inspect/bfcl", "provider_id": "inspect", "test_data_ref": { "s3": { "bucket": "mlpipeline", "key": "disconnected/tool-use-v1/bfcl/", "secret_ref": "minio-credentials" } } } ] }}The adapter checks the staged data and runtime files needed by the selected categories. When they are complete, it points INSPECT_EVALS_CACHE_DIR to /test_data and skips the GitHub preflight. If required files are missing, the staged cache is not accepted; with the default preflight: fail, the job fails fast when GitHub is unreachable. BFCL categories with additional external-service or snapshot requirements need those dependencies staged or reachable separately; staging the BFCL data tree alone does not make every optional category air-gap compatible. See the Inspect-Evals BFCL documentation and the Inspect adapter’s disconnected-run notes.