Using Custom Data

Some benchmarks require external test datasets. EvalHub can load this data from S3-compatible storage (AWS S3, MinIO, or any S3 API-compatible service) before the evaluation job runs. Data is downloaded by an init container and mounted at /test_data for the adapter.

This guide gives a minimal, reproducible setup using MinIO in Kubernetes.

For air-gapped clusters running lm-evaluation-harness with pre-staged HuggingFace datasets and tokenizers, see Disconnected Cluster Evaluation.

How it works

1. You configure a benchmark with a test data reference (bucket, key prefix, and a Kubernetes Secret name for credentials).
2. When the evaluation job starts, an init container runs first: it uses the secret to connect to S3/MinIO, lists objects under the given key prefix, and downloads them to a shared volume.
3. The adapter container runs with /test_data populated; the benchmark can read files from that path.

The init container expects credentials from a Kubernetes Secret mounted at /var/run/secrets/test-data with these keys (as filenames):

Key	Description
AWS_ACCESS_KEY_ID	S3 access key
AWS_SECRET_ACCESS_KEY	S3 secret key
AWS_DEFAULT_REGION	Region (e.g. us-east-1; required even for MinIO)
AWS_S3_ENDPOINT	Full endpoint URL (e.g. http://minio:9000)

Test data reference types

test_data_ref.type tells EvalHub how to interpret the staged data. It defaults to data_set when omitted.

Type	Description
data_set	(default) Input dataset for a benchmark. EvalHub calls the model at model.url to generate responses, then scores them.
pre_recorded_data	Previously captured model/agent responses. EvalHub does not call a live model — the adapter reads the recorded responses and scores them directly.

Pre-recorded data

Pre-recorded data lets you evaluate responses that were already generated, instead of producing fresh responses at evaluation time. Rather than EvalHub calling model.url to run inference, the adapter reads recorded responses (for example, captured agent interaction traces) from your test data reference and scores them.

This is useful for:

• Evaluating captured production or agent traces after the fact.
• Reproducing results deterministically from a fixed set of responses.
• Running offline / air-gapped, where a live model endpoint is unavailable.

Because the responses are already recorded, model.url is not required when every benchmark in the job uses pre_recorded_data. The model section is still required — model.name identifies the source of the recorded responses. If any benchmark in the job uses data_set, model.url remains required.

Provider support

pre_recorded_data is currently supported and tested only with the ibm-clear provider (benchmark agentic-evaluation) for agentic evaluation. Other adapters do not support pre_recorded_data yet.

Set type on the benchmark’s test_data_ref alongside the storage reference (S3, PVC, or Git). Because the responses are pre-recorded, model.url can be left empty:

Upload a CLEAR-compatible trace as a .json file, such as this sample trace, at the S3 key below.

"model": {
  "name": "my-agent-traces",
  "url": ""
},
"benchmarks": [
  {
    "test_data_ref": {
      "type": "pre_recorded_data",
      "s3": {
        "bucket": "my-eval-data",
        "key": "eval-hub/tr-236907d3019ea2a5f8e81baeb258753b.json",
        "secret_ref": "my-s3-secret"
      }
    }
  }
]

Prerequisites

• Kubernetes cluster with EvalHub installed (see Installation).
• kubectl configured to use that cluster.

Set up

1. Deploy MinIO

   Create a namespace and deploy MinIO (API on port 9000, console on 9001):

   minio.yaml

   apiVersion: v1
   kind: Namespace
   metadata:
     name: evalhub-demo
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: minio
     namespace: evalhub-demo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: minio
     template:
       metadata:
         labels:
           app: minio
       spec:
         containers:
           - name: minio
             image: quay.io/minio/minio:latest
             args:
               - server
               - /data
               - --console-address
               - ":9001"
             env:
               - name: MINIO_ROOT_USER
                 value: "minioadmin"
               - name: MINIO_ROOT_PASSWORD
                 value: "minioadmin"
             ports:
               - containerPort: 9000
                 name: api
               - containerPort: 9001
                 name: console
             volumeMounts:
               - name: data
                 mountPath: /data
             readinessProbe:
               httpGet:
                 path: /minio/health/ready
                 port: 9000
               initialDelaySeconds: 5
               periodSeconds: 10
         volumes:
           - name: data
             emptyDir: {}
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: minio
     namespace: evalhub-demo
   spec:
     selector:
       app: minio
     ports:
       - name: api
         port: 9000
         targetPort: 9000
       - name: console
         port: 9001
         targetPort: 9001

   kubectl apply -f minio.yaml
   kubectl wait -n evalhub-demo deployment/minio --for=condition=Available --timeout=120s

2. Create bucket and upload test data

   Use the MinIO client or the MinIO web console (port 9001) to create a bucket and upload files. Example using a temporary pod:

   # Create bucket and upload a sample file from a one-off pod
   kubectl run -n evalhub-demo mc --rm -i --restart=Never --image=quay.io/minio/mc:latest -- \
     sh -c "
     mc alias set myminio http://minio.evalhub-demo.svc.cluster.local:9000 minioadmin minioadmin
     mc mb myminio/evalhub-test --ignore-existing
     echo '{\"question\": \"What is 2+2?\", \"answer\": \"4\"}' | mc pipe myminio/evalhub-test/dataset/sample.json
     mc ls myminio/evalhub-test/dataset/
     "

   This creates bucket evalhub-test and key prefix dataset/ with one file. Adjust bucket name and prefix to match your benchmark.

3. Create the credentials Secret

   Create a Secret in the same namespace where EvalHub runs evaluation jobs. The init container reads these keys from the mounted secret volume:

   evalhub-s3-credentials.yaml

   apiVersion: v1
   kind: Secret
   metadata:
     name: evalhub-s3-credentials
     namespace: evalhub-demo   # use the namespace where EvalHub creates jobs
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "minioadmin"
     AWS_SECRET_ACCESS_KEY: "minioadmin"
     AWS_DEFAULT_REGION: "us-east-1"
     AWS_S3_ENDPOINT: "http://minio.evalhub-demo.svc.cluster.local:9000"

   Use the in-cluster MinIO service DNS name so the job pod can reach MinIO. If EvalHub runs in a different namespace, create this secret in that namespace and set AWS_S3_ENDPOINT to http://minio.evalhub-demo.svc.cluster.local:9000 (or the correct service FQDN).

   kubectl apply -f evalhub-s3-credentials.yaml

4. Submit a job with S3 test data

   Attach the test data reference to the benchmark via test_data_ref.s3: bucket, key (prefix), and secret_ref (Secret name).

   CLI

   evalhub eval run \
     --name s3-data-eval \
     --model-url http://your-model:8000/v1 \
     --model-name my-model \
     --provider your_provider_id \
     --benchmark your_benchmark_id \
     --test-data-s3-bucket evalhub-test \
     --test-data-s3-key dataset/ \
     --test-data-s3-secret evalhub-s3-credentials

   Python SDK

   import os
   from evalhub import SyncEvalHubClient
   from evalhub.models.api import (
       ModelConfig, BenchmarkConfig, JobSubmissionRequest, TestDataRef, S3TestDataRef
   )
   
   client = SyncEvalHubClient(base_url=os.environ["EVALHUB_URL"])
   
   job = client.jobs.submit(JobSubmissionRequest(
       name="s3-data-eval",
       model=ModelConfig(url="http://your-model:8000/v1", name="my-model"),
       benchmarks=[
           BenchmarkConfig(
               id="your_benchmark_id",
               provider_id="your_provider_id",
               test_data_ref=TestDataRef(
                   s3=S3TestDataRef(
                       bucket="evalhub-test",
                       key="dataset/",
                       secret_ref="evalhub-s3-credentials",
                   )
               ),
           )
       ],
   ))
   print(f"Job submitted: {job.id}")

   REST API

   curl -s -X POST $EVALHUB_URL/api/v1/evaluations/jobs \
     -H "Content-Type: application/json" \
     -d '{
       "model": {
         "url": "http://your-model:8000/v1",
         "name": "my-model"
       },
       "benchmarks": [
         {
           "id": "your_benchmark_id",
           "provider_id": "your_provider_id",
           "test_data_ref": {
             "s3": {
               "bucket": "evalhub-test",
               "key": "dataset/",
               "secret_ref": "evalhub-s3-credentials"
             }
           }
         }
       ]
     }'

   • bucket: MinIO bucket name (e.g. evalhub-test).
   • key: Object key prefix; all objects under this prefix are downloaded (e.g. dataset/ or dataset).
   • secret_ref: Name of the Kubernetes Secret that holds the four keys above. The secret must exist in the namespace where the job is created.

   The init container downloads every object under s3://bucket/key into /test_data, preserving relative paths. The adapter can then read files from /test_data.

Troubleshooting

• Init container fails (list or get object) Check that the job pod runs in a namespace where it can resolve the MinIO service (e.g. minio.evalhub-demo.svc.cluster.local) and that AWS_S3_ENDPOINT uses that URL. For MinIO, use http:// (or https:// if TLS is configured).

• No objects found Ensure the bucket and key prefix exist and contain at least one object (the init container requires at least one file under the prefix).

• Secret not found Create the Secret in the same namespace where EvalHub creates evaluation jobs, and use that exact name in secret_ref.