Skip to content
EvalHub mascot EvalHub
Docs Blog

Local Mode Tutorial

since 0.4.3 eval-hub-sdk

End-to-end walkthrough: run a LightEval evaluation locally with EvalHub, MLflow experiment tracking, and OCI artifact storage — no Kubernetes required.

This tutorial uses the LightEval adapter from eval-hub-contrib. For background on how local mode works, see the Local Mode guide.

Prerequisites

Section titled “Prerequisites”

Install the following tools before starting:

  • uv — Python package manager
  • podman (or Docker) — for running the OCI registry
  • ollama (or any OpenAI-compatible LLM server) — for serving a local model

Setup

Section titled “Setup”

  1. Create the project directory and virtual environment

    Terminal window
    mkdir local-lighteval && cd local-lighteval
    uv venv --python 3.12

  2. Download the LightEval adapter

    Download the adapter driver and its requirements from eval-hub-contrib:

    Terminal window
    curl -o main.py https://raw.githubusercontent.com/eval-hub/eval-hub-contrib/main/adapters/lighteval/main.py
    curl -o requirements.txt https://raw.githubusercontent.com/eval-hub/eval-hub-contrib/main/adapters/lighteval/requirements.txt

  3. Install dependencies

    Install the EvalHub server, CLI, and adapter requirements:

    Terminal window
    uv pip install -r requirements.txt
    uv pip install "eval-hub-sdk[server,cli]>=0.4.3"

  4. Start MLflow (optional)

    Only needed if you want experiment tracking. Skip this step if you just want to run evaluations.

    Install MLflow and start the server:

    Terminal window
    uv pip install "mlflow>=3.11"
    Terminal window
    source .venv/bin/activate
    

    mlflow server \
      --backend-store-uri sqlite:///mlflow.db \
      --host localhost \
      --port 5000

    Verify from another terminal:

    Terminal window
    curl http://localhost:5000/health

    The MLflow UI dashboard is accessible at http://localhost:5000.

  5. Start the OCI registry (optional)

    Only needed if you want to persist evaluation artifacts to an OCI registry. Skip this step if you just want to run evaluations.

    In another terminal, pull the registry image and start it on localhost:5001:

    Terminal window
    podman pull docker.io/library/registry:2
    

    podman run -d -p 5001:5000 \
        --name eval-hub-oci-registry \
        -e REGISTRY_STORAGE_DELETE_ENABLED=true \
        docker.io/library/registry:2

  6. Start the LLM server

    Pull a model:

    Terminal window
    ollama pull llama3.2:3b-instruct-q4_K_M

    Verify it’s running:

    Terminal window
    curl -s http://localhost:11434/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
        "model": "llama3.2:3b-instruct-q4_K_M",
        "messages": [{"role": "user", "content": "Why is the sky blue?"}],
        "max_tokens": 100
      }'

  7. Start the EvalHub server

    Create a server configuration file my-config.yaml:

    service:
      port: 8080
    

    database:
      driver: sqlite
      url: file::eval_hub:?mode=memory&cache=shared
    

    mlflow:
      tracking_uri: http://localhost:5000

    If you skipped the MLflow step, omit the mlflow section.

    Set the configuration and start the server:

    Terminal window
    evalhub config set server_config_file my-config.yaml
    evalhub server start
    Server started (PID 12345).
      URL:  http://localhost:8080
      Logs: ~/.config/evalhub/server/server.log

    Verify the server is healthy:

    Terminal window
    evalhub config set base_url http://localhost:8080
    evalhub health
    EvalHub service: healthy (2ms)

  8. Register the provider

    Create a provider definition file lighteval.yaml:

    name: lighteval
    description: LightEval adapter for evaluation framework
    runtime:
      local:
        command: "python main.py"
        env:
          - name: MLFLOW_TRACKING_URI
            value: http://localhost:5000
          - name: OCI_INSECURE
            value: "true"
    benchmarks:
      - id: gsm8k
        name: Grade-school math word problems
        description: |-
          Multi-step arithmetic word problems requiring 2-8 reasoning steps
          (8-shot, 1,319 examples).
        category: math
        metrics:
          - exact_match
          - acc
        num_few_shot: 8
        dataset_size: 1319
        tags:
          - math
          - reasoning
          - lighteval
        primary_score:
          metric: acc
          lower_is_better: false
        pass_criteria:
          threshold: 0.25

    Register it with the CLI:

    Terminal window
    evalhub providers create --file lighteval.yaml
    Provider created: a578921e-053a-496a-889b-ecd97b8dbd1a

    Save the provider ID for subsequent commands (your ID will differ):

    Terminal window
    PROVIDER_ID="a578921e-053a-496a-889b-ecd97b8dbd1a"

    See the Local Mode guide — Provider Configuration for details on the runtime.local section.

    Verify the provider was registered:

    Terminal window
    evalhub providers list
    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃ ID                                   ┃ NAME      ┃ DESCRIPTION                                ┃ BENCHMARKS ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    │ a578921e-053a-496a-889b-ecd97b8dbd1a │ lighteval │ LightEval adapter for evaluation framework │ 1          │
    └──────────────────────────────────────┴───────────┴────────────────────────────────────────────┴────────────┘

    Query the provider’s available benchmarks:

    Terminal window
    evalhub providers describe $PROVIDER_ID
    Provider: lighteval
    ID:       a578921e-053a-496a-889b-ecd97b8dbd1a
    Description: LightEval adapter for evaluation framework
    

    Benchmarks (1):
    ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
    ┃ ID    ┃ NAME                            ┃ CATEGORY ┃ METRICS          ┃
    ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
    │ gsm8k │ Grade-school math word problems │ math     │ exact_match, acc │
    └───────┴─────────────────────────────────┴──────────┴──────────────────┘

    Save the benchmark ID for subsequent commands:

    Terminal window
    BENCHMARK_ID="gsm8k"

Run an evaluation

Section titled “Run an evaluation”

With all services running, submit a job using the CLI:

Terminal window
evalhub eval run \
  --name my-eval-job \
  --model-url http://localhost:11434/v1 \
  --model-name llama3.2:3b-instruct-q4_K_M \
  --provider $PROVIDER_ID \
  --benchmark $BENCHMARK_ID \
  --param num_examples=10 \
  --param num_few_shot=0 \
  --wait

Save the job ID from the output and check results:

Terminal window
JOB_ID="<job-id-from-output>"
evalhub eval results $JOB_ID
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ BENCHMARK ┃ PROVIDER                             ┃ METRIC                 ┃ VALUE ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ gsm8k     │ a578921e-053a-496a-889b-ecd97b8dbd1a │ all.extractive_match   │ 0.7   │
│ gsm8k     │ a578921e-053a-496a-889b-ecd97b8dbd1a │ gsm8k.extractive_match │ 0.7   │
└───────────┴──────────────────────────────────────┴────────────────────────┴───────┘

With MLflow experiments and OCI artifact storage

Section titled “With MLflow experiments and OCI artifact storage”

MLflow experiment tracking and OCI artifact export are optional. If you started MLflow and the OCI registry in the setup steps, add the --experiment and --oci-* flags:

Terminal window
evalhub eval run \
  --name my-eval-job \
  --model-url http://localhost:11434/v1 \
  --model-name llama3.2:3b-instruct-q4_K_M \
  --provider $PROVIDER_ID \
  --benchmark $BENCHMARK_ID \
  --param num_examples=10 \
  --param num_few_shot=0 \
  --experiment my-local-experiment \
  --oci-host localhost:5001 \
  --oci-repository local-eval-results \
  --wait
Job submitted: ef42ad16-57b6-4bf4-a7ff-eb907e848bb1
Waiting for job ef42ad16-57b6-4bf4-a7ff-eb907e848bb1 to complete...
Job ef42ad16-57b6-4bf4-a7ff-eb907e848bb1 finished with state: completed

Save the job ID and check results:

Terminal window
JOB_ID="ef42ad16-57b6-4bf4-a7ff-eb907e848bb1"
evalhub eval results $JOB_ID --format json

Example output:

[
  {
    "id": "gsm8k",
    "provider_id": "a578921e-053a-496a-889b-ecd97b8dbd1a",
    "benchmark_index": 0,
    "metrics": {
      "all.extractive_match": 0.7,
      "gsm8k.extractive_match": 0.7
    },
    "artifacts": {
      "evalhub.env_card": {
        "aggregate_results": {},
        "autograder_bias": {},
        "capture_completeness": 0.15,
        "confidence_intervals": {},
        "cpu_model": "arm",
        "custom": {},
        "k8s_pod_labels": {},
        "k8s_resource_limits": {},
        "key_packages": {
          "mlflow": "3.14.0",
          "torch": "2.13.0",
          "transformers": "5.13.0"
        },
        "os_info": "macOS-26.5.2-arm64-arm-64bit",
        "per_task_results": {},
        "python_version": "3.12.12",
        "scorer_ids": []
      },
      "oci_digest": "sha256:5f853d7fa2c67247ca35b511deaec78a56d64c98734974e274504c039bedc558",
      "oci_reference": "localhost:5001/local-eval-results:evalhub-9129f2cac2db213158f8c106e1b658e0e05ecd9db54630dfe6bc2edf2cdb413b@sha256:5f853d7fa2c67247ca35b511deaec78a56d64c98734974e274504c039bedc558"
    },
    "mlflow_run_id": "3c03bb8128a349fe822e793637b7ffe2",
    "logs_path": null
  }
]

Using the Python SDK

Section titled “Using the Python SDK”

The included evalhub-client.ipynb notebook demonstrates the full evaluation lifecycle using the eval-hub-sdk Python client — submitting jobs, polling status, and retrieving results programmatically.

What’s running

Section titled “What’s running”

After completing setup, you have four services on localhost:

ServiceURLPurpose
EvalHub server (evalhub server)http://localhost:8080Evaluation orchestration
MLflowhttp://localhost:5000Experiment tracking dashboard
OCI registryhttp://localhost:5001Artifact storage
Ollamahttp://localhost:11434LLM inference

Next steps

Section titled “Next steps”
  • Browse the MLflow UI at http://localhost:5000 to see experiment metrics
  • Read the MLflow guide for experiment schema, adapter backends, and troubleshooting
  • Read the Local Mode guide for provider configuration details and troubleshooting