EvalHub

Open source evaluation orchestration platform for Large Language Models.

EvalHub provides a unified way to evaluate LLMs across multiple frameworks — submit evaluations via CLI, Python SDK, or REST API and let EvalHub handle orchestration, tracking, and artifact storage. It runs locally for development and scales on Kubernetes for production.

Key Features

Multi-framework

Evaluate with LightEval, GuideLLM, lm-eval-harness, Garak, MTEB, IBM CLEAR, DeepEval, Inspect AI, RAGAS, RULER, SWE-bench, WildGuard, or bring your own framework.

Kubernetes-native

Each evaluation runs as an isolated Kubernetes Job with automatic lifecycle management.

Benchmark Collections

Group benchmarks with weighted scoring for domain-specific evaluations in a single API call.

MLflow Integration

Track experiments, compare runs, and persist metrics to MLflow automatically.

• Versioned REST API (v1) with OpenAPI specification
• Provider registry with benchmark discovery
• OCI artifact persistence for evaluation results
• MLflow experiment tracking for jobs and adapters
• Prometheus metrics and OpenTelemetry tracing, metrics, and logs
• Tenancy with multi- and single-namespace deployment modes and Kubernetes RBAC

How It Works

EvalHub consists of three components:

• Server — Go REST API that orchestrates evaluation jobs, manages providers, and stores results (SQLite or PostgreSQL)
• SDK — Python client library, CLI, and adapter framework for building integrations
• Contrib — Community-contributed framework adapters packaged as container images

There are some animatated diagrams that show the flow of data through the system.

• Job Workflow - shows the flow of data through the system when an evaluation job is run.
• MCP EDD Workflow - shows the flow of data through the system when an MCP EDD job is run.

Available Adapters

Community providers with a provider.yaml are listed in the Provider Catalog — filter, inspect benchmarks, and download Kubernetes ConfigMaps.

Adapter	Provider	Support Tier	What it measures
lm-eval-harness	lm_evaluation_harness	Core	167 benchmarks across 12 categories (MMLU, HellaSwag, GSM8K, …)
LightEval	lighteval	Core	Language model benchmarks: accuracy, exact match
Garak	garak	Core	Safety and vulnerability scanning (OWASP Top 10)
GuideLLM	guidellm	Validated	Inference performance: TTFT, ITL, throughput, latency
IBM CLEAR	ibm-clear	Validated	Agentic trace evaluation: LLM-as-judge failure pattern analysis on JSON traces
DeepEval	deepeval	Community	LLM-as-judge evaluation: faithfulness, relevancy, hallucination, correctness
Inspect AI	inspect	Community	79 benchmarks across agent, alignment, coding, cybersecurity
MTEB	mteb	Community	Embedding evaluation: STS, retrieval, classification
RAGAS	ragas	Community	RAG evaluation: context precision, answer relevancy
RULER	ruler	Community	Long-context evaluation: needle-in-a-haystack, variable tracking, aggregation
SWE-bench	swebench	Community	Code generation: real-world GitHub issue resolution
WildGuard	wildguard	Community	Safety: prompt harmfulness classification and refusal detection

Quick Taste

1. Install the SDK (includes the server and CLI):

   pip install "eval-hub-sdk[server,cli]"

2. Start a model server (e.g. Ollama):

   ollama run qwen2.5:1.5b

   Keep Ollama running in this terminal and execute Steps 3–4 in a separate terminal.

3. Start EvalHub locally:

   evalhub server start

   Point the CLI at the local server:

   evalhub config set base_url http://localhost:8080

   For provider registration and other advanced options (e.g. server_config_file), see Local Mode.

4. Run an evaluation:

   evalhub eval run \
     --name my-first-eval \
     --model-url http://localhost:11434/v1 \
     --model-name qwen2.5:1.5b \
     --provider lm_evaluation_harness \
     --benchmark mmlu \
     --wait

See the Quick Start for a full walkthrough or Local Mode for advanced local configuration (MLflow, custom server config, provider registration).

Get Started

Installation Install the server and a client

Quick Start Run your first evaluation step by step

CLI Reference Full command reference for the evalhub CLI

MLflow Configure experiment tracking for jobs and adapters

Python SDK Programmatic access with sync and async clients

MCP support Exposing EvalHub as a MCP server

Agent Discoverability How AI agents choose and interpret evaluations

License

Apache 2.0 — see LICENSE.