Installation

Install the EvalHub server and a client to interact with it.

Prerequisites

• Python 3.11+ (client SDK and CLI)
• Go 1.22+ (building the server from source)
• Kubernetes/OpenShift cluster (for production deployment)
• kubectl or oc CLI configured for your cluster

Install a Client

CLI

pip install "eval-hub-sdk[cli]"

Verify:

evalhub version

Python SDK

pip install "eval-hub-sdk[client]"

REST API

No installation needed — use curl or any HTTP client directly against the REST API.

Deploy the Server

Kubernetes/OpenShift (SQLite)

Install the TrustyAI Operator and deploy EvalHub with SQLite. No external database needed — suitable for development and testing. Data is lost when the pod restarts.

kubectl apply -f https://github.com/trustyai-explainability/trustyai-service-operator/releases/latest/download/trustyai-operator.yaml

kubectl apply -f - <<EOF
apiVersion: trustyai.opendatahub.io/v1alpha1
kind: EvalHub
metadata:
  name: evalhub
  namespace: evalhub
spec:
  replicas: 1
  database:
    type: sqlite
  providers:
    - lm-evaluation-harness
    - garak
    - garak-kfp
    - guidellm
    - lighteval
    - ibm-clear
  collections:
    - leaderboard-v2
    - safety-and-fairness-v1
    - toxicity-and-ethical-principles
EOF

Kubernetes/OpenShift (PostgreSQL)

Install the TrustyAI Operator and deploy EvalHub with PostgreSQL for durable, persistent storage. Create the credentials Secret first, then the CR.

kubectl apply -f https://github.com/trustyai-explainability/trustyai-service-operator/releases/latest/download/trustyai-operator.yaml

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: evalhub-db-credentials
  namespace: evalhub
type: Opaque
stringData:
  db-url: "postgres://user:password@db-host:5432/evalhub"
EOF

kubectl apply -f - <<EOF
apiVersion: trustyai.opendatahub.io/v1alpha1
kind: EvalHub
metadata:
  name: evalhub
  namespace: evalhub
spec:
  replicas: 1
  database:
    type: postgresql
    secret: evalhub-db-credentials
  providers:
    - lm-evaluation-harness
    - garak
    - garak-kfp
    - guidellm
    - lighteval
    - ibm-clear
  collections:
    - leaderboard-v2
    - safety-and-fairness-v1
    - toxicity-and-ethical-principles
EOF

See OpenShift Setup for a full description of all spec fields.

Local Development

Build and run from source. Requires Go 1.22+ and Make.

git clone https://github.com/eval-hub/eval-hub.git
cd eval-hub
make build
./bin/eval-hub --local

The --local flag disables authentication and enables CORS, which is required for local development — without it, all endpoints except /api/v1/health return 401 Unauthorized.

The server starts at http://localhost:8080 with SQLite in-memory storage.

Custom port: If port 8080 is already in use, set the PORT environment variable:

PORT=8090 ./bin/eval-hub --local

Persistent storage: By default the server uses in-memory SQLite and all data is lost when the process stops. To persist data across restarts, point DB_URL at a file:

DB_URL="file:./data/eval-hub.db?cache=shared" ./bin/eval-hub --local

Verify the Installation

For local development servers, first point the CLI at your server:

evalhub config set base_url http://localhost:8080

CLI

evalhub health
evalhub providers list

Python SDK

import os
from evalhub import SyncEvalHubClient

client = SyncEvalHubClient(base_url=os.environ["EVALHUB_URL"])
print(client.health())
print(client.providers.list())

REST API

curl -s $EVALHUB_URL/api/v1/health
curl -s $EVALHUB_URL/api/v1/evaluations/providers | jq .