Skip to content

OpenShift Development Setup

Complete guide for deploying and developing EvalHub on OpenShift with OpenDataHub.

  • OpenShift CLI (oc) - Version 4.12+
  • kubectl - Compatible with your OpenShift version
  • jq - JSON processor for manifest manipulation
  • Git - For cloning repositories
  • Podman or Docker - For building custom images

Ensure you have access to an OpenShift cluster:

Terminal window
# Login to OpenShift
oc login --server=https://api.your-cluster.example.com:6443
# Verify access
oc whoami
oc cluster-info

OpenDataHub provides the foundation for deploying EvalHub on OpenShift.

  1. Install OpenDataHub Operator

    Install the OpenDataHub Operator (version 3.3 or higher) from OperatorHub in the OpenShift web console:

    1. Navigate to Operators → OperatorHub
    2. Search for “OpenDataHub”
    3. Click Install
    4. Select fast channel
    5. Choose All namespaces on the cluster installation mode
    6. Click Install

    Wait for the operator installation to complete:

    Terminal window
    # Check operator installation
    oc get csv -n openshift-operators | grep opendatahub
    # Should show PHASE: "Succeeded"
  2. Configure OpenDataHub

    After the operator is installed, configure OpenDataHub from the operator’s dashboard:

    1. Navigate to Operators → Installed Operators
    2. Select OpenDataHub Operator
    3. Go to the DSCInitialization tab
    4. Click Create DSCInitialization
    5. Review or just use default settings
    6. Click Create
    7. Await “Phase: Ready”
    8. Go to the Data Science Cluster tab
    9. Click Create DataScienceCluster
    10. Review or just use default settings for most components, do not save/create yet
    11. Important: Under the TrustyAI component, configure LMEval security settings:
      • Set permitCodeExecution to allow
      • Set permitOnline to allow
    12. Click Create

    Verify the DataScienceCluster is ready:

    Terminal window
    # Check DSC status
    oc get datasciencecluster default-dsc -o jsonpath='{.status.phase}'
    # Should output: "Ready"
    # List deployed components
    oc get pods -n opendatahub

The TrustyAI Operator manages EvalHub custom resources.

The TrustyAI Operator is installed automatically as part of the DataScienceCluster:

Terminal window
# Check TrustyAI Operator pods
oc get pods -n opendatahub -l app.kubernetes.io/part-of=trustyai
# Check TrustyAI CRDs
oc get crd | grep trustyai

Example CRDs (actual list may vary):

  • evalhubs.trustyai.opendatahub.io - EvalHub instances
  • trustyaiservices.trustyai.opendatahub.io - TrustyAI services
  • lmevaljobrequests.trustyai.opendatahub.io - LMEval job requests

Deploy an EvalHub instance using a custom resource.

  1. Create EvalHub Namespace

    Terminal window
    # Create namespace for EvalHub workloads
    oc create namespace evalhub-test
    # Label for monitoring and networking
    oc label namespace evalhub-test \
    opendatahub.io/dashboard=true \
    evalhub.trustyai.io/managed=true
  2. Deploy EvalHub Instance

    EvalHub requires an explicit database configuration via spec.database.type. Choose SQLite for development and testing, or PostgreSQL for production.

    SQLite runs in-process with no external database required. The operator configures an in-memory SQLite database automatically — no Secret is needed. Data is lost when the pod restarts.

    Terminal window
    oc apply -f - <<EOF
    apiVersion: trustyai.opendatahub.io/v1alpha1
    kind: EvalHub
    metadata:
    name: evalhub
    namespace: evalhub-test
    spec:
    replicas: 1
    database:
    type: sqlite
    providers:
    - lm-evaluation-harness
    - garak
    - garak-kfp
    - guidellm
    - lighteval
    - ibm-clear
    collections:
    - leaderboard-v2
    - safety-and-fairness-v1
    - toxicity-and-ethical-principles
    EOF
    FieldRequiredDescription
    spec.replicasNoNumber of EvalHub server replicas. Default: 1.
    spec.database.typeYesDatabase backend. sqlite for in-memory development/testing (no external database needed); postgresql for production.
    spec.database.secretWhen type: postgresqlName of a Kubernetes Secret in the same namespace containing a db-url key with the PostgreSQL connection string (e.g. postgres://user:pass@host:5432/evalhub). Ignored for SQLite.
    spec.database.maxOpenConnsNoMaximum open database connections (PostgreSQL only). Default: 25.
    spec.database.maxIdleConnsNoMaximum idle database connections (PostgreSQL only). Default: 5.
    spec.providersNoList of built-in provider names to enable. Defaults to garak, guidellm, lighteval, lm-evaluation-harness when omitted.
    spec.collectionsNoList of built-in collection names to expose (e.g. leaderboard-v2).
    spec.envNoAdditional environment variables injected into the EvalHub server container, as a list of name/value pairs.
    spec.otelNoOpenTelemetry configuration. See System Overview for details.
  3. Verify Deployment

    Terminal window
    # Check EvalHub CR status
    oc get evalhub evalhub -n evalhub-test
    # Should show STATUS: Ready
    # Check EvalHub pods
    oc get pods -n evalhub-test -l app=eval-hub
    # Check EvalHub service
    oc get svc -n evalhub-test -l app=eval-hub
    # Check route (if OAuth is enabled)
    oc get route -n evalhub-test -l app=eval-hub
  4. Access EvalHub

    Get the EvalHub URL:

    Terminal window
    # Get route URL
    EVALHUB_URL=$(oc get route evalhub -n evalhub-test -o jsonpath='{.spec.host}')
    echo "EvalHub URL: https://$EVALHUB_URL"
    # Test health endpoint
    curl -k "https://$EVALHUB_URL/api/v1/health"

    With OAuth enabled:

    Terminal window
    # Get authentication token
    TOKEN=$(oc whoami -t)
    # Access EvalHub with token
    curl -k -H "Authorization: Bearer $TOKEN" \
    "https://$EVALHUB_URL/api/v1/evaluations/providers"

Develop and test custom EvalHub providers on OpenShift.

  1. Clone Repositories

    Terminal window
    # Clone EvalHub repositories
    git clone https://github.com/eval-hub/eval-hub.git
    git clone https://github.com/eval-hub/eval-hub-sdk.git
    git clone https://github.com/eval-hub/eval-hub-contrib.git
    # Clone TrustyAI Operator (for operator development)
    git clone https://github.com/trustyai-explainability/trustyai-service-operator.git
  2. Upload Custom Operator Manifests

    You can deploy EvalHub directly using the manifests from the cloned TrustyAI operator repository without any modifications. Alternatively, if you want to use custom images or configurations, you can modify the manifests before uploading them.

    For development and testing, you can upload TrustyAI operator manifests directly to the OpenDataHub operator without rebuilding operator images. This allows you to iterate quickly on operator configurations, CRDs, and RBAC definitions.

    The process involves creating a PVC for manifests, patching the operator CSV to mount it, copying manifests in, and restarting the operators. This is based on the OpenDataHub component development workflow.

    Prepare custom manifests:

    Terminal window
    cd trustyai-service-operator

    The operator repository has the following structure under config/:

    config/
    ├── crd/bases/
    │ ├── trustyai.opendatahub.io_evalhubs.yaml
    │ ├── trustyai.opendatahub.io_lmevaljobrequests.yaml
    │ └── trustyai.opendatahub.io_trustyaiservices.yaml
    ├── default/
    ├── manager/
    ├── manifests/
    ├── overlays/odh/
    │ ├── kustomization.yaml
    │ └── params.env ← image references
    ├── rbac/
    └── samples/

    To customize, edit the configuration files:

    Terminal window
    # Example: Update image references
    vim config/overlays/odh/params.env
    # Example: Modify RBAC permissions
    vim config/rbac/evalhub_resource_manager_role.yaml

    Upload manifests to the operator:

    1. Create PersistentVolumeClaim

      Terminal window
      cat <<EOF | oc apply -n openshift-operators -f -
      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
      name: trustyai-manifests
      spec:
      accessModes:
      - ReadWriteOnce
      resources:
      requests:
      storage: 1Gi
      EOF
    2. Patch CSV to mount manifests

      Terminal window
      # Get the OpenDataHub operator CSV name
      CSV=$(oc get csv -n openshift-operators -o name | grep opendatahub-operator | head -n1 | cut -d/ -f2)
      # Patch CSV to mount the PVC
      oc patch csv "${CSV}" -n openshift-operators --type json -p '[
      {"op": "replace", "path": "/spec/install/spec/deployments/0/spec/replicas", "value": 1},
      {"op": "replace", "path": "/spec/install/spec/deployments/0/spec/strategy/type", "value": "Recreate"},
      {"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/securityContext", "value": {"fsGroup": 1001}},
      {"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/volumes/-", "value": {"name": "trustyai-manifests", "persistentVolumeClaim": {"claimName": "trustyai-manifests"}}},
      {"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/volumeMounts/-", "value": {"name": "trustyai-manifests", "mountPath": "/opt/manifests/trustyai"}}
      ]'
    3. Copy manifests into operator pod

      Terminal window
      # Wait for the operator pod to be ready after patching
      oc wait --for=condition=ready pod -n openshift-operators -l name=opendatahub-operator --timeout=120s
      # Get the operator pod name
      POD=$(oc get pod -n openshift-operators -l name=opendatahub-operator -o jsonpath='{.items[0].metadata.name}')
      # Copy manifests from your local repository to the operator pod
      oc cp ./config/. openshift-operators/${POD}:/opt/manifests/trustyai
    4. Restart operators

      Terminal window
      # Restart OpenDataHub operator
      oc rollout restart deploy -n openshift-operators -l name=opendatahub-operator
      oc rollout status deploy -n openshift-operators -l name=opendatahub-operator
      # Restart TrustyAI operator
      oc rollout restart deployment/trustyai-service-operator-controller-manager -n opendatahub

      Wait for the operators to be ready:

      Terminal window
      # Wait for OpenDataHub operator
      oc wait --for=condition=available deployment -n openshift-operators -l name=opendatahub-operator --timeout=120s
      # Wait for TrustyAI operator
      oc wait --for=condition=available deployment/trustyai-service-operator-controller-manager -n opendatahub --timeout=120s

    Verify custom manifests:

    Terminal window
    # Check operator pods are running
    oc get pods -n openshift-operators -l name=opendatahub-operator
    oc get pods -n opendatahub -l app.kubernetes.io/part-of=trustyai
    # Check TrustyAI operator logs for manifest loading
    oc logs -n opendatahub -l app.kubernetes.io/part-of=trustyai --tail=50
    # Verify ConfigMap has your custom values
    oc get configmap trustyai-service-operator-config -n opendatahub -o yaml

    Test your changes by creating or updating an EvalHub CR:

    Terminal window
    # Delete existing EvalHub if it exists
    oc delete evalhub evalhub -n evalhub-test --ignore-not-found
    # Create EvalHub with custom manifests
    oc apply -f - <<EOF
    apiVersion: trustyai.opendatahub.io/v1alpha1
    kind: EvalHub
    metadata:
    name: evalhub
    namespace: evalhub-test
    spec:
    replicas: 1
    database:
    type: sqlite
    providers:
    - lm-evaluation-harness
    - garak
    - garak-kfp
    - guidellm
    - lighteval
    - ibm-clear
    collections:
    - leaderboard-v2
    - safety-and-fairness-v1
    - toxicity-and-ethical-principles
    EOF
    # Check if custom changes are applied
    oc get evalhub evalhub -n evalhub-test -o yaml
    oc get pods -n evalhub-test -l app=eval-hub

    Iterate on changes by repeating the copy and restart commands:

    Terminal window
    POD=$(oc get pod -n openshift-operators -l name=opendatahub-operator -o jsonpath='{.items[0].metadata.name}')
    oc cp ./config/. openshift-operators/${POD}:/opt/manifests/trustyai
    oc rollout restart deploy -n openshift-operators -l name=opendatahub-operator
    oc rollout restart deployment/trustyai-service-operator-controller-manager -n opendatahub

    Cleanup — to remove custom manifests and restore the operator to default:

    Terminal window
    # Delete the PVC
    oc delete pvc trustyai-manifests -n openshift-operators
    # Restore CSV to default by reinstalling the operator
    # Or manually remove the volumeMounts and volumes from the CSV
  3. Build Custom EvalHub Image

    If you want to use a custom EvalHub server image (for example, with modified code or dependencies), build and push it to your container registry, then update the operator manifests before uploading them.

    Login to Quay if you haven’t done already:

    Terminal window
    podman login quay.io

    Build a custom EvalHub server image:

    Terminal window
    cd eval-hub
    IMG=quay.io/your-org/evalhub
    # Build with Podman
    podman build --platform linux/amd64 -t $IMG .
    # Push to registry
    podman push $IMG

    Update the operator manifests to use your custom image:

    Terminal window
    # Edit the params.env file in the operator repository
    cd trustyai-service-operator
    vim config/overlays/odh/params.env
    # Change evalHubImage to your custom image
    # evalHubImage=quay.io/your-org/evalhub:latest

    Then upload the modified manifests using the commands from the previous step:

    Terminal window
    # Copy manifests to operator pod
    POD=$(oc get pod -n openshift-operators -l name=opendatahub-operator -o jsonpath='{.items[0].metadata.name}')
    oc cp ./config/. openshift-operators/${POD}:/opt/manifests/trustyai
    # Restart operators
    oc rollout restart deploy -n openshift-operators -l name=opendatahub-operator
    oc rollout status deploy -n openshift-operators -l name=opendatahub-operator
    oc rollout restart deployment/trustyai-service-operator-controller-manager -n opendatahub

    Delete and recreate the EvalHub instance to use the new image:

    Terminal window
    # Delete existing instance
    oc delete evalhub evalhub -n evalhub-test
    # Recreate with new image
    oc apply -f - <<EOF
    apiVersion: trustyai.opendatahub.io/v1alpha1
    kind: EvalHub
    metadata:
    name: evalhub
    namespace: evalhub-test
    spec:
    replicas: 1
    database:
    type: sqlite
    providers:
    - lm-evaluation-harness
    - garak
    - garak-kfp
    - guidellm
    - lighteval
    - ibm-clear
    collections:
    - leaderboard-v2
    - safety-and-fairness-v1
    - toxicity-and-ethical-principles
    EOF
  4. Test Deployment

    Verify the EvalHub deployment by listing available providers and submitting a test evaluation.

    List available providers:

    Terminal window
    # Get EvalHub URL and token
    EVALHUB_URL=$(oc get route evalhub -n evalhub-test -o jsonpath='{.spec.host}')
    TOKEN=$(oc whoami -t)
    # List all providers
    curl -sS -k -H "Authorization: Bearer $TOKEN" \
    "https://$EVALHUB_URL/api/v1/evaluations/providers" | jq .
    # Get a specific provider (includes benchmarks by default)
    curl -sS -k -H "Authorization: Bearer $TOKEN" \
    "https://$EVALHUB_URL/api/v1/evaluations/providers/lm_evaluation_harness" | jq .

    Submit a test evaluation:

    Terminal window
    # Create evaluation request
    cat > eval-request.json <<EOF
    {
    "model": {
    "url": "http://vllm-server.models.svc.cluster.local:8000/v1",
    "name": "Qwen/Qwen3.5-397B-A17B"
    },
    "benchmarks": [
    {
    "id": "mmlu",
    "provider_id": "lm_evaluation_harness"
    }
    ],
    "experiment": {
    "name": "test-deployment",
    "tags": [
    {
    "key": "environment",
    "value": "development"
    }
    ]
    }
    }
    EOF
    # Submit evaluation
    curl -sS -k -X POST \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @eval-request.json \
    "https://$EVALHUB_URL/api/v1/evaluations/jobs" | jq .

For specific testing purposes when the Model’s answers quality is not relevant, you can deploy an OpenAI-compatible inference simulator instead of a real model server.

Deploy the simulator in the currently active namespace with:

Terminal window
curl -s https://raw.githubusercontent.com/llm-d/llm-d-inference-sim/refs/heads/main/manifests/deployment.yaml \
| yq 'select(.kind == "Deployment").spec.replicas = 1 | select(.kind == "Deployment").spec.template.spec.containers[0].image |= sub(":dev$", ":latest")' \
| oc apply -f -

This makes the simulator available as an internal service at:

  • Service: vllm-sim-demo-svc.evalhub-test.svc.cluster.local (Accessible within the cluster and the namespace only)
  • Port: 8000
  • Model name: Qwen/Qwen3.5-397B-A17B

The service exposes an OpenAI-compatible endpoint (e.g. /v1/chat/completions).

You can verify the simulator is working from within the namespace in the cluster (for example by opening a Terminal on the evalhub Pod) with:

Terminal window
export SIM_URL=vllm-sim-demo-svc.evalhub-test.svc.cluster.local:8000
# List available models
curl -s "http://$SIM_URL/v1/models"
# Chat completion
curl -s "http://$SIM_URL/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.5-397B-A17B",
"messages": [{"role": "user", "content": "Hello"}]
}'

Exposing the simulator externally via a Route

Section titled “Exposing the simulator externally via a Route”

If you need to invoke the simulator externally, create a Route in the OpenShift console:

  1. Navigate to Networking → Routes → Create Route
  2. Select the Service vllm-sim-demo-svc
  3. Select the only available Target Port
  4. Check Secure Route
  5. Set TLS Termination to Edge
  6. Set Insecure Traffic to None

Once the Route is created, you can test it:

Terminal window
export SIM_URL=$(oc get route -n evalhub-test -o jsonpath='{.items[?(@.spec.to.name=="vllm-sim-demo-svc")].spec.host}')
# List available models
curl -s "https://$SIM_URL/v1/models" | jq .
# Chat completion
curl -s "https://$SIM_URL/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.5-397B-A17B",
"messages": [{"role": "user", "content": "Hello"}]
}' | jq .

Symptoms: EvalHub pod stuck in Pending or CrashLoopBackOff

Diagnostics:

Terminal window
# Check pod status
oc get pods -n evalhub-test -l app=eval-hub
# Describe pod for events
oc describe pod -n evalhub-test -l app=eval-hub
# Check logs
oc logs -n evalhub-test -l app=eval-hub --tail=100

Common causes:

  1. Image pull failure

    Terminal window
    # Check image pull secrets
    oc get secret -n evalhub-test
    # Verify image exists
    podman pull quay.io/opendatahub/odh-eval-hub:latest
  2. Insufficient resources

    Terminal window
    # Check node capacity
    oc describe node | grep -A5 "Allocated resources"
    # Reduce resource requests in CR
    oc edit evalhub evalhub -n evalhub-test
  3. ConfigMap missing

    Terminal window
    # Check operator ConfigMap
    oc get configmap trustyai-service-operator-config -n opendatahub

Symptoms: Jobs complete but report failure status

Diagnostics:

Terminal window
# Find evaluation job
oc get jobs -n evalhub-test -l app=eval-hub
# Check job status
oc describe job <job-name> -n evalhub-test
# Check adapter pod logs
oc logs -n evalhub-test -l job-name=<job-name> -c adapter
# Check sidecar logs (if present)
oc logs -n evalhub-test -l job-name=<job-name> -c sidecar

Common causes:

  1. Callback URL unreachable

    Terminal window
    # Verify EvalHub service is accessible from job pods
    oc get svc evalhub -n evalhub-test
    # Test connectivity from job pod
    oc exec -it <job-pod> -n evalhub-test -- \
    curl -v http://evalhub.evalhub-test.svc.cluster.local:8080/api/v1/health
  2. Model endpoint unreachable

    Terminal window
    # Check if model server is accessible
    oc exec -it <job-pod> -n evalhub-test -- \
    curl -v http://vllm-server.models.svc.cluster.local:8000/v1/models
  3. Insufficient job resources

    Terminal window
    # Check for OOMKilled status
    oc get pods -n evalhub-test -l job-name=<job-name> \
    -o jsonpath='{.items[*].status.containerStatuses[*].state}'
    # Increase memory limits in provider config

Symptoms: 401 Unauthorized when accessing EvalHub API

Diagnostics:

Terminal window
# Check OAuth configuration
oc get route evalhub -n evalhub-test -o yaml | grep -A10 tls
# Verify token is valid
oc whoami -t
# Test authentication
TOKEN=$(oc whoami -t)
curl -k -v -H "Authorization: Bearer $TOKEN" \
"https://$(oc get route evalhub -n evalhub-test -o jsonpath='{.spec.host}')/api/v1/health"

Solutions:

  1. Regenerate token

    Terminal window
    oc login --token=<new-token>
  2. Check RBAC configuration

    Terminal window
    # Verify ClusterRoleBinding exists
    oc get clusterrolebinding | grep evalhub
    # Check user permissions
    oc auth can-i create evaluationjobs.trustyai.opendatahub.io

Symptoms: After pushing a new version of an image with the same tag (e.g., :dev), pods continue using the old cached image.

When using a tag other than latest, Kubernetes defaults (if no otherwise specified) to imagePullPolicy: IfNotPresent, meaning nodes will reuse a cached image rather than pulling the updated one. If you don’t want to change the imagePullPolicy, you can force nodes to re-fetch the image by removing it from each node’s container runtime cache.

Remove a cached image from all worker nodes:

Terminal window
IMG=quay.io/your-org/evalhub:dev
oc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | while read NODE; do
oc debug node/$NODE --quiet -- chroot /host crictl rmi $IMG
done

Replace the IMG value with the image reference you want to remove.

After removing the cached image, delete the existing pod(s) so that the new pod pulls the updated image:

Terminal window
oc delete pod -n evalhub-test -l app=eval-hub

Symptoms: EvalHub CR created but no pods deployed

Diagnostics:

Terminal window
# Check operator logs
oc logs -n opendatahub \
-l app.kubernetes.io/part-of=trustyai \
--tail=100
# Check EvalHub CR status
oc get evalhub evalhub -n evalhub-test -o yaml
# Check events
oc get events -n evalhub-test --sort-by='.lastTimestamp'

Solutions:

  1. Restart operator

    Terminal window
    oc delete pod -n opendatahub \
    -l app.kubernetes.io/part-of=trustyai
  2. Check CRD versions

    Terminal window
    # Verify CRD is installed
    oc get crd evalhubs.trustyai.opendatahub.io
    # Check stored versions
    oc get crd evalhubs.trustyai.opendatahub.io \
    -o jsonpath='{.status.storedVersions}'
  3. Recreate EvalHub CR

    Terminal window
    oc delete evalhub evalhub -n evalhub-test
    oc apply -f evalhub-cr.yaml