OpenShift Development Setup
Complete guide for deploying and developing EvalHub on OpenShift with OpenDataHub.
Prerequisites
Section titled “Prerequisites”Required Tools
Section titled “Required Tools”- OpenShift CLI (
oc) - Version 4.12+ - kubectl - Compatible with your OpenShift version
- jq - JSON processor for manifest manipulation
- Git - For cloning repositories
- Podman or Docker - For building custom images
Cluster Access
Section titled “Cluster Access”Ensure you have access to an OpenShift cluster:
# Login to OpenShiftoc login --server=https://api.your-cluster.example.com:6443
# Verify accessoc whoamioc cluster-infoOpenDataHub Installation
Section titled “OpenDataHub Installation”OpenDataHub provides the foundation for deploying EvalHub on OpenShift.
-
Install OpenDataHub Operator
Install the OpenDataHub Operator (version 3.3 or higher) from OperatorHub in the OpenShift web console:
- Navigate to Operators → OperatorHub
- Search for “OpenDataHub”
- Click Install
- Select fast channel
- Choose All namespaces on the cluster installation mode
- Click Install
Wait for the operator installation to complete:
Terminal window # Check operator installationoc get csv -n openshift-operators | grep opendatahub# Should show PHASE: "Succeeded" -
Configure OpenDataHub
After the operator is installed, configure OpenDataHub from the operator’s dashboard:
- Navigate to Operators → Installed Operators
- Select OpenDataHub Operator
- Go to the DSCInitialization tab
- Click Create DSCInitialization
- Review or just use default settings
- Click Create
- Await “Phase: Ready”
- Go to the Data Science Cluster tab
- Click Create DataScienceCluster
- Review or just use default settings for most components, do not save/create yet
- Important: Under the TrustyAI component, configure LMEval security settings:
- Set
permitCodeExecutionto allow - Set
permitOnlineto allow
- Set
- Click Create
Verify the DataScienceCluster is ready:
Terminal window # Check DSC statusoc get datasciencecluster default-dsc -o jsonpath='{.status.phase}'# Should output: "Ready"# List deployed componentsoc get pods -n opendatahub
TrustyAI Operator Deployment
Section titled “TrustyAI Operator Deployment”The TrustyAI Operator manages EvalHub custom resources.
The TrustyAI Operator is installed automatically as part of the DataScienceCluster:
# Check TrustyAI Operator podsoc get pods -n opendatahub -l app.kubernetes.io/part-of=trustyai
# Check TrustyAI CRDsoc get crd | grep trustyaiExample CRDs (actual list may vary):
evalhubs.trustyai.opendatahub.io- EvalHub instancestrustyaiservices.trustyai.opendatahub.io- TrustyAI serviceslmevaljobrequests.trustyai.opendatahub.io- LMEval job requests
EvalHub Custom Resource
Section titled “EvalHub Custom Resource”Deploy an EvalHub instance using a custom resource.
-
Create EvalHub Namespace
Terminal window # Create namespace for EvalHub workloadsoc create namespace evalhub-test# Label for monitoring and networkingoc label namespace evalhub-test \opendatahub.io/dashboard=true \evalhub.trustyai.io/managed=true -
Deploy EvalHub Instance
EvalHub requires an explicit database configuration via
spec.database.type. Choose SQLite for development and testing, or PostgreSQL for production.SQLite runs in-process with no external database required. The operator configures an in-memory SQLite database automatically — no Secret is needed. Data is lost when the pod restarts.
Terminal window oc apply -f - <<EOFapiVersion: trustyai.opendatahub.io/v1alpha1kind: EvalHubmetadata:name: evalhubnamespace: evalhub-testspec:replicas: 1database:type: sqliteproviders:- lm-evaluation-harness- garak- garak-kfp- guidellm- lighteval- ibm-clearcollections:- leaderboard-v2- safety-and-fairness-v1- toxicity-and-ethical-principlesEOFPostgreSQL provides durable, persistent storage. Create a Secret containing the connection URL first, then reference it in the CR.
Terminal window oc apply -f - <<EOFapiVersion: v1kind: Secretmetadata:name: evalhub-db-credentialsnamespace: evalhub-testtype: OpaquestringData:db-url: "postgres://user:password@db-host:5432/evalhub"EOFoc apply -f - <<EOFapiVersion: trustyai.opendatahub.io/v1alpha1kind: EvalHubmetadata:name: evalhubnamespace: evalhub-testspec:replicas: 1database:type: postgresqlsecret: evalhub-db-credentialsproviders:- lm-evaluation-harness- garak- garak-kfp- guidellm- lighteval- ibm-clearcollections:- leaderboard-v2- safety-and-fairness-v1- toxicity-and-ethical-principlesEOFEvalHub Custom Resource Spec
Section titled “EvalHub Custom Resource Spec”Field Required Description spec.replicasNo Number of EvalHub server replicas. Default: 1.spec.database.typeYes Database backend. sqlitefor in-memory development/testing (no external database needed);postgresqlfor production.spec.database.secretWhen type: postgresqlName of a Kubernetes Secretin the same namespace containing adb-urlkey with the PostgreSQL connection string (e.g.postgres://user:pass@host:5432/evalhub). Ignored for SQLite.spec.database.maxOpenConnsNo Maximum open database connections (PostgreSQL only). Default: 25.spec.database.maxIdleConnsNo Maximum idle database connections (PostgreSQL only). Default: 5.spec.providersNo List of built-in provider names to enable. Defaults to garak,guidellm,lighteval,lm-evaluation-harnesswhen omitted.spec.collectionsNo List of built-in collection names to expose (e.g. leaderboard-v2).spec.envNo Additional environment variables injected into the EvalHub server container, as a list of name/valuepairs.spec.otelNo OpenTelemetry configuration. See System Overview for details. -
Verify Deployment
Terminal window # Check EvalHub CR statusoc get evalhub evalhub -n evalhub-test# Should show STATUS: Ready# Check EvalHub podsoc get pods -n evalhub-test -l app=eval-hub# Check EvalHub serviceoc get svc -n evalhub-test -l app=eval-hub# Check route (if OAuth is enabled)oc get route -n evalhub-test -l app=eval-hub -
Access EvalHub
Get the EvalHub URL:
Terminal window # Get route URLEVALHUB_URL=$(oc get route evalhub -n evalhub-test -o jsonpath='{.spec.host}')echo "EvalHub URL: https://$EVALHUB_URL"# Test health endpointcurl -k "https://$EVALHUB_URL/api/v1/health"With OAuth enabled:
Terminal window # Get authentication tokenTOKEN=$(oc whoami -t)# Access EvalHub with tokencurl -k -H "Authorization: Bearer $TOKEN" \"https://$EVALHUB_URL/api/v1/evaluations/providers"
Development Workflow
Section titled “Development Workflow”Develop and test custom EvalHub providers on OpenShift.
-
Clone Repositories
Terminal window # Clone EvalHub repositoriesgit clone https://github.com/eval-hub/eval-hub.gitgit clone https://github.com/eval-hub/eval-hub-sdk.gitgit clone https://github.com/eval-hub/eval-hub-contrib.git# Clone TrustyAI Operator (for operator development)git clone https://github.com/trustyai-explainability/trustyai-service-operator.git -
Upload Custom Operator Manifests
You can deploy EvalHub directly using the manifests from the cloned TrustyAI operator repository without any modifications. Alternatively, if you want to use custom images or configurations, you can modify the manifests before uploading them.
For development and testing, you can upload TrustyAI operator manifests directly to the OpenDataHub operator without rebuilding operator images. This allows you to iterate quickly on operator configurations, CRDs, and RBAC definitions.
The process involves creating a PVC for manifests, patching the operator CSV to mount it, copying manifests in, and restarting the operators. This is based on the OpenDataHub component development workflow.
Prepare custom manifests:
Terminal window cd trustyai-service-operatorThe operator repository has the following structure under
config/:config/├── crd/bases/│ ├── trustyai.opendatahub.io_evalhubs.yaml│ ├── trustyai.opendatahub.io_lmevaljobrequests.yaml│ └── trustyai.opendatahub.io_trustyaiservices.yaml├── default/├── manager/├── manifests/├── overlays/odh/│ ├── kustomization.yaml│ └── params.env ← image references├── rbac/└── samples/To customize, edit the configuration files:
Terminal window # Example: Update image referencesvim config/overlays/odh/params.env# Example: Modify RBAC permissionsvim config/rbac/evalhub_resource_manager_role.yamlUpload manifests to the operator:
-
Create PersistentVolumeClaim
Terminal window cat <<EOF | oc apply -n openshift-operators -f -apiVersion: v1kind: PersistentVolumeClaimmetadata:name: trustyai-manifestsspec:accessModes:- ReadWriteOnceresources:requests:storage: 1GiEOF -
Patch CSV to mount manifests
Terminal window # Get the OpenDataHub operator CSV nameCSV=$(oc get csv -n openshift-operators -o name | grep opendatahub-operator | head -n1 | cut -d/ -f2)# Patch CSV to mount the PVCoc patch csv "${CSV}" -n openshift-operators --type json -p '[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/replicas", "value": 1},{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/strategy/type", "value": "Recreate"},{"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/securityContext", "value": {"fsGroup": 1001}},{"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/volumes/-", "value": {"name": "trustyai-manifests", "persistentVolumeClaim": {"claimName": "trustyai-manifests"}}},{"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/volumeMounts/-", "value": {"name": "trustyai-manifests", "mountPath": "/opt/manifests/trustyai"}}]' -
Copy manifests into operator pod
Terminal window # Wait for the operator pod to be ready after patchingoc wait --for=condition=ready pod -n openshift-operators -l name=opendatahub-operator --timeout=120s# Get the operator pod namePOD=$(oc get pod -n openshift-operators -l name=opendatahub-operator -o jsonpath='{.items[0].metadata.name}')# Copy manifests from your local repository to the operator podoc cp ./config/. openshift-operators/${POD}:/opt/manifests/trustyai -
Restart operators
Terminal window # Restart OpenDataHub operatoroc rollout restart deploy -n openshift-operators -l name=opendatahub-operatoroc rollout status deploy -n openshift-operators -l name=opendatahub-operator# Restart TrustyAI operatoroc rollout restart deployment/trustyai-service-operator-controller-manager -n opendatahubWait for the operators to be ready:
Terminal window # Wait for OpenDataHub operatoroc wait --for=condition=available deployment -n openshift-operators -l name=opendatahub-operator --timeout=120s# Wait for TrustyAI operatoroc wait --for=condition=available deployment/trustyai-service-operator-controller-manager -n opendatahub --timeout=120s
Verify custom manifests:
Terminal window # Check operator pods are runningoc get pods -n openshift-operators -l name=opendatahub-operatoroc get pods -n opendatahub -l app.kubernetes.io/part-of=trustyai# Check TrustyAI operator logs for manifest loadingoc logs -n opendatahub -l app.kubernetes.io/part-of=trustyai --tail=50# Verify ConfigMap has your custom valuesoc get configmap trustyai-service-operator-config -n opendatahub -o yamlTest your changes by creating or updating an EvalHub CR:
Terminal window # Delete existing EvalHub if it existsoc delete evalhub evalhub -n evalhub-test --ignore-not-found# Create EvalHub with custom manifestsoc apply -f - <<EOFapiVersion: trustyai.opendatahub.io/v1alpha1kind: EvalHubmetadata:name: evalhubnamespace: evalhub-testspec:replicas: 1database:type: sqliteproviders:- lm-evaluation-harness- garak- garak-kfp- guidellm- lighteval- ibm-clearcollections:- leaderboard-v2- safety-and-fairness-v1- toxicity-and-ethical-principlesEOF# Check if custom changes are appliedoc get evalhub evalhub -n evalhub-test -o yamloc get pods -n evalhub-test -l app=eval-hubIterate on changes by repeating the copy and restart commands:
Terminal window POD=$(oc get pod -n openshift-operators -l name=opendatahub-operator -o jsonpath='{.items[0].metadata.name}')oc cp ./config/. openshift-operators/${POD}:/opt/manifests/trustyaioc rollout restart deploy -n openshift-operators -l name=opendatahub-operatoroc rollout restart deployment/trustyai-service-operator-controller-manager -n opendatahubCleanup — to remove custom manifests and restore the operator to default:
Terminal window # Delete the PVCoc delete pvc trustyai-manifests -n openshift-operators# Restore CSV to default by reinstalling the operator# Or manually remove the volumeMounts and volumes from the CSV -
-
Build Custom EvalHub Image
If you want to use a custom EvalHub server image (for example, with modified code or dependencies), build and push it to your container registry, then update the operator manifests before uploading them.
Login to Quay if you haven’t done already:
Terminal window podman login quay.ioBuild a custom EvalHub server image:
Terminal window cd eval-hubIMG=quay.io/your-org/evalhub# Build with Podmanpodman build --platform linux/amd64 -t $IMG .# Push to registrypodman push $IMGUpdate the operator manifests to use your custom image:
Terminal window # Edit the params.env file in the operator repositorycd trustyai-service-operatorvim config/overlays/odh/params.env# Change evalHubImage to your custom image# evalHubImage=quay.io/your-org/evalhub:latestThen upload the modified manifests using the commands from the previous step:
Terminal window # Copy manifests to operator podPOD=$(oc get pod -n openshift-operators -l name=opendatahub-operator -o jsonpath='{.items[0].metadata.name}')oc cp ./config/. openshift-operators/${POD}:/opt/manifests/trustyai# Restart operatorsoc rollout restart deploy -n openshift-operators -l name=opendatahub-operatoroc rollout status deploy -n openshift-operators -l name=opendatahub-operatoroc rollout restart deployment/trustyai-service-operator-controller-manager -n opendatahubDelete and recreate the EvalHub instance to use the new image:
Terminal window # Delete existing instanceoc delete evalhub evalhub -n evalhub-test# Recreate with new imageoc apply -f - <<EOFapiVersion: trustyai.opendatahub.io/v1alpha1kind: EvalHubmetadata:name: evalhubnamespace: evalhub-testspec:replicas: 1database:type: sqliteproviders:- lm-evaluation-harness- garak- garak-kfp- guidellm- lighteval- ibm-clearcollections:- leaderboard-v2- safety-and-fairness-v1- toxicity-and-ethical-principlesEOF -
Test Deployment
Verify the EvalHub deployment by listing available providers and submitting a test evaluation.
List available providers:
Terminal window # Get EvalHub URL and tokenEVALHUB_URL=$(oc get route evalhub -n evalhub-test -o jsonpath='{.spec.host}')TOKEN=$(oc whoami -t)# List all providerscurl -sS -k -H "Authorization: Bearer $TOKEN" \"https://$EVALHUB_URL/api/v1/evaluations/providers" | jq .# Get a specific provider (includes benchmarks by default)curl -sS -k -H "Authorization: Bearer $TOKEN" \"https://$EVALHUB_URL/api/v1/evaluations/providers/lm_evaluation_harness" | jq .Submit a test evaluation:
Terminal window # Create evaluation requestcat > eval-request.json <<EOF{"model": {"url": "http://vllm-server.models.svc.cluster.local:8000/v1","name": "Qwen/Qwen3.5-397B-A17B"},"benchmarks": [{"id": "mmlu","provider_id": "lm_evaluation_harness"}],"experiment": {"name": "test-deployment","tags": [{"key": "environment","value": "development"}]}}EOF# Submit evaluationcurl -sS -k -X POST \-H "Authorization: Bearer $TOKEN" \-H "Content-Type: application/json" \-d @eval-request.json \"https://$EVALHUB_URL/api/v1/evaluations/jobs" | jq .
Deploying an OpenAI-compatible simulator
Section titled “Deploying an OpenAI-compatible simulator”For specific testing purposes when the Model’s answers quality is not relevant, you can deploy an OpenAI-compatible inference simulator instead of a real model server.
Deploy the simulator in the currently active namespace with:
curl -s https://raw.githubusercontent.com/llm-d/llm-d-inference-sim/refs/heads/main/manifests/deployment.yaml \ | yq 'select(.kind == "Deployment").spec.replicas = 1 | select(.kind == "Deployment").spec.template.spec.containers[0].image |= sub(":dev$", ":latest")' \ | oc apply -f -This makes the simulator available as an internal service at:
- Service:
vllm-sim-demo-svc.evalhub-test.svc.cluster.local(Accessible within the cluster and the namespace only) - Port:
8000 - Model name:
Qwen/Qwen3.5-397B-A17B
The service exposes an OpenAI-compatible endpoint (e.g. /v1/chat/completions).
You can verify the simulator is working from within the namespace in the cluster (for example by opening a Terminal on the evalhub Pod) with:
export SIM_URL=vllm-sim-demo-svc.evalhub-test.svc.cluster.local:8000
# List available modelscurl -s "http://$SIM_URL/v1/models"
# Chat completioncurl -s "http://$SIM_URL/v1/chat/completions" \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen3.5-397B-A17B", "messages": [{"role": "user", "content": "Hello"}] }'Exposing the simulator externally via a Route
Section titled “Exposing the simulator externally via a Route”If you need to invoke the simulator externally, create a Route in the OpenShift console:
- Navigate to Networking → Routes → Create Route
- Select the Service
vllm-sim-demo-svc - Select the only available Target Port
- Check Secure Route
- Set TLS Termination to Edge
- Set Insecure Traffic to None
Once the Route is created, you can test it:
export SIM_URL=$(oc get route -n evalhub-test -o jsonpath='{.items[?(@.spec.to.name=="vllm-sim-demo-svc")].spec.host}')
# List available modelscurl -s "https://$SIM_URL/v1/models" | jq .
# Chat completioncurl -s "https://$SIM_URL/v1/chat/completions" \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen3.5-397B-A17B", "messages": [{"role": "user", "content": "Hello"}] }' | jq .Troubleshooting
Section titled “Troubleshooting”EvalHub Pod Not Starting
Section titled “EvalHub Pod Not Starting”Symptoms: EvalHub pod stuck in Pending or CrashLoopBackOff
Diagnostics:
# Check pod statusoc get pods -n evalhub-test -l app=eval-hub
# Describe pod for eventsoc describe pod -n evalhub-test -l app=eval-hub
# Check logsoc logs -n evalhub-test -l app=eval-hub --tail=100Common causes:
-
Image pull failure
Terminal window # Check image pull secretsoc get secret -n evalhub-test# Verify image existspodman pull quay.io/opendatahub/odh-eval-hub:latest -
Insufficient resources
Terminal window # Check node capacityoc describe node | grep -A5 "Allocated resources"# Reduce resource requests in CRoc edit evalhub evalhub -n evalhub-test -
ConfigMap missing
Terminal window # Check operator ConfigMapoc get configmap trustyai-service-operator-config -n opendatahub
Evaluation Jobs Failing
Section titled “Evaluation Jobs Failing”Symptoms: Jobs complete but report failure status
Diagnostics:
# Find evaluation joboc get jobs -n evalhub-test -l app=eval-hub
# Check job statusoc describe job <job-name> -n evalhub-test
# Check adapter pod logsoc logs -n evalhub-test -l job-name=<job-name> -c adapter
# Check sidecar logs (if present)oc logs -n evalhub-test -l job-name=<job-name> -c sidecarCommon causes:
-
Callback URL unreachable
Terminal window # Verify EvalHub service is accessible from job podsoc get svc evalhub -n evalhub-test# Test connectivity from job podoc exec -it <job-pod> -n evalhub-test -- \curl -v http://evalhub.evalhub-test.svc.cluster.local:8080/api/v1/health -
Model endpoint unreachable
Terminal window # Check if model server is accessibleoc exec -it <job-pod> -n evalhub-test -- \curl -v http://vllm-server.models.svc.cluster.local:8000/v1/models -
Insufficient job resources
Terminal window # Check for OOMKilled statusoc get pods -n evalhub-test -l job-name=<job-name> \-o jsonpath='{.items[*].status.containerStatuses[*].state}'# Increase memory limits in provider config
OAuth Authentication Issues
Section titled “OAuth Authentication Issues”Symptoms: 401 Unauthorized when accessing EvalHub API
Diagnostics:
# Check OAuth configurationoc get route evalhub -n evalhub-test -o yaml | grep -A10 tls
# Verify token is validoc whoami -t
# Test authenticationTOKEN=$(oc whoami -t)curl -k -v -H "Authorization: Bearer $TOKEN" \ "https://$(oc get route evalhub -n evalhub-test -o jsonpath='{.spec.host}')/api/v1/health"Solutions:
-
Regenerate token
Terminal window oc login --token=<new-token> -
Check RBAC configuration
Terminal window # Verify ClusterRoleBinding existsoc get clusterrolebinding | grep evalhub# Check user permissionsoc auth can-i create evaluationjobs.trustyai.opendatahub.io
Removing Cached Images from Nodes
Section titled “Removing Cached Images from Nodes”Symptoms: After pushing a new version of an image with the same tag (e.g., :dev), pods continue using the old cached image.
When using a tag other than latest, Kubernetes defaults (if no otherwise specified) to imagePullPolicy: IfNotPresent, meaning nodes will reuse a cached image rather than pulling the updated one. If you don’t want to change the imagePullPolicy, you can force nodes to re-fetch the image by removing it from each node’s container runtime cache.
Remove a cached image from all worker nodes:
IMG=quay.io/your-org/evalhub:devoc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | while read NODE; do oc debug node/$NODE --quiet -- chroot /host crictl rmi $IMGdoneReplace the IMG value with the image reference you want to remove.
After removing the cached image, delete the existing pod(s) so that the new pod pulls the updated image:
oc delete pod -n evalhub-test -l app=eval-hubOperator Not Reconciling
Section titled “Operator Not Reconciling”Symptoms: EvalHub CR created but no pods deployed
Diagnostics:
# Check operator logsoc logs -n opendatahub \ -l app.kubernetes.io/part-of=trustyai \ --tail=100
# Check EvalHub CR statusoc get evalhub evalhub -n evalhub-test -o yaml
# Check eventsoc get events -n evalhub-test --sort-by='.lastTimestamp'Solutions:
-
Restart operator
Terminal window oc delete pod -n opendatahub \-l app.kubernetes.io/part-of=trustyai -
Check CRD versions
Terminal window # Verify CRD is installedoc get crd evalhubs.trustyai.opendatahub.io# Check stored versionsoc get crd evalhubs.trustyai.opendatahub.io \-o jsonpath='{.status.storedVersions}' -
Recreate EvalHub CR
Terminal window oc delete evalhub evalhub -n evalhub-testoc apply -f evalhub-cr.yaml