Skip to content
EvalHub mascot EvalHub
Docs Blog

MCP Installation

since 0.4.2

Summary

Section titled “Summary”

Install the EvalHub MCP server on macOS, Linux, or Windows.

Prerequisites

Section titled “Prerequisites”
  • An EvalHub instance (cluster deployment or local mode)
  • An authentication token (ServiceAccount token for cluster, or none for local mode)

Install the MCP server

Section titled “Install the MCP server”

The EvalHub container image includes the MCP server binary. This is used for Kubernetes/OpenShift deployments managed by the TrustyAI Operator.

See the OpenShift Setup guide for deploying with the operator. The operator’s EvalHub CR supports an optional mcp section to deploy the MCP server as a sidecar or standalone pod.

To run the container locally:

Terminal window
podman run --rm -p 3001:3001 \
  -e EVALHUB_BASE_URL=http://host.containers.internal:8080 \
  -e EVALHUB_TRANSPORT=http \
  quay.io/trustyai/eval-hub:latest \
  /app/evalhub-mcp

Configuration

Section titled “Configuration”

The MCP server connects to an EvalHub API instance. Configure the connection using a YAML file, environment variables, or CLI flags.

Configuration file

Section titled “Configuration file”

Create ~/.evalhub/config.yaml:

base_url: "https://evalhub.apps.my-cluster.example.com"
token: "sha256~..."
tenant: "my-team"
transport: "stdio"

Pass the config file to the server:

Terminal window
evalhub-mcp --config ~/.evalhub/config.yaml

Environment variables

Section titled “Environment variables”

All settings can be set via environment variables:

VariableDescriptionDefault
EVALHUB_BASE_URLEvalHub API URL—
EVALHUB_TOKENAuthentication token—
EVALHUB_TENANTTenant / namespace—
EVALHUB_TRANSPORTTransport mode: stdio, http, http-ssestdio
EVALHUB_HOSTHTTP server bind hostlocalhost
EVALHUB_PORTHTTP server bind port3001
EVALHUB_INSECURESkip TLS certificate verificationfalse
EVALHUB_LIST_PAGE_LIMITDefault page size for list operations200
EVALHUB_TLS_CERT_FILETLS certificate file (for HTTPS serving)—
EVALHUB_TLS_KEY_FILETLS private key file (for HTTPS serving)—
EVALHUB_AUTH_TYPEInbound authentication: none or rbac-proxynone
EVALHUB_CA_CERT_PATHCustom CA certificate path for TLS verification—

CLI flags

Section titled “CLI flags”
Terminal window
evalhub-mcp --transport http --port 3001 --config ~/.evalhub/config.yaml
FlagDescription
--transportstdio, http, or http-sse
--hostHTTP server bind address
--portHTTP server port
--configPath to YAML config file
--insecureSkip TLS verification
--tls-certTLS certificate for HTTPS serving
--tls-keyTLS private key for HTTPS serving
--auth-typeInbound authentication (none or rbac-proxy)
--versionPrint version and exit

Configuration precedence

Section titled “Configuration precedence”

Settings are resolved in this order (highest priority first):

  1. CLI flags
  2. Environment variables
  3. YAML config file
  4. Defaults

Transport modes

Section titled “Transport modes”
ModeProtocolUse case
stdiostdin/stdout JSON-RPCLocal clients (Claude Code, VS Code)
httpStreamable HTTPRemote clients, web integrations
http-sseHTTP + Server-Sent EventsLegacy MCP clients (deprecated)

The http transport includes a health endpoint at GET /health returning {"status":"ok"}.

OpenShift multi-tenant setup

Section titled “OpenShift multi-tenant setup”

When using EvalHub in a multi-tenant OpenShift deployment, create a dedicated ServiceAccount for your AI agent:

Terminal window
oc apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: team-a-agent
  namespace: team-a
EOF

Grant it the required permissions:

Terminal window
oc apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: evalhub-evaluator
  namespace: team-a
rules:
  - apiGroups: [trustyai.opendatahub.io]
    resources: [evaluations, collections, providers]
    verbs: [get, list, create, update, delete]
  - apiGroups: [mlflow.kubeflow.org]
    resources: [experiments]
    verbs: [create, get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: evalhub-evaluator-binding
  namespace: team-a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: evalhub-evaluator
subjects:
  - kind: ServiceAccount
    name: team-a-agent
    namespace: team-a
EOF

Generate a long-lived token for the MCP server:

Terminal window
export EVALHUB_TOKEN=$(oc create token team-a-agent -n team-a --duration=8760h)