Skip to content
EvalHub mascot EvalHub
Docs Blog

MCP Installation

since 0.4.2

Summary

Section titled “Summary”

Install the EvalHub MCP server on macOS, Linux, or Windows.

Prerequisites

Section titled “Prerequisites”
  • An EvalHub instance (cluster deployment or local mode)
  • An authentication token (ServiceAccount token for cluster, or none for local mode)

Install the MCP server

Section titled “Install the MCP server”

The EvalHub container image includes the MCP server binary. This is used for Kubernetes/OpenShift deployments managed by the TrustyAI Operator.

See the OpenShift Setup guide for deploying with the operator. The operator’s EvalHub CR supports an optional mcp section to deploy the MCP server as a sidecar or standalone pod.

To run the container locally:

Terminal window
podman run --rm -p 3001:3001 \
  -e EVALHUB_BASE_URL=http://host.containers.internal:8080 \
  -e EVALHUB_TRANSPORT=http \
  quay.io/trustyai/eval-hub:latest \
  /app/evalhub-mcp

Configuration

Section titled “Configuration”

The MCP server connects to an EvalHub API instance. Configure the connection using a YAML file, environment variables, or CLI flags.

Configuration file

Section titled “Configuration file”

Create ~/.evalhub/config.yaml:

base_url: "https://evalhub.apps.my-cluster.example.com"
token: "sha256~..."
tenant: "my-team"
transport: "stdio"

Pass the config file to the server:

Terminal window
evalhub-mcp --config ~/.evalhub/config.yaml

Environment variables

Section titled “Environment variables”

All settings can be set via environment variables:

VariableDescriptionDefault
EVALHUB_BASE_URLEvalHub API URL—
EVALHUB_TOKENAuthentication token—
EVALHUB_TENANTTenant / namespace—
EVALHUB_TRANSPORTTransport mode: stdio, http, http-ssestdio
EVALHUB_HOSTHTTP server bind hostlocalhost
EVALHUB_PORTHTTP server bind port3001
EVALHUB_INSECURESkip TLS certificate verificationfalse
EVALHUB_LIST_PAGE_LIMITDefault page size for list operations200
EVALHUB_TLS_CERT_FILETLS certificate file (for HTTPS serving)—
EVALHUB_TLS_KEY_FILETLS private key file (for HTTPS serving)—
EVALHUB_AUTH_TYPEInbound authentication: none or rbac-proxynone
EVALHUB_CA_CERT_PATHCustom CA certificate path for TLS verification—

CLI flags

Section titled “CLI flags”
Terminal window
evalhub-mcp --transport http --port 3001 --config ~/.evalhub/config.yaml
FlagDescription
--transportstdio, http, or http-sse
--hostHTTP server bind address
--portHTTP server port
--configPath to YAML config file
--insecureSkip TLS verification
--tls-certTLS certificate for HTTPS serving
--tls-keyTLS private key for HTTPS serving
--auth-typeInbound authentication (none or rbac-proxy)
--versionPrint version and exit

Configuration precedence

Section titled “Configuration precedence”

Settings are resolved in this order (highest priority first):

  1. CLI flags
  2. Environment variables
  3. YAML config file
  4. Defaults

Transport modes

Section titled “Transport modes”
ModeProtocolUse case
stdiostdin/stdout JSON-RPCLocal clients (Claude Code, VS Code)
httpStreamable HTTPRemote clients, web integrations
http-sseHTTP + Server-Sent EventsLegacy MCP clients (deprecated)

The http transport includes a health endpoint at GET /health returning {"status":"ok"}.

CLI-managed configuration (evalhub mcp) — Python SDK CLI only

Section titled “CLI-managed configuration (evalhub mcp) — Python SDK CLI only”
since 0.4.3 eval-hub-sdk

This section applies when managing the Go MCP binary through the Python SDK CLI (evalhub mcp commands). If you run the evalhub-mcp binary directly, use the configuration file, environment variables, or CLI flags above instead.

Configuration is managed with a file-based config key instead of individual profile keys.

Set the MCP config file

Section titled “Set the MCP config file”

Provide a YAML config file with MCP-specific settings. The CLI validates the file and stores a profile-scoped copy:

Terminal window
evalhub config set mcp_config_file mcp-config.yaml

Example mcp-config.yaml:

transport: http
host: localhost
port: 3001

Config merging

Section titled “Config merging”

The CLI merges connection keys from the active profile (base_url, token, tenant, insecure) with the MCP config file. MCP config values take precedence. The merged result is written to ~/.config/evalhub/mcp/<profile>/mcp-config.yaml and passed to the evalhub-mcp binary automatically.

Resolution priority (highest first):

  1. Values in mcp_config_file
  2. Root profile values (base_url, token, tenant, insecure)

If neither source provides a value, the key is omitted.

Inspect the config

Section titled “Inspect the config”
Terminal window
evalhub config get mcp_config_file              # show stored path
evalhub config get mcp_config_file --unfold      # show file contents (tokens masked)
evalhub config get mcp_config_file --unfold --unmask  # show file contents with tokens revealed

Remove the MCP config

Section titled “Remove the MCP config”
Terminal window
evalhub config unset mcp_config_file

This removes the stored config file and cleans up the profile directory.

Default transports

Section titled “Default transports”
CommandDefault transportUse case
evalhub mcp runstdioLocal clients (Claude Code, VS Code)
evalhub mcp starthttpBackground daemon for remote clients

evalhub mcp start rejects stdio transport — use evalhub mcp run for stdio.

Checking MCP server status

Section titled “Checking MCP server status”

evalhub mcp status reports whether the background MCP server is running. If running, it performs a full MCP JSON-RPC handshake to report the server name, version, and git commit hash:

Terminal window
evalhub mcp status
# MCP server is running (PID 12345).
#   Name:    evalhub-mcp
#   Version: 0.4.3
#   Commit:  abc1234
#   URL:     http://localhost:3001
#   Logs:    ~/.config/evalhub/mcp/mcp.log

OpenShift multi-tenant setup

Section titled “OpenShift multi-tenant setup”

When using EvalHub in a multi-tenant OpenShift deployment, create a dedicated ServiceAccount for your AI agent:

Terminal window
oc apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: team-a-agent
  namespace: team-a
EOF

Grant it the required permissions:

Terminal window
oc apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: evalhub-evaluator
  namespace: team-a
rules:
  - apiGroups: [trustyai.opendatahub.io]
    resources: [evaluations, collections, providers]
    verbs: [get, list, create, update, delete]
  - apiGroups: [mlflow.kubeflow.org]
    resources: [experiments]
    verbs: [create, get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: evalhub-evaluator-binding
  namespace: team-a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: evalhub-evaluator
subjects:
  - kind: ServiceAccount
    name: team-a-agent
    namespace: team-a
EOF

Generate a long-lived token for the MCP server:

Terminal window
export EVALHUB_TOKEN=$(oc create token team-a-agent -n team-a --duration=8760h)