Multi-Tenancy¶

Guide for deploying EvalHub with namespace-based multi-tenancy on OpenShift.

Overview¶

EvalHub uses Kubernetes namespaces as tenant boundaries. Each tenant operates in its own namespace, and access is enforced through Kubernetes RBAC via SubjectAccessReview (SAR) checks.

The core principle is: namespace = tenant = MLFlow workspace.

flowchart LR
    subgraph "Control Plane (opendatahub)"
        Operator[TrustyAI Operator]
        EvalHub[EvalHub API]
    end

    subgraph "Tenant A (team-a)"
        SA_A[Job SA]
        Job_A[Eval Job]
    end

    subgraph "Tenant B (team-b)"
        SA_B[Job SA]
        Job_B[Eval Job]
    end

    Operator -->|creates RBAC| SA_A
    Operator -->|creates RBAC| SA_B
    EvalHub -->|schedules jobs| Job_A
    EvalHub -->|schedules jobs| Job_B

How tenant isolation works¶

Authentication -- Bearer tokens are validated via the Kubernetes TokenReview API. The token can belong to a ServiceAccount or a real OpenShift User — EvalHub treats both identically at this stage.
Authorisation -- Every API request includes an X-Tenant header specifying the target namespace. EvalHub runs a SAR check: "can this principal perform this action in that namespace?"
Data isolation -- All database queries are filtered by tenant_id
Job isolation -- Evaluation jobs run in the tenant namespace with a scoped ServiceAccount

Tenant ID and user identity are independent

The tenant is always determined by the X-Tenant request header — not by the identity in the token. A ServiceAccount has a home namespace, but EvalHub does not use it. A real OpenShift User has no namespace at all. In both cases the caller must supply X-Tenant explicitly, and a matching RoleBinding must exist in that namespace.

Prerequisites¶

Before setting up multi-tenancy, ensure you have:

A working EvalHub deployment (see OpenShift Setup)
Cluster-admin access (for creating namespaces and ClusterRoleBindings)
The TrustyAI Operator installed and reconciling

Step 1: Deploy EvalHub¶

Create an EvalHub instance in the control-plane namespace. For multi-tenancy, a persistent database is recommended:

apiVersion: trustyai.opendatahub.io/v1alpha1
kind: EvalHub
metadata:
  name: evalhub
  namespace: opendatahub
spec:
  replicas: 1
  database:
    type: postgresql
    secret: evalhub-db-credentials

The operator automatically creates:

Resource	Namespace	Purpose
`evalhub-service` SA	`opendatahub`	API server identity
`evalhub-opendatahub-job` SA	`opendatahub`	Jobs in the control-plane namespace
ClusterRoleBinding for auth-reviewer	cluster-wide	Allows SAR and TokenReview checks
RoleBindings for jobs-writer, job-config	`opendatahub`	Allows creating Jobs and ConfigMaps

Step 2: Register tenant namespaces with the operator¶

Create a namespace for each tenant and label it so the TrustyAI Operator's namespace watcher provisions job resources and RBAC automatically:

oc create namespace team-a
oc create namespace team-b

# Label each namespace as an EvalHub tenant (value can be empty or e.g. "true")
oc label namespace team-a evalhub.trustyai.opendatahub.io/tenant=
oc label namespace team-b evalhub.trustyai.opendatahub.io/tenant=

The operator watches for namespaces with the label evalhub.trustyai.opendatahub.io/tenant. When it sees a labelled namespace (other than the EvalHub instance namespace), it automatically creates in that namespace:

Resource	Purpose
Job ServiceAccount (`{evalhub-name}-{evalhub-namespace}-job`)	Identity for evaluation job pods
Job access Role + RoleBinding	Allows job pods to create `status-events`
jobs-writer RoleBinding	Binds EvalHub API SA to `evalhub-jobs-writer` ClusterRole (create/delete Jobs)
job-config RoleBinding	Binds EvalHub API SA to `evalhub-job-config` ClusterRole (ConfigMaps)
MLFlow job RoleBinding	Binds job SA to `evalhub-mlflow-jobs-access` ClusterRole
Service CA ConfigMap	For TLS callbacks from job pods to EvalHub (OpenShift service CA injection)

Instance namespace

In the EvalHub instance namespace (e.g. opendatahub), the controller also creates the evalhub-service SA, the job SA, all RoleBindings, and the auth-reviewer ClusterRoleBinding. You do not need to create these manually. Labelling the instance namespace as a tenant is unnecessary and is skipped by the watcher.

If you remove the tenant label from a namespace, the operator will clean up the job-related resources from that namespace.

Step 3: Create tenant principals (users or service accounts)¶

The operator provisions the job ServiceAccount and RBAC in each labelled namespace. You still need to create a tenant user identity and bind it to evaluation permissions. This is the principal that API consumers use to authenticate when calling the EvalHub API.

EvalHub supports two principal types:

Principal	Token source	RoleBinding `kind`	Has home namespace?
ServiceAccount	`oc create token`	`ServiceAccount`	Yes (where SA lives)
OpenShift User	`oc whoami -t` (OAuth)	`User`	No

In both cases the tenant namespace is supplied by the caller via the X-Tenant header, not inferred from the identity. A RoleBinding granting evaluation permissions must exist in that namespace.

ServiceAccountOpenShift User

Create a ServiceAccount in the tenant namespace and grant it evaluation permissions there.

Create the ServiceAccount:

oc apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: team-a-user
  namespace: team-a
EOF

Grant evaluation permissions:

oc apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: evalhub-evaluator
  namespace: team-a
rules:
  - apiGroups: [trustyai.opendatahub.io]
    resources: [evaluations, collections, providers]
    verbs: [get, list, create, update, delete]
  - apiGroups: [mlflow.kubeflow.org]
    resources: [experiments]
    verbs: [create, get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: evalhub-evaluator-binding
  namespace: team-a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: evalhub-evaluator
subjects:
  - kind: ServiceAccount
    name: team-a-user
    namespace: team-a       # required for ServiceAccount subjects
EOF

Bind an existing OpenShift User (a human identity from the OAuth server) to evaluation permissions in the tenant namespace. No ServiceAccount needs to be created.

Grant evaluation permissions:

oc apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: evalhub-evaluator
  namespace: team-a
rules:
  - apiGroups: [trustyai.opendatahub.io]
    resources: [evaluations, collections, providers]
    verbs: [get, list, create, update, delete]
  - apiGroups: [mlflow.kubeflow.org]
    resources: [experiments]
    verbs: [create, get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: evalhub-evaluator-binding
  namespace: team-a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: evalhub-evaluator
subjects:
  - kind: User
    name: alice             # OpenShift username (oc whoami)
    apiGroup: rbac.authorization.k8s.io
                            # no namespace field — Users are cluster-scoped
EOF

Username must match exactly

The name field must match the OpenShift username as returned by oc whoami. The TokenReview resolves the OAuth token to this username, and the SAR engine matches it against the kind: User subject in the binding.

Virtual resources

The resources in the Role (evaluations, collections, providers, status-events) are virtual -- they don't correspond to actual Kubernetes API resources. EvalHub uses them as SAR targets to enforce fine-grained access control via the Kubernetes authorisation API.

Step 4: Access the API with tenant scoping¶

All evaluation API requests must include the X-Tenant header set to the target namespace. The header is the sole source of the tenant ID — it is not derived from the token.

Get the EvalHub route:

EVALHUB_URL=$(oc get route evalhub -n opendatahub -o jsonpath='{.spec.host}')

Get a token:

ServiceAccountOpenShift User

TOKEN=$(oc create token team-a-user -n team-a --duration=1h)

The TokenReview will resolve this to system:serviceaccount:team-a:team-a-user.

# Log in as the user first (if not already)
oc login --username=alice --password=<password>

TOKEN=$(oc whoami -t)

The TokenReview will resolve this to alice (the plain OpenShift username).

Use oc whoami -t, not oc create token

oc create token always creates a ServiceAccount token, even when run as a logged-in user. To get an OAuth token for a real OpenShift User, use oc whoami -t after logging in.

List providers (scoped to team-a) (or alternatively, use the CLI):

curl -sS -k \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Tenant: team-a" \
  "https://$EVALHUB_URL/api/v1/evaluations/providers" | jq .

Submit an evaluation job:

curl -sS -k -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Tenant: team-a" \
  -H "Content-Type: application/json" \
  -d '{
    "model": {
      "url": "http://vllm-server.team-a.svc.cluster.local:8000/v1",
      "name": "meta-llama/Llama-3.2-1B-Instruct"
    },
    "benchmarks": [
      {
        "id": "mmlu",
        "provider_id": "lm_evaluation_harness"
      }
    ]
  }' \
  "https://$EVALHUB_URL/api/v1/evaluations/jobs" | jq .

The job pod will be created in the team-a namespace using the evalhub-opendatahub-job ServiceAccount, regardless of which principal type submitted the request.

Cross-tenant access is denied:

# This will return 403 Forbidden
curl -sS -k -X GET \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Tenant: team-b" \
  "https://$EVALHUB_URL/api/v1/evaluations/jobs"

The SAR check fails because the principal (whether a ServiceAccount or a User) has no RoleBinding in the team-b namespace.

Authorisation model¶

EvalHub uses an embedded SAR authoriser. The auth config (config/auth.yaml) maps API endpoints to Kubernetes resource attributes:

Endpoint	Resource	Verb	Namespace source
`POST /api/v1/evaluations/jobs`	`evaluations`	`create`	`X-Tenant` header
`GET /api/v1/evaluations/jobs`	`evaluations`	`get`	`X-Tenant` header
`POST /api/v1/evaluations/jobs/*/events`	`status-events`	`create`	`X-Tenant` header
`* /api/v1/evaluations/collections`	`collections`	(from HTTP method)	`X-Tenant` header
`* /api/v1/evaluations/providers`	`providers`	(from HTTP method)	`X-Tenant` header

For POST /api/v1/evaluations/jobs, two additional SAR checks are performed for MLFlow access (mlflow.kubeflow.org/experiments with create and get verbs).

Request flow¶

The flow is identical for both principal types. The only difference is what the TokenReview returns and which subject kind is matched in the RoleBinding.

ServiceAccountOpenShift User

sequenceDiagram
    participant User as Tenant SA
    participant EH as EvalHub API
    participant K8s as Kubernetes API
    participant Job as Eval Job Pod

    User->>EH: POST /jobs (Bearer token + X-Tenant: team-a)
    EH->>K8s: TokenReview (validate token)
    K8s-->>EH: user=system:serviceaccount:team-a:team-a-user
    EH->>K8s: SAR (can SA create evaluations in team-a?)
    K8s-->>EH: Allowed (kind:ServiceAccount RoleBinding matched)
    EH->>K8s: Create Job in team-a (SA: evalhub-opendatahub-job)
    K8s-->>EH: Job created
    EH-->>User: 202 Accepted

    Job->>EH: POST /jobs/{id}/events (job SA token + X-Tenant: team-a)
    EH->>K8s: SAR (can job SA create status-events in team-a?)
    K8s-->>EH: Allowed
    EH-->>Job: 200 OK

sequenceDiagram
    participant User as OpenShift User (alice)
    participant EH as EvalHub API
    participant K8s as Kubernetes API
    participant Job as Eval Job Pod

    User->>EH: POST /jobs (OAuth token + X-Tenant: team-a)
    EH->>K8s: TokenReview (validate token)
    K8s-->>EH: user=alice
    EH->>K8s: SAR (can alice create evaluations in team-a?)
    K8s-->>EH: Allowed (kind:User RoleBinding matched)
    EH->>K8s: Create Job in team-a (SA: evalhub-opendatahub-job)
    K8s-->>EH: Job created
    EH-->>User: 202 Accepted

    Job->>EH: POST /jobs/{id}/events (job SA token + X-Tenant: team-a)
    EH->>K8s: SAR (can job SA create status-events in team-a?)
    K8s-->>EH: Allowed
    EH-->>Job: 200 OK

Note that the job pod always runs as the operator-provisioned job ServiceAccount (evalhub-opendatahub-job), regardless of whether the submitter was a User or a ServiceAccount.

RBAC reference¶

ClusterRoles (installed by operator)¶

ClusterRole	Purpose	Key permissions
`evalhub-auth-reviewer-role`	Token and SAR validation	`tokenreviews`, `subjectaccessreviews`
`evalhub-jobs-writer`	Create evaluation jobs	`batch/jobs` (create, delete)
`evalhub-job-config`	Manage job config	`configmaps` (create, get, update, delete)
`evalhub-providers-access`	Providers endpoint SAR	`providers` (get)
`evalhub-collections-access`	Collections endpoint SAR	`collections` (get)
`evalhub-mlflow-access`	API server MLFlow access	`experiments` (create, get, list, update, delete)
`evalhub-mlflow-jobs-access`	Job pod MLFlow access	`experiments` (create, get, list)

Per-tenant resources (operator-provisioned)¶

For each namespace labelled with evalhub.trustyai.opendatahub.io/tenant, the operator creates:

Resource	Name pattern	Purpose
ServiceAccount	`{name}-{evalhub-ns}-job`	Identity for job pods
Role	`{name}-{evalhub-ns}-job-access-role`	Allows `status-events/create`
RoleBinding	`{name}-{evalhub-ns}-job-access-rb`	Binds job SA to access Role
RoleBinding	`{name}-{tenant-ns}-job-writer-rb`	Binds API SA to jobs-writer ClusterRole
RoleBinding	`{name}-{tenant-ns}-job-config-rb`	Binds API SA to job-config ClusterRole
RoleBinding	`{name}-{evalhub-ns}-mlflow-job-rb`	Binds job SA to MLFlow ClusterRole
ConfigMap	`{name}-service-ca`	Service CA for TLS (OpenShift)

ServiceAccount naming¶

The job SA name includes the EvalHub instance namespace to prevent collisions when multiple EvalHub instances (potentially with the same CR name in different namespaces) create jobs in the same tenant namespace:

{evalhub-cr-name}-{evalhub-namespace}-job

For example, an EvalHub CR named evalhub in namespace opendatahub produces:

evalhub-opendatahub-job

Verifying the setup¶

Check operator-provisioned resources¶

After labelling a namespace with evalhub.trustyai.opendatahub.io/tenant, the operator should create the job SA and RoleBindings within a short time. Verify:

# Ensure the namespace has the tenant label
oc get namespace team-a --show-labels | grep evalhub.trustyai.opendatahub.io/tenant

# Verify job SA exists in tenant namespace (created by operator)
oc get sa evalhub-opendatahub-job -n team-a

# Verify RoleBindings (created by operator)
oc get rolebindings -n team-a | grep evalhub

Test permissions for the tenant principal:

ServiceAccountOpenShift User

# Test access in own tenant namespace
oc auth can-i create evaluations.trustyai.opendatahub.io \
  -n team-a \
  --as=system:serviceaccount:team-a:team-a-user

# Test cross-tenant denial
oc auth can-i create evaluations.trustyai.opendatahub.io \
  -n team-b \
  --as=system:serviceaccount:team-a:team-a-user

# Test access in own tenant namespace
oc auth can-i create evaluations.trustyai.opendatahub.io \
  -n team-a \
  --as=alice

# Test cross-tenant denial
oc auth can-i create evaluations.trustyai.opendatahub.io \
  -n team-b \
  --as=alice

Check job execution¶

# List jobs in tenant namespace
oc get jobs -n team-a -l app.kubernetes.io/part-of=eval-hub

# Check job pod SA
oc get pod -n team-a -l app.kubernetes.io/part-of=eval-hub \
  -o jsonpath='{.items[0].spec.serviceAccountName}'
# Expected: evalhub-opendatahub-job

Troubleshooting¶

403 Forbidden on API calls¶

The SAR check is failing. Verify:

The X-Tenant header matches a namespace where the principal has a RoleBinding
The Role includes the correct resources and verbs
The RoleBinding subjects[].kind matches the principal type (ServiceAccount or User)

ServiceAccountOpenShift User

oc auth can-i create evaluations.trustyai.opendatahub.io \
  -n team-a \
  --as=system:serviceaccount:team-a:team-a-user -v=6

oc auth can-i create evaluations.trustyai.opendatahub.io \
  -n team-a \
  --as=alice -v=6

Also confirm the RoleBinding subject kind is User and has no namespace field:

oc get rolebinding evalhub-evaluator-binding -n team-a -o yaml | grep -A5 subjects

Jobs not created in tenant namespace¶

Verify the EvalHub API SA has the jobs-writer and job-config RoleBindings in the tenant namespace:

oc get rolebindings -n team-a -o wide | grep -E "jobs-writer|job-config"

Job pods failing to post status events¶

The job SA needs the status-events/create permission in the tenant namespace:

oc auth can-i create status-events.trustyai.opendatahub.io \
  -n team-a \
  --as=system:serviceaccount:team-a:evalhub-opendatahub-job

Next steps¶

OpenShift Setup -- Base deployment guide
Architecture -- System architecture overview