Evaluation Event Violations

EvalHub emits a dedicated Kubernetes signal when an evaluation benchmark finishes but its measured score falls below the configured threshold. This lets cluster operators, GitOps pipelines, and alerting systems react to quality gate failures without polling the EvalHub API.

What triggers a violation

A threshold violation fires when a benchmark completes successfully at the infrastructure level but its primary score does not meet the configured pass_criteria.threshold. Configure the threshold per benchmark in the job YAML:

benchmarks:
  - id: hellaswag
    provider_id: lighteval
    primary_score:
      metric: hellaswag.acc_norm
    pass_criteria:
      threshold: 0.75
    parameters:
      num_examples: 100

primary_score is required

pass_criteria.threshold only takes effect when EvalHub knows which metric to evaluate against. Specify primary_score.metric in the benchmark config (or configure it on the provider benchmark). If neither is set, the threshold check is silently skipped.

When the measured score is below threshold, EvalHub calls NotifyThresholdViolation, which:

1. Sets the evaluation-phase label on the backing Job to ThresholdViolated
2. Emits a Warning Kubernetes Event with reason EvaluationThresholdViolated

Both operations are best-effort and do not affect the overall job completion state.

The signals

Signal	Kubernetes kind	Key / reason
Phase label	Job label	trustyai.opendatahub.io/evaluation-phase = ThresholdViolated
Violation event	Event	EvaluationThresholdViolated (Warning)

evaluation-phase label

The label trustyai.opendatahub.io/evaluation-phase is set to ThresholdViolated when the score check fails. This value sits alongside the standard lifecycle phases:

Value	When set
Pending	At Job creation
Running	When the adapter container starts processing
Completed	When the benchmark finishes and meets its threshold
ThresholdViolated	When the benchmark finishes but the score is below the threshold
Failed	When the adapter exits non-zero or an infrastructure failure is detected

ThresholdViolated vs Failed

ThresholdViolated means the benchmark ran to completion and produced a score — the score just did not meet the quality bar. Failed means the benchmark could not complete (adapter crash, OOM, unreachable model, etc.). They are distinct phases and produce distinct events.

Query by phase

# All jobs that violated their threshold
kubectl get jobs -n <tenant-namespace> \
  -l trustyai.opendatahub.io/evaluation-phase=ThresholdViolated \
  -L trustyai.opendatahub.io/evaluation-phase

NAME                   STATUS     COMPLETIONS   DURATION   AGE   EVALUATION-PHASE
eval-abc123-...        Complete   1/1           4m10s      6m    ThresholdViolated

Kubernetes Event

EvalHub emits a Warning Event against the backing Job when a threshold violation is detected.

Reason	Type	When emitted
EvaluationThresholdViolated	Warning	Score falls below pass_criteria.threshold

The event message carries the metric name, measured value, and configured threshold:

metric=acc actual=0.3412 threshold=0.9900

# Violation events for a specific job
kubectl get events -n <tenant-namespace> \
  --field-selector involvedObject.name=<job-name>,involvedObject.kind=Job,reason=EvaluationThresholdViolated \
  --sort-by=.lastTimestamp

# All threshold violation events across the namespace
kubectl get events -n <tenant-namespace> \
  --field-selector reason=EvaluationThresholdViolated \
  --sort-by=.lastTimestamp

Tip

Combine threshold violation events with the lifecycle event stream to get a full picture. Both EvaluationCompleted (the benchmark finished) and EvaluationThresholdViolated (the score failed the check) will appear for the same job.

Use cases

Quality gates — block downstream pipeline steps from consuming an evaluation result until the phase label confirms the benchmark passed. Use the exact label selector trustyai.opendatahub.io/evaluation-phase=Completed; only the Completed phase permits downstream consumption, and no EvalHub API token is required:

kubectl wait jobs -n <tenant-namespace> \
  -l job_id=<id>,trustyai.opendatahub.io/evaluation-phase=Completed \
  --for=condition=Complete --timeout=600s

Policy enforcement — Kyverno or OPA/Gatekeeper policies can gate on the evaluation-phase label before admitting downstream resources (e.g. model serving deployments) that depend on a passing evaluation.

Alerting — route EvaluationThresholdViolated Warning events to Alertmanager or a webhook via the Kubernetes events API to receive immediate notification when a model scores below your quality bar. The event message includes the actual score and threshold, so alert annotations can surface the values directly.

Audit trail — use kubectl get events filtered by reason=EvaluationThresholdViolated to investigate recent quality gate failures; events include the metric name, actual score, and timestamp. Note that Kubernetes event retention is configurable and events expire, so this is suitable only for short-term investigation. For a durable audit record, export events to persistent storage (e.g. Loki, Elasticsearch, or a custom sink) as they are emitted.