Skip to content

Collections

A collection is a named set of benchmarks that can be evaluated together as a single job. Each benchmark in a collection can have its own weight, primary score metric, and pass criteria, and the collection itself can define an overall pass threshold. This lets you define what “good” means for your use case — for example, a safety collection that requires a model to score above 0.75 across weighted safety benchmarks.

Browse the Collection Catalog for curated and non-curated collections and their benchmarks.

ConceptDescription
WeightRelative importance of a benchmark in the collection’s aggregate score. Defaults to 1 when omitted. Setting 0 is also treated as 1.
Primary scoreWhich metric from a benchmark’s results to use as the representative score (e.g. acc, f1, attack_success_rate).
Pass criteriaA threshold value. A benchmark passes when its primary score meets or exceeds the threshold (or is at or below, if lower_is_better).
Collection thresholdAn overall pass threshold for the entire collection. The aggregate score (weighted average) must meet or exceed this value for the job to pass.

When a collection-based job completes:

  1. Each benchmark’s primary score is extracted from its results using the configured metric name.
  2. If lower_is_better is set, the score is flipped to 1 - score for aggregation.
  3. Each score is multiplied by its weight and summed.
  4. The aggregate score = sum of weighted scores / sum of weights.
  5. The aggregate score is compared against the pass threshold to determine if the job passes.

The pass threshold can be set in multiple places. When more than one is present, the most specific one wins:

PrioritySourceExample use case
1 (highest)pass_criteria.threshold on the job request”Just this run, I want a stricter bar of 0.9”
2pass_criteria.threshold on the collection definitionThe collection’s default bar (e.g. 0.758 for safety)
3 (fallback)Hard-coded default: 0.5Neither the job nor the collection defines a threshold

EvalHub exposes two collection scopes. The resource object on every response identifies the collection ID, owner, tenant, and timestamps.

System collectionsTenant collections
Created byInstalled and configured by an administratorCreated or copied through the API
resource.ownersystemThe creating user
VisibilityAvailable across tenantsAvailable within the owning tenant
Mutable through the APINoYes
Listingscope=system filterscope=tenant filter

When scope is omitted, the list endpoint returns both scopes visible to the requesting tenant.

System collections are read-only through the collection API. PUT, PATCH, and DELETE attempts return 403 Forbidden. To customize a system collection, copy it to a tenant collection first. Administrators of operator-managed EvalHub deployments can control which system collections are installed and their curation priority through the Administration guide.

Built-in collections in the current catalog

Section titled “Built-in collections in the current catalog”
CollectionCategoryPass thresholdBenchmarksDescription
standard-llm-evals-v1general0.4512Core capability benchmarks (MMLU, ARC, HellaSwag, etc.)
leaderboard-v2general38.06Open LLM Leaderboard v2 benchmarks
reasoning-v1reasoning0.386Mathematical and logical reasoning
coding-v1code0.251Code generation and understanding
instruction-following-v1instruction_following0.505Instruction-following ability
instruction-output-v1instruction_output0.3252JSON schema compliance + IFBench OOD instruction constraints
knowledge-reasoning-v1knowledge_reasoning0.2352Knowledge and reasoning benchmarks
document-understanding-v1document_understanding0.3514Grounded document understanding (RULER, DocVQA, etc.)
tool-use-v1tool_use0.601Tool use and function calling
software-v1software0.203Software engineering benchmarks
multimodal-v1multimodal0.501Multimodal understanding
trustworthiness-v1trustworthiness0.3252Trustworthiness and reliability
safety-and-fairness-v1safety0.7586Bias, toxicity, and safety
model-validationsafety0.751Security-focused validation (uses lower_is_better)
toxicity-and-ethical-principlessafety0.753Toxicity and ethical evaluation
long-context-v1long_context0.454Long-context understanding
open-telco-v1telecom0.4754Telecom domain benchmarks

A collection requires a name, at least one benchmark entry, and either a non-empty domains array or the deprecated category field. Each benchmark references an existing provider and benchmark pair.

{
"name": "My Safety Suite",
"description": "Custom safety evaluation for our models",
"domains": ["trustworthiness"],
"tags": ["safety", "production"],
"pass_criteria": {
"threshold": 0.8
},
"benchmarks": [
{
"id": "toxigen",
"provider_id": "lm_evaluation_harness",
"weight": 3,
"primary_score": {
"metric": "acc",
"lower_is_better": false
},
"pass_criteria": {
"threshold": 0.7
}
},
{
"id": "quick",
"provider_id": "garak",
"weight": 2,
"primary_score": {
"metric": "attack_success_rate",
"lower_is_better": true
},
"pass_criteria": {
"threshold": 0.1
}
}
]
}
POST /api/v1/evaluations/collections
{
"name": "My Safety Suite",
"domains": ["trustworthiness"],
"tags": ["safety"],
"pass_criteria": { "threshold": 0.8 },
"benchmarks": [
{
"id": "toxigen",
"provider_id": "lm_evaluation_harness",
"weight": 3,
"primary_score": { "metric": "acc" },
"pass_criteria": { "threshold": 0.7 }
},
{
"id": "quick",
"provider_id": "garak",
"weight": 2,
"primary_score": { "metric": "attack_success_rate", "lower_is_better": true },
"pass_criteria": { "threshold": 0.1 }
}
]
}

When creating or updating a collection, the server validates the request and rejects it with 400 Bad Request if any of the following rules are violated:

  • name is required.
  • Either a non-empty domains array or the deprecated category field is required.
  • category is deprecated. When used for backwards compatibility, it must be between 1 and 128 characters.
  • At least one benchmark entry is required.
  • Each benchmark must reference a valid provider_id.
  • weight must be ≥ 0 (0 is treated as 1 during scoring).
  • pass_criteria.threshold must be present when pass_criteria is set (value of 0 is valid).
  • description must be between 1 and 1024 characters when set.
  • tags cannot contain , or | characters.

Classification metadata makes collections discoverable.

FieldMeaningPopulation behavior
domainsHigh-level evaluation domains, such as knowledge_and_reasoning, grounded_document_understanding, or trustworthinessExplicit collection value, or the union of values from its benchmarks
tasksEvaluation tasks, such as reasoning, rag, safety, or code_generationExplicit collection value, or the union of values from its benchmarks
modalitiesData modalities, such as text, vision, or multimodalExplicit collection value, or the union of values from its benchmarks
evaluation_targetsTypes of entities described by the evaluation, such as model or agentExplicit collection value, or the union of values from its benchmarks
industriesCollection-level business context, such as health, telco, financial, or governmentSet explicitly on the collection; it is not derived from benchmarks

If an explicitly supplied derived field is non-empty, EvalHub keeps that value instead of combining it with benchmark metadata. The returned benchmark-derived values are unions, so they describe everything represented by the collection; they do not guarantee that every benchmark supports every value.

category remains accepted for backwards compatibility but is deprecated. New collections should use domains.

To run all benchmarks in a collection, submit a job with a collection reference instead of listing individual benchmarks.

Some benchmarks require a judge or scorer model. See Using Judge and Scorer Models for the job parameters and Secret setup.

POST /api/v1/evaluations/jobs
{
"name": "safety-eval-llama3",
"model": {
"url": "http://my-model:8000/v1",
"name": "llama3"
},
"collection": {
"id": "safety-and-fairness-v1"
}
}

When submitting a collection-based job, you can override test_data_ref and hardware_config for specific benchmarks by including them in collection.benchmarks. You can also add new parameters keys that are not already defined in the collection. However, parameter keys that the collection already defines with non-empty values cannot be overridden — the collection’s values take precedence. The overrides are matched by id and provider_id. Weight, primary score, and pass criteria always come from the stored collection definition and cannot be overridden at run time.

{
"name": "collection-parameter-override-eval",
"model": { "url": "http://my-model:8000/v1", "name": "llama3" },
"collection": {
"id": "safety-and-fairness-v1",
"benchmarks": [
{
"id": "toxigen",
"provider_id": "lm_evaluation_harness",
"parameters": {
"num_fewshot": 0
}
}
]
}
}

Set pass_criteria at the job level to override the collection’s threshold for this run:

{
"name": "collection-threshold-override-eval",
"model": { "url": "http://my-model:8000/v1", "name": "llama3" },
"pass_criteria": { "threshold": 0.9 },
"collection": {
"id": "safety-and-fairness-v1"
}
}

Filter the jobs endpoint by collection ID to find jobs submitted from a particular collection:

Terminal window
curl -s --get "$EVALHUB_URL/api/v1/evaluations/jobs" \
-H "X-Tenant: my-team" \
--data-urlencode "collection_id=my-collection-id" | jq .

When EvalHub accepts a job for a tenant collection, it increments that collection’s status.run_count. System collections do not expose tenant-owned run status.

The job records the collection’s modification timestamp in collection.collection_updated_at. Compare it with the collection’s current resource.updated_at to determine whether two jobs used the same observed collection state:

Terminal window
jq '.collection.collection_updated_at' job.json
jq '.resource.updated_at' collection.json

GET /api/v1/evaluations/collections returns a paginated object. Use limit and offset, and follow the response links when present.

Terminal window
# All visible system and tenant collections, 25 at a time
curl -s "$EVALHUB_URL/api/v1/evaluations/collections?limit=25&offset=0" \
-H "X-Tenant: my-team" | jq '{total_count, items}'
# Only system collections
curl -s "$EVALHUB_URL/api/v1/evaluations/collections?scope=system" \
-H "X-Tenant: my-team" | jq .
# Only custom collections of this tenant
curl -s "$EVALHUB_URL/api/v1/evaluations/collections?scope=tenant" \
-H "X-Tenant: my-team" | jq .

You can filter by name, category (deprecated), tags, domains, tasks, modalities, industries, and evaluation_targets. Repeat an array-field parameter to require all supplied values. For example, this request returns only collections containing both domain values:

Terminal window
curl -s --get "$EVALHUB_URL/api/v1/evaluations/collections" \
-H "X-Tenant: my-team" \
--data-urlencode "domains=knowledge_and_reasoning" \
--data-urlencode "domains=trustworthiness" | jq .

The repeated-value behavior is AND, not OR. The same rule applies to repeated tasks, modalities, industries, and evaluation_targets parameters.

Sorting by curation priority is opt-in:

Terminal window
curl -s "$EVALHUB_URL/api/v1/evaluations/collections?scope=system&sort_by=curation_order&limit=25&offset=0" \
-H "X-Tenant: my-team" | jq .

EvalHub sorts the full filtered result before pagination. Positive curation_order values sort in ascending order, and zero or absent values sort last. When sort_by is omitted, the existing default ordering is unchanged.

Terminal window
curl -s $EVALHUB_URL/api/v1/evaluations/collections/safety-and-fairness-v1 \
-H "X-Tenant: my-team" | jq .

See the Server API reference for the complete request and response schemas.

Use the clone endpoint to create a tenant-scoped copy of any visible collection. This is the normal way to customize a read-only system collection.

Terminal window
curl -s -X POST \
"$EVALHUB_URL/api/v1/evaluations/collections/safety-and-fairness-v1/clones" \
-H "Content-Type: application/json" \
-H "X-Tenant: my-team" \
-H "X-User: me" \
-d '{
"name": "My safety release gate",
"domains": ["trustworthiness"],
"tags": ["release-gate"]
}' | jq .

The request body is optional. Omitted configuration fields are inherited from the source. The new resource:

  • receives a new ID and tenant ownership;
  • records the source ID in immutable derived_from;
  • starts with curation_order reset to 0, even if the source is curated;
  • starts with pinned_order: 0 and status.run_count: 0.

The clone body accepts configuration overrides but rejects unknown fields and curation_order. A malformed body or invalid merged collection returns 400 Bad Request; an unavailable source returns 404 Not Found; authentication or authorization failures return 401 or 403.

Tenant collections can be updated via PUT (full replace) or PATCH (partial update).

System collections are read-only. Attempts to update a system collection return 403 Forbidden. Use the clone endpoint when you need an editable tenant copy.

Terminal window
curl -s -X PATCH $EVALHUB_URL/api/v1/evaluations/collections/my-collection-id \
-H "Content-Type: application/json-patch+json" \
-H "X-Tenant: my-team" \
-H "X-User: me" \
-d '[
{"op": "replace", "path": "/pass_criteria", "value": {"threshold": 0.85}},
{"op": "replace", "path": "/name", "value": "Updated Safety Suite"},
{"op": "replace", "path": "/pinned_order", "value": 1}
]'

Patchable fields are /name, /description, /tags, /custom, /category, /benchmarks, /pass_criteria, /domains, /tasks, /modalities, /industries, /evaluation_targets, and /pinned_order.

pinned_order is a user-controlled priority for tenant collections. curation_order is the administrator-controlled priority for system collections and is not a supported tenant update field. For both fields, 0 means no priority and lower positive values have higher priority.

PUT replaces the complete collection configuration, so include every value that should remain.

Terminal window
curl -s -X DELETE $EVALHUB_URL/api/v1/evaluations/collections/my-collection-id \
-H "X-Tenant: my-team" \
-H "X-User: me"

Attempts to delete system collections return 403 Forbidden.

When a job is submitted with a collection reference, the results include both per-benchmark and aggregate scoring:

{
"results": {
"test": {
"score": 0.89,
"threshold": 0.8,
"pass": true
},
"benchmarks": [
{
"test": {
"primary_score": 0.85,
"primary_score_metric": "acc",
"threshold": 0.7,
"pass": true
}
},
{
"test": {
"primary_score": 0.05,
"primary_score_metric": "attack_success_rate",
"threshold": 0.1,
"pass": true
}
}
]
}
}
  • results.test — the aggregate weighted score and overall pass/fail
  • results.benchmarks[].test — per-benchmark primary score and individual pass/fail