Collections
A collection is a named set of benchmarks that can be evaluated together as a single job. Each benchmark in a collection can have its own weight, primary score metric, and pass criteria, and the collection itself can define an overall pass threshold. This lets you define what “good” means for your use case — for example, a safety collection that requires a model to score above 0.75 across weighted safety benchmarks.
Browse the Collection Catalog for curated and non-curated collections and their benchmarks.
Key concepts
Section titled “Key concepts”| Concept | Description |
|---|---|
| Weight | Relative importance of a benchmark in the collection’s aggregate score. Defaults to 1 when omitted. Setting 0 is also treated as 1. |
| Primary score | Which metric from a benchmark’s results to use as the representative score (e.g. acc, f1, attack_success_rate). |
| Pass criteria | A threshold value. A benchmark passes when its primary score meets or exceeds the threshold (or is at or below, if lower_is_better). |
| Collection threshold | An overall pass threshold for the entire collection. The aggregate score (weighted average) must meet or exceed this value for the job to pass. |
How scoring works
Section titled “How scoring works”When a collection-based job completes:
- Each benchmark’s primary score is extracted from its results using the configured metric name.
- If
lower_is_betteris set, the score is flipped to1 - scorefor aggregation. - Each score is multiplied by its weight and summed.
- The aggregate score = sum of weighted scores / sum of weights.
- The aggregate score is compared against the pass threshold to determine if the job passes.
The pass threshold can be set in multiple places. When more than one is present, the most specific one wins:
| Priority | Source | Example use case |
|---|---|---|
| 1 (highest) | pass_criteria.threshold on the job request | ”Just this run, I want a stricter bar of 0.9” |
| 2 | pass_criteria.threshold on the collection definition | The collection’s default bar (e.g. 0.758 for safety) |
| 3 (fallback) | Hard-coded default: 0.5 | Neither the job nor the collection defines a threshold |
System vs tenant collections
Section titled “System vs tenant collections”EvalHub exposes two collection scopes. The resource object on every response identifies the collection ID, owner, tenant, and timestamps.
| System collections | Tenant collections | |
|---|---|---|
| Created by | Installed and configured by an administrator | Created or copied through the API |
resource.owner | system | The creating user |
| Visibility | Available across tenants | Available within the owning tenant |
| Mutable through the API | No | Yes |
| Listing | scope=system filter | scope=tenant filter |
When scope is omitted, the list endpoint returns both scopes visible to the requesting tenant.
System collections are read-only through the collection API. PUT, PATCH, and DELETE attempts return 403 Forbidden. To customize a system collection, copy it to a tenant collection first. Administrators of operator-managed EvalHub deployments can control which system collections are installed and their curation priority through the Administration guide.
Built-in collections in the current catalog
Section titled “Built-in collections in the current catalog”| Collection | Category | Pass threshold | Benchmarks | Description |
|---|---|---|---|---|
standard-llm-evals-v1 | general | 0.45 | 12 | Core capability benchmarks (MMLU, ARC, HellaSwag, etc.) |
leaderboard-v2 | general | 38.0 | 6 | Open LLM Leaderboard v2 benchmarks |
reasoning-v1 | reasoning | 0.38 | 6 | Mathematical and logical reasoning |
coding-v1 | code | 0.25 | 1 | Code generation and understanding |
instruction-following-v1 | instruction_following | 0.50 | 5 | Instruction-following ability |
instruction-output-v1 | instruction_output | 0.325 | 2 | JSON schema compliance + IFBench OOD instruction constraints |
knowledge-reasoning-v1 | knowledge_reasoning | 0.235 | 2 | Knowledge and reasoning benchmarks |
document-understanding-v1 | document_understanding | 0.35 | 14 | Grounded document understanding (RULER, DocVQA, etc.) |
tool-use-v1 | tool_use | 0.60 | 1 | Tool use and function calling |
software-v1 | software | 0.20 | 3 | Software engineering benchmarks |
multimodal-v1 | multimodal | 0.50 | 1 | Multimodal understanding |
trustworthiness-v1 | trustworthiness | 0.325 | 2 | Trustworthiness and reliability |
safety-and-fairness-v1 | safety | 0.758 | 6 | Bias, toxicity, and safety |
model-validation | safety | 0.75 | 1 | Security-focused validation (uses lower_is_better) |
toxicity-and-ethical-principles | safety | 0.75 | 3 | Toxicity and ethical evaluation |
long-context-v1 | long_context | 0.45 | 4 | Long-context understanding |
open-telco-v1 | telecom | 0.475 | 4 | Telecom domain benchmarks |
Creating a collection
Section titled “Creating a collection”Collection structure
Section titled “Collection structure”A collection requires a name, at least one benchmark entry, and either a non-empty domains array or the deprecated category field. Each benchmark references an existing provider and benchmark pair.
{ "name": "My Safety Suite", "description": "Custom safety evaluation for our models", "domains": ["trustworthiness"], "tags": ["safety", "production"], "pass_criteria": { "threshold": 0.8 }, "benchmarks": [ { "id": "toxigen", "provider_id": "lm_evaluation_harness", "weight": 3, "primary_score": { "metric": "acc", "lower_is_better": false }, "pass_criteria": { "threshold": 0.7 } }, { "id": "quick", "provider_id": "garak", "weight": 2, "primary_score": { "metric": "attack_success_rate", "lower_is_better": true }, "pass_criteria": { "threshold": 0.1 } } ]}Via REST API
Section titled “Via REST API”POST /api/v1/evaluations/collections
{ "name": "My Safety Suite", "domains": ["trustworthiness"], "tags": ["safety"], "pass_criteria": { "threshold": 0.8 }, "benchmarks": [ { "id": "toxigen", "provider_id": "lm_evaluation_harness", "weight": 3, "primary_score": { "metric": "acc" }, "pass_criteria": { "threshold": 0.7 } }, { "id": "quick", "provider_id": "garak", "weight": 2, "primary_score": { "metric": "attack_success_rate", "lower_is_better": true }, "pass_criteria": { "threshold": 0.1 } } ]}curl -s -X POST $EVALHUB_URL/api/v1/evaluations/collections \ -H "Content-Type: application/json" \ -H "X-Tenant: my-team" \ -H "X-User: me" \ -d '{ "name": "My Safety Suite", "domains": ["trustworthiness"], "tags": ["safety"], "pass_criteria": { "threshold": 0.8 }, "benchmarks": [ { "id": "toxigen", "provider_id": "lm_evaluation_harness", "weight": 3, "primary_score": { "metric": "acc" }, "pass_criteria": { "threshold": 0.7 } }, { "id": "quick", "provider_id": "garak", "weight": 2, "primary_score": { "metric": "attack_success_rate", "lower_is_better": true }, "pass_criteria": { "threshold": 0.1 } } ] }'Define the collection in a YAML file:
name: My Safety Suitedomains: - trustworthinesstags: - safetypass_criteria: threshold: 0.8benchmarks: - id: toxigen provider_id: lm_evaluation_harness weight: 3 primary_score: metric: acc pass_criteria: threshold: 0.7 - id: quick provider_id: garak weight: 2 primary_score: metric: attack_success_rate lower_is_better: true pass_criteria: threshold: 0.1evalhub collections create --file my-safety-suite.yamlfrom evalhub import SyncEvalHubClientfrom evalhub.models.api import ( CollectionCreateRequest, BenchmarkReference, PrimaryScore, PassCriteria,)
client = SyncEvalHubClient(base_url="http://evalhub:8080")
request = CollectionCreateRequest( name="My Safety Suite", domains=["trustworthiness"], tags=["safety"], pass_criteria=PassCriteria(threshold=0.8), benchmarks=[ BenchmarkReference( id="toxigen", provider_id="lm_evaluation_harness", weight=3, primary_score=PrimaryScore(metric="acc"), pass_criteria=PassCriteria(threshold=0.7), ), BenchmarkReference( id="quick", provider_id="garak", weight=2, primary_score=PrimaryScore(metric="attack_success_rate", lower_is_better=True), pass_criteria=PassCriteria(threshold=0.1), ), ],)collection = client.collections.create(request.model_dump(mode="json"))Validation
Section titled “Validation”When creating or updating a collection, the server validates the request and rejects it with 400 Bad Request if any of the following rules are violated:
nameis required.- Either a non-empty
domainsarray or the deprecatedcategoryfield is required. categoryis deprecated. When used for backwards compatibility, it must be between 1 and 128 characters.- At least one benchmark entry is required.
- Each benchmark must reference a valid
provider_id. weightmust be ≥ 0 (0 is treated as 1 during scoring).pass_criteria.thresholdmust be present whenpass_criteriais set (value of 0 is valid).descriptionmust be between 1 and 1024 characters when set.tagscannot contain,or|characters.
Classifying collections
Section titled “Classifying collections”Classification metadata makes collections discoverable.
| Field | Meaning | Population behavior |
|---|---|---|
domains | High-level evaluation domains, such as knowledge_and_reasoning, grounded_document_understanding, or trustworthiness | Explicit collection value, or the union of values from its benchmarks |
tasks | Evaluation tasks, such as reasoning, rag, safety, or code_generation | Explicit collection value, or the union of values from its benchmarks |
modalities | Data modalities, such as text, vision, or multimodal | Explicit collection value, or the union of values from its benchmarks |
evaluation_targets | Types of entities described by the evaluation, such as model or agent | Explicit collection value, or the union of values from its benchmarks |
industries | Collection-level business context, such as health, telco, financial, or government | Set explicitly on the collection; it is not derived from benchmarks |
If an explicitly supplied derived field is non-empty, EvalHub keeps that value instead of combining it with benchmark metadata. The returned benchmark-derived values are unions, so they describe everything represented by the collection; they do not guarantee that every benchmark supports every value.
category remains accepted for backwards compatibility but is deprecated. New collections should use domains.
Running a collection
Section titled “Running a collection”To run all benchmarks in a collection, submit a job with a collection reference instead of listing individual benchmarks.
Some benchmarks require a judge or scorer model. See Using Judge and Scorer Models for the job parameters and Secret setup.
POST /api/v1/evaluations/jobs
{ "name": "safety-eval-llama3", "model": { "url": "http://my-model:8000/v1", "name": "llama3" }, "collection": { "id": "safety-and-fairness-v1" }}curl -s -X POST $EVALHUB_URL/api/v1/evaluations/jobs \ -H "Content-Type: application/json" \ -H "X-Tenant: my-team" \ -H "X-User: me" \ -d '{ "name": "safety-eval-llama3", "model": { "url": "http://my-model:8000/v1", "name": "llama3" }, "collection": { "id": "safety-and-fairness-v1" } }'evalhub collections run safety-and-fairness-v1 \ --model-url http://my-model:8000/v1 \ --model-name llama3from evalhub.models.api import ( JobSubmissionRequest, ModelConfig, CollectionRef,)
job = client.jobs.submit(JobSubmissionRequest( name="safety-eval-llama3", model=ModelConfig(url="http://my-model:8000/v1", name="llama3"), collection=CollectionRef(id="safety-and-fairness-v1"),))Overriding parameters at run time
Section titled “Overriding parameters at run time”When submitting a collection-based job, you can override test_data_ref and hardware_config for specific benchmarks by including them in collection.benchmarks. You can also add new parameters keys that are not already defined in the collection. However, parameter keys that the collection already defines with non-empty values cannot be overridden — the collection’s values take precedence. The overrides are matched by id and provider_id. Weight, primary score, and pass criteria always come from the stored collection definition and cannot be overridden at run time.
{ "name": "collection-parameter-override-eval", "model": { "url": "http://my-model:8000/v1", "name": "llama3" }, "collection": { "id": "safety-and-fairness-v1", "benchmarks": [ { "id": "toxigen", "provider_id": "lm_evaluation_harness", "parameters": { "num_fewshot": 0 } } ] }}Overriding the pass threshold
Section titled “Overriding the pass threshold”Set pass_criteria at the job level to override the collection’s threshold for this run:
{ "name": "collection-threshold-override-eval", "model": { "url": "http://my-model:8000/v1", "name": "llama3" }, "pass_criteria": { "threshold": 0.9 }, "collection": { "id": "safety-and-fairness-v1" }}Tracking collection runs
Section titled “Tracking collection runs”Filter the jobs endpoint by collection ID to find jobs submitted from a particular collection:
curl -s --get "$EVALHUB_URL/api/v1/evaluations/jobs" \ -H "X-Tenant: my-team" \ --data-urlencode "collection_id=my-collection-id" | jq .When EvalHub accepts a job for a tenant collection, it increments that collection’s status.run_count. System collections do not expose tenant-owned run status.
The job records the collection’s modification timestamp in collection.collection_updated_at. Compare it with the collection’s current resource.updated_at to determine whether two jobs used the same observed collection state:
jq '.collection.collection_updated_at' job.jsonjq '.resource.updated_at' collection.jsonBrowsing collections
Section titled “Browsing collections”GET /api/v1/evaluations/collections returns a paginated object. Use limit and offset, and follow the response links when present.
# All visible system and tenant collections, 25 at a timecurl -s "$EVALHUB_URL/api/v1/evaluations/collections?limit=25&offset=0" \ -H "X-Tenant: my-team" | jq '{total_count, items}'
# Only system collectionscurl -s "$EVALHUB_URL/api/v1/evaluations/collections?scope=system" \ -H "X-Tenant: my-team" | jq .
# Only custom collections of this tenantcurl -s "$EVALHUB_URL/api/v1/evaluations/collections?scope=tenant" \ -H "X-Tenant: my-team" | jq .You can filter by name, category (deprecated), tags, domains, tasks, modalities, industries, and evaluation_targets. Repeat an array-field parameter to require all supplied values. For example, this request returns only collections containing both domain values:
curl -s --get "$EVALHUB_URL/api/v1/evaluations/collections" \ -H "X-Tenant: my-team" \ --data-urlencode "domains=knowledge_and_reasoning" \ --data-urlencode "domains=trustworthiness" | jq .The repeated-value behavior is AND, not OR. The same rule applies to repeated tasks, modalities, industries, and evaluation_targets parameters.
Sort by curation priority
Section titled “Sort by curation priority”Sorting by curation priority is opt-in:
curl -s "$EVALHUB_URL/api/v1/evaluations/collections?scope=system&sort_by=curation_order&limit=25&offset=0" \ -H "X-Tenant: my-team" | jq .EvalHub sorts the full filtered result before pagination. Positive curation_order values sort in ascending order, and zero or absent values sort last. When sort_by is omitted, the existing default ordering is unchanged.
Get collection details
Section titled “Get collection details”curl -s $EVALHUB_URL/api/v1/evaluations/collections/safety-and-fairness-v1 \ -H "X-Tenant: my-team" | jq .evalhub collections describe safety-and-fairness-v1
# JSON outputevalhub collections describe safety-and-fairness-v1 --format jsoncollection = client.collections.get("safety-and-fairness-v1")print(f"Name: {collection.name}")print(f"Domains: {collection.domains}")print(f"Benchmarks: {len(collection.benchmarks)}")if collection.pass_criteria: print(f"Pass threshold: {collection.pass_criteria.threshold}")See the Server API reference for the complete request and response schemas.
Copying a collection
Section titled “Copying a collection”Use the clone endpoint to create a tenant-scoped copy of any visible collection. This is the normal way to customize a read-only system collection.
curl -s -X POST \ "$EVALHUB_URL/api/v1/evaluations/collections/safety-and-fairness-v1/clones" \ -H "Content-Type: application/json" \ -H "X-Tenant: my-team" \ -H "X-User: me" \ -d '{ "name": "My safety release gate", "domains": ["trustworthiness"], "tags": ["release-gate"] }' | jq .The request body is optional. Omitted configuration fields are inherited from the source. The new resource:
- receives a new ID and tenant ownership;
- records the source ID in immutable
derived_from; - starts with
curation_orderreset to0, even if the source is curated; - starts with
pinned_order: 0andstatus.run_count: 0.
The clone body accepts configuration overrides but rejects unknown fields and curation_order. A malformed body or invalid merged collection returns 400 Bad Request; an unavailable source returns 404 Not Found; authentication or authorization failures return 401 or 403.
Updating a collection
Section titled “Updating a collection”Tenant collections can be updated via PUT (full replace) or PATCH (partial update).
System collections are read-only. Attempts to update a system collection return 403 Forbidden. Use the clone endpoint when you need an editable tenant copy.
PATCH example
Section titled “PATCH example”curl -s -X PATCH $EVALHUB_URL/api/v1/evaluations/collections/my-collection-id \ -H "Content-Type: application/json-patch+json" \ -H "X-Tenant: my-team" \ -H "X-User: me" \ -d '[ {"op": "replace", "path": "/pass_criteria", "value": {"threshold": 0.85}}, {"op": "replace", "path": "/name", "value": "Updated Safety Suite"}, {"op": "replace", "path": "/pinned_order", "value": 1} ]'Patchable fields are /name, /description, /tags, /custom, /category, /benchmarks, /pass_criteria, /domains, /tasks, /modalities, /industries, /evaluation_targets, and /pinned_order.
pinned_order is a user-controlled priority for tenant collections. curation_order is the administrator-controlled priority for system collections and is not a supported tenant update field. For both fields, 0 means no priority and lower positive values have higher priority.
PUT replaces the complete collection configuration, so include every value that should remain.
Deleting a collection
Section titled “Deleting a collection”curl -s -X DELETE $EVALHUB_URL/api/v1/evaluations/collections/my-collection-id \ -H "X-Tenant: my-team" \ -H "X-User: me"evalhub collections delete my-collection-id
# Skip confirmation promptevalhub collections delete my-collection-id --yesclient.collections.delete("my-collection-id")Attempts to delete system collections return 403 Forbidden.
Interpreting results
Section titled “Interpreting results”When a job is submitted with a collection reference, the results include both per-benchmark and aggregate scoring:
{ "results": { "test": { "score": 0.89, "threshold": 0.8, "pass": true }, "benchmarks": [ { "test": { "primary_score": 0.85, "primary_score_metric": "acc", "threshold": 0.7, "pass": true } }, { "test": { "primary_score": 0.05, "primary_score_metric": "attack_success_rate", "threshold": 0.1, "pass": true } } ] }}results.test— the aggregate weighted score and overall pass/failresults.benchmarks[].test— per-benchmark primary score and individual pass/fail