API docs by Redocly

Eval Hub (0.4.0)

Download OpenAPI specification:

License: Apache 2.0

API REST server for evaluation backend orchestration

Evaluations

Evaluation job management endpoints

Create Evaluation

Create and execute evaluation request using the simplified benchmark schema.

Request Body schema: application/json
required
One of
name
required
string

The evaluation job name.

description
string

The evaluation job description.

tags
Array of strings

The evaluation job tags.

required
object (ModelRef)

The model to evaluate.

required
Array of objects (EvaluationBenchmarkConfig)

The evaluation benchmarks to run.

object (PassCriteriaWithDefault)

The overall pass criteria for the evaluation job.

object (ExperimentConfig)

The MLFlow experiment configuration. When provided, the evaluation job will be tracked in MLFlow.

object (EvaluationExports)

Optional exports configuration for the evaluation job. When provided, the evaluation job results will be exported to the specified location.

object (QueueConfig)

Optional scheduling queue for Kubernetes-backed evaluation jobs (e.g. Kueue).

object

Custom request data. This can be used for user specific job data.

Responses

Request samples

Content type
application/json
Example
{
  • "name": "string",
  • "description": "string",
  • "tags": [
    ],
  • "model": {
    },
  • "benchmarks": [
    ],
  • "pass_criteria": {
    },
  • "experiment": {
    },
  • "exports": {
    },
  • "queue": {
    },
  • "custom": { }
}

Response samples

Content type
application/json
Example
{
  • "name": "string",
  • "description": "string",
  • "tags": [
    ],
  • "model": {
    },
  • "benchmarks": [
    ],
  • "pass_criteria": {
    },
  • "experiment": {
    },
  • "exports": {
    },
  • "queue": {
    },
  • "custom": { },
  • "resource": {
    },
  • "status": {
    },
  • "results": {
    }
}

List Evaluations

List all evaluation requests.

query Parameters
limit
integer (Limit) [ 1 .. 100 ]
Default: 50

Maximum number of evaluations to return

offset
integer (Offset) >= 0
Default: 0

Offset for pagination

status
string (Status Filter)

Filter by status

name
string (Name)

Name to search for

tags
string (Tags)

Tags to search for

Responses

Response samples

Content type
application/json
{
  • "first": {
    },
  • "next": {
    },
  • "limit": 0,
  • "total_count": 0,
  • "items": [
    ],
  • "errors": [
    ]
}

Get Evaluation

Returns the evaluation job resource with the current status and results.

path Parameters
id
required
string (Id)

Responses

Response samples

Content type
application/json
Example
{
  • "name": "string",
  • "description": "string",
  • "tags": [
    ],
  • "model": {
    },
  • "benchmarks": [
    ],
  • "pass_criteria": {
    },
  • "experiment": {
    },
  • "exports": {
    },
  • "queue": {
    },
  • "custom": { },
  • "resource": {
    },
  • "status": {
    },
  • "results": {
    }
}

Cancel Evaluation

Cancel a running evaluation.

path Parameters
id
required
string (Id)
query Parameters
hard_delete
boolean (Hard Delete)
Default: false

If true, delete the evaluation job permanently so that GET /api/v1/evaluations/jobs/{id} will return a 404.

Responses

Response samples

Content type
application/json
{
  • "message": "The field 'state' is not valid.",
  • "message_code": "invalid_value",
  • "trace": "b12692e1-8582-4628-88ca-7a13fefb73e2"
}

Send a new evaluation status event

Send an evaluation job status or results. Note that this endpoint is internal and should not be used by clients, eventually it will not be possible to call this endpoint without a special token.

path Parameters
id
required
string (Id)
Request Body schema: application/json
required
required
object (BenchmarkStatusEvent)

Event payload to update benchmark status at runtime

Responses

Request samples

Content type
application/json

A simple request update with a completed status.

{
  • "benchmark_status_event": {
    }
}

Response samples

Content type
application/json
{
  • "message": "The field 'state' is not valid.",
  • "message_code": "invalid_value",
  • "trace": "b12692e1-8582-4628-88ca-7a13fefb73e2"
}

Collections

Benchmark collection management endpoints

List Collections

List all benchmark collections.

query Parameters
limit
integer (Limit) [ 1 .. 100 ]
Default: 50

Maximum number of collections to return

offset
integer (Offset) >= 0
Default: 0

Offset for pagination

name
string (Name)

Name to search for

category
string (Category)

Category to search for

tags
string (Tags)

Tags to search for

scope
string (Scope of collections)
Enum: "system" "tenant"

Set to system to get only system defined collections, or tenant to get only user defined collections. If scope is not provided, both system and user defined collections will be returned.

Responses

Response samples

Content type
application/json
{
  • "first": {
    },
  • "next": {
    },
  • "limit": 0,
  • "total_count": 0,
  • "items": [
    ]
}

Create Collection

Create a new collection.

Request Body schema: application/json
required
name
required
string

Collection name.

category
required
string

Collection category.

description
string

Optional description.

tags
Array of strings

Tags.

object

Custom key-value data.

object (PassCriteria)

Pass criteria for the collection.

required
Array of objects (CollectionBenchmarkConfig)

Benchmarks in the collection.

Responses

Request samples

Content type
application/json
{
  • "name": "string",
  • "category": "string",
  • "description": "string",
  • "tags": [
    ],
  • "custom": { },
  • "pass_criteria": {
    },
  • "benchmarks": [
    ]
}

Response samples

Content type
application/json
{
  • "resource": {
    },
  • "name": "string",
  • "category": "string",
  • "description": "string",
  • "tags": [
    ],
  • "custom": { },
  • "pass_criteria": {
    },
  • "benchmarks": [
    ]
}

Get Collection

Get details of a specific collection.

path Parameters
id
required
string (Collection Id)

Responses

Response samples

Content type
application/json
{
  • "resource": {
    },
  • "name": "string",
  • "category": "string",
  • "description": "string",
  • "tags": [
    ],
  • "custom": { },
  • "pass_criteria": {
    },
  • "benchmarks": [
    ]
}

Update Collection

Update an existing collection.

path Parameters
id
required
string (Collection Id)
Request Body schema: application/json
required
name
required
string

Collection name.

category
required
string

Collection category.

description
string

Optional description.

tags
Array of strings

Tags.

object

Custom key-value data.

object (PassCriteria)

Pass criteria for the collection.

required
Array of objects (CollectionBenchmarkConfig)

Benchmarks in the collection.

Responses

Request samples

Content type
application/json
{
  • "name": "string",
  • "category": "string",
  • "description": "string",
  • "tags": [
    ],
  • "custom": { },
  • "pass_criteria": {
    },
  • "benchmarks": [
    ]
}

Response samples

Content type
application/json
{
  • "resource": {
    },
  • "name": "string",
  • "category": "string",
  • "description": "string",
  • "tags": [
    ],
  • "custom": { },
  • "pass_criteria": {
    },
  • "benchmarks": [
    ]
}

Patch Collection

Partially update an existing collection.

path Parameters
id
required
string (Collection Id)
Request Body schema: application/json
required
Array
op
required
string (PatchOp)
Enum: "replace" "add" "remove"

Patch operation type

path
required
string

JSON Pointer path

value
any

Value for add/replace (omit for remove)

Responses

Request samples

Content type
application/json
[
  • {
    }
]

Response samples

Content type
application/json
{
  • "resource": {
    },
  • "name": "string",
  • "category": "string",
  • "description": "string",
  • "tags": [
    ],
  • "custom": { },
  • "pass_criteria": {
    },
  • "benchmarks": [
    ]
}

Delete Collection

Delete a collection.

path Parameters
id
required
string (Collection Id)

Responses

Response samples

Content type
application/json
{
  • "message": "The field 'state' is not valid.",
  • "message_code": "invalid_value",
  • "trace": "b12692e1-8582-4628-88ca-7a13fefb73e2"
}

Providers

Evaluation provider endpoints

List Providers

List all registered evaluation providers.

query Parameters
limit
integer (Limit) [ 1 .. 100 ]
Default: 50

Maximum number of providers to return

offset
integer (Offset) >= 0
Default: 0

Offset for pagination

benchmarks
boolean (Benchmarks)
Default: true

Include or exclude benchmarks supported by this provider in the response

name
string (Name)

Name to search for

tags
string (Tags)

Tags to search for

scope
string (Scope of providers)
Enum: "system" "tenant"

Set to system to get only system defined providers, or tenant to get only user defined providers. If scope is not provided, both system and user defined providers will be returned.

Responses

Response samples

Content type
application/json
{
  • "first": {
    },
  • "next": {
    },
  • "limit": 0,
  • "total_count": 0,
  • "items": [
    ],
  • "errors": [
    ]
}

Create a new provider scoped to the current tenant (Bring Your Own Provider)

Create a new provider scoped to the current tenant (Bring Your Own Provider)

Request Body schema: application/json
required
name
required
string

Provider name

title
string

Provider display title

description
string

Provider description

tags
Array of strings

Provider tags

required
object (Runtime)

Provider runtime configuration

required
Array of objects (BenchmarkResource)

Benchmarks offered by this provider

Responses

Request samples

Content type
application/json
{
  • "name": "string",
  • "title": "string",
  • "description": "string",
  • "tags": [
    ],
  • "runtime": {
    },
  • "benchmarks": [
    ]
}

Response samples

Content type
application/json
{
  • "resource": {
    },
  • "name": "string",
  • "title": "string",
  • "description": "string",
  • "tags": [
    ],
  • "runtime": {
    },
  • "benchmarks": [
    ]
}

Get Provider

Get a provider by ID.

path Parameters
id
required
string (Provider Id)

Provider ID

Responses

Response samples

Content type
application/json
{
  • "resource": {
    },
  • "name": "string",
  • "title": "string",
  • "description": "string",
  • "tags": [
    ],
  • "runtime": {
    },
  • "benchmarks": [
    ]
}

Update Provider

Update an existing provider.

path Parameters
id
required
string (Provider Id)

Provider ID

Request Body schema: application/json
required
name
required
string

Provider name

title
string

Provider display title

description
string

Provider description

tags
Array of strings

Provider tags

required
object (Runtime)

Provider runtime configuration

required
Array of objects (BenchmarkResource)

Benchmarks offered by this provider

Responses

Request samples

Content type
application/json
{
  • "name": "string",
  • "title": "string",
  • "description": "string",
  • "tags": [
    ],
  • "runtime": {
    },
  • "benchmarks": [
    ]
}

Response samples

Content type
application/json
{
  • "resource": {
    },
  • "name": "string",
  • "title": "string",
  • "description": "string",
  • "tags": [
    ],
  • "runtime": {
    },
  • "benchmarks": [
    ]
}

Patch Provider

Partially update an existing provider.

path Parameters
id
required
string (Provider Id)
Request Body schema: application/json
required
Array
op
required
string (PatchOp)
Enum: "replace" "add" "remove"

Patch operation type

path
required
string

JSON Pointer path

value
any

Value for add/replace (omit for remove)

Responses

Request samples

Content type
application/json
[
  • {
    }
]

Response samples

Content type
application/json
{
  • "resource": {
    },
  • "name": "string",
  • "title": "string",
  • "description": "string",
  • "tags": [
    ],
  • "runtime": {
    },
  • "benchmarks": [
    ]
}

Delete Provider

Delete provider by ID.

path Parameters
id
required
string (Provider Id)

Provider ID

Responses

Response samples

Content type
application/json
{
  • "message": "The field 'state' is not valid.",
  • "message_code": "invalid_value",
  • "trace": "b12692e1-8582-4628-88ca-7a13fefb73e2"
}

Health

Health check endpoints

Health Check

Health check endpoint.

Responses

Response samples

Content type
application/json
{
  • "status": "string",
  • "version": "string",
  • "timestamp": "2019-08-24T14:15:22Z",
  • "components": {
    },
  • "uptime": 0,
  • "active_evaluations": 0
}

Metrics

Metrics and monitoring endpoints

Metrics

Prometheus metrics endpoint with MLflow health check.

Responses

Response samples

Content type
application/json
null

Documentation

API documentation endpoints

OpenAPI specification

Returns the OpenAPI 3.1.0 specification for this API

Responses

Response samples

Content type
No sample

API documentation

Interactive API documentation (Swagger UI)

Responses

Response samples

Content type
application/json
{
  • "message": "The field 'state' is not valid.",
  • "message_code": "invalid_value",
  • "trace": "b12692e1-8582-4628-88ca-7a13fefb73e2"
}