Configuration Reference
Complete reference for DeepEval adapter configuration options.
JobSpec Structure
Section titled “JobSpec Structure”The DeepEval adapter uses the standard EvalHub JobSpec structure:
{ "id": "string", "provider_id": "deepeval", "benchmark_id": "string", "benchmark_index": 0, "experiment_name": "string", "model": { "url": "string", "name": "string", "auth": { "secret_ref": "string" } }, "parameters": { "threshold": 0.5 }, "test_data_ref": { "s3": { "bucket": "string", "key": "string", "secret_ref": "string" } }, "callback_url": "string"}Core Parameters
Section titled “Core Parameters”Required Parameters
Section titled “Required Parameters”| Parameter | Type | Description | Example |
|---|---|---|---|
id | string | Unique job identifier | "deepeval-faithfulness-001" |
provider_id | string | Must be "deepeval" | "deepeval" |
benchmark_id | string | Benchmark to run (see Benchmarks) | "faithfulness" |
model.name | string | Name of the evaluated model | "gpt-4o" |
model.url | string | OpenAI-compatible endpoint for the evaluated model | "https://api.openai.com/v1" |
Optional Parameters
Section titled “Optional Parameters”| Parameter | Type | Description | Default |
|---|---|---|---|
benchmark_index | integer | Index when running multiple benchmarks | 0 |
experiment_name | string | MLflow experiment name for result tracking | null |
parameters | object | DeepEval-specific configuration (see below) | {} |
test_data_ref | object | S3 reference for the evaluation dataset | null |
callback_url | string | EvalHub service callback URL | null |
Benchmark Configuration
Section titled “Benchmark Configuration”Eight benchmarks are available across two evaluation modes. See Benchmarks for the full reference including required dataset columns and output metrics.
| Benchmark ID | Name | Mode | Category |
|---|---|---|---|
faithfulness | Faithfulness | Single-turn | RAG evaluation |
relevancy | Answer Relevancy | Single-turn | RAG evaluation |
hallucination | Hallucination | Single-turn | Safety |
correctness | Correctness | Single-turn | Accuracy |
summarization | Summarization | Single-turn | NLP |
conversation-completeness | Conversation Completeness | Multi-turn | Multi-turn |
role-adherence | Role Adherence | Multi-turn | Multi-turn |
knowledge-retention | Knowledge Retention | Multi-turn | Multi-turn |
Parameters Reference
Section titled “Parameters Reference”All configuration is specified in the parameters object of the JobSpec.
Judge Model
Section titled “Judge Model”| Parameter | Type | Description | Default |
|---|---|---|---|
eval_model_name | string | Name of the judge model | model.name |
Model-auth Secret
Section titled “Model-auth Secret”Use the shared Secret setup,
naming the Secret deepeval-model-auth for the examples on this site. Keep
model connection URLs and API keys out of parameters.
- Select the judge with
eval_model_name(defaultmodel.name). Its connection usesjudge_api-key/judge_url. - Legacy
eval_model_url,judge_url,judge_base_url, andjudge_api_keyparameters are ignored.OPENAI_API_KEY/ANTHROPIC_API_KEYruntime values do not configure this judge connection.
See the shared migration and image requirements.
Scoring
Section titled “Scoring”| Parameter | Type | Description | Default |
|---|---|---|---|
threshold | float | Minimum score for a test case to pass | 0.5 |
| Parameter | Type | Description | Default |
|---|---|---|---|
dataset_format | string | Input dataset format: csv, jsonl, or json | csv |
data_dir | string | Path to dataset directory | Auto-resolved |
Data resolution order (when data_dir is not set):
/test_data/— populated by EvalHub’s S3 init container/data/
Concurrency
Section titled “Concurrency”| Parameter | Type | Description | Default |
|---|---|---|---|
max_concurrent | integer | Maximum test cases evaluated concurrently | 1 |
throttle_value | float | Seconds to wait between test case evaluations | 0 |
Timeouts and Retries
Section titled “Timeouts and Retries”| Parameter | Type | Description | Default |
|---|---|---|---|
per_attempt_timeout_seconds | float | Per-attempt timeout for each LLM judge call | 300.0 |
retry_max_attempts | integer | Total LLM call attempts per metric (including first) | 2 |
retry_cap_seconds | float | Maximum backoff between retry attempts | 5.0 |
Multi-Turn
Section titled “Multi-Turn”| Parameter | Type | Description | Default |
|---|---|---|---|
chatbot_role | string | Chatbot persona for Role Adherence benchmark | null |
chatbot_role can also be provided per record in the dataset as a chatbot_role column, which takes precedence over the parameter value.
MLflow Integration
Section titled “MLflow Integration”When experiment_name is set in the JobSpec and MLFLOW_TRACKING_URI is configured, the adapter logs evaluation results to MLflow automatically.
What Is Logged
Section titled “What Is Logged”For each completed evaluation, the adapter logs:
- Metrics: Primary score for the benchmark (e.g.
faithfulness_score,hallucination_score) - Parameters:
benchmark_id,threshold,eval_model_name,dataset_format,num_test_cases - Tags:
provider_id,job_id
Environment Variables for MLflow
Section titled “Environment Variables for MLflow”| Variable | Description | Example |
|---|---|---|
MLFLOW_TRACKING_URI | MLflow server URL | http://mlflow-service:5000 |
MLFLOW_EXPERIMENT | Default experiment name (overridden by experiment_name in JobSpec) | deepeval-evals |
See the MLflow guide for setup and configuration details.
Environment Variables
Section titled “Environment Variables”| Variable | Description | Required | Default |
|---|---|---|---|
EVALHUB_MODE | Execution mode: k8s or local | No | k8s |
EVALHUB_JOB_SPEC_PATH | Path to the job spec JSON file | Yes (local mode) | /meta/job.json |
DEEPEVAL_CACHE_DIR | DeepEval cache directory | No | /tmp |
MLFLOW_TRACKING_URI | MLflow server URL | When using MLflow | — |
MLFLOW_EXPERIMENT | MLflow experiment name (fallback) | No | — |
Complete Example
Section titled “Complete Example”{ "id": "faithfulness-test-001", "provider_id": "deepeval", "benchmark_id": "faithfulness", "benchmark_index": 0, "experiment_name": "faithfulness-eval", "model": { "url": "https://api.openai.com/v1", "name": "gpt-4o", "auth": { "secret_ref": "deepeval-model-auth" } }, "parameters": { "eval_model_name": "gpt-4o", "threshold": 0.7, "dataset_format": "csv", "max_concurrent": 2, "throttle_value": 0.5, "per_attempt_timeout_seconds": 60.0, "retry_max_attempts": 3, "retry_cap_seconds": 10.0 }, "test_data_ref": { "s3": { "bucket": "deepeval-datasets", "key": "faithfulness/", "secret_ref": "deepeval-data-bucket" } }, "callback_url": "http://evalhub-service:8080"}