Skip to content

DeepEval Adapter

The DeepEval adapter integrates DeepEval with the eval-hub evaluation service using the evalhub-sdk framework adapter pattern. DeepEval is an open-source LLM evaluation framework that uses an LLM-as-judge approach, where a separate model scores outputs for faithfulness, hallucination, answer relevancy, factual correctness, summarization quality, and multi-turn conversation properties.

DeepEval provides a structured suite of metrics for evaluating LLM outputs across two evaluation modes: single-turn assessments (one input → one response) and multi-turn conversation assessments (sequences of user and assistant turns). The adapter constructs the appropriate DeepEval test case type—LLMTestCase for single-turn or ConversationalTestCase for multi-turn—and maps raw scores into EvalHub’s unified JobResults format.

  • Single-turn metrics: Faithfulness, answer relevancy, hallucination detection, factual correctness, and summarization quality
  • Multi-turn evaluation: Conversation completeness, role adherence, and knowledge retention across multi-turn dialogues
  • LLM-as-judge: Uses an OpenAI-compatible endpoint as the judge model; works with vLLM, TGI, Ollama, and any OpenAI-compatible server
  • Flexible data input: CSV (single-turn) and JSONL/JSON (recommended for multi-turn) with configurable dataset paths
  • MLflow integration: Logs scores and pass/fail results to a configurable MLflow experiment
  • Retry and concurrency controls: Configurable retry logic and throttling to accommodate rate-limited or reasoning-model judge endpoints

Single-Turn (uses LLMTestCase):

  • faithfulness — Is the output faithful to the retrieved context?
  • relevancy — Does the output address the input query?
  • hallucination — Does the output contain hallucinated content not grounded in context?
  • correctness — Does the output match the expected ground-truth answer?
  • summarization — How well does the output summarise the source text?

Multi-Turn (uses ConversationalTestCase):

  • conversation-completeness — Does the chatbot address all user needs across the conversation?
  • role-adherence — Does the chatbot stay in its assigned persona throughout?
  • knowledge-retention — Does the chatbot correctly retain information disclosed earlier in the conversation?

The adapter follows the eval-hub framework adapter pattern:

Workflow:

  1. JobSpec loading: Job configuration auto-loaded from the mounted file at /meta/job.json (or EVALHUB_JOB_SPEC_PATH in local mode)
  2. Data resolution: Evaluation dataset loaded from /test_data (S3 init container), /data, or an explicit data_dir parameter
  3. Test case construction: Each dataset row is mapped to either an LLMTestCase (single-turn) or ConversationalTestCase (multi-turn) based on the benchmark
  4. Judge model setup: An OpenAI-compatible wrapper is configured using the resolved model credentials
  5. DeepEval evaluation: The metric is applied to all test cases via deepeval.evaluate()
  6. Result mapping: Raw DeepEval scores are mapped to EvaluationResult objects and an aggregate score is computed
  7. Artifact persistence: A results_summary.json is optionally pushed as an OCI artifact
  8. Callback-based communication: Progress and results are reported to the eval-hub sidecar

Keep model URLs and API keys out of parameters; use the shared Model Authentication guide. DeepEval’s configuration reference lists its judge parameter and legacy fields.

Follow the faithfulness example to prepare the Secret and dataset, then save its submission as deepeval.json:

Terminal window
evalhub eval run --config deepeval.json --wait

See Configuration for the parameter reference and Examples for hallucination detection, multi-turn benchmarks, and MLflow-tracked evaluations.

FieldValue
Provider IDdeepeval
Container Imagequay.io/evalhub/community-deepeval:latest

The adapter is distributed as a pre-built container image. Pull it directly for use with EvalHub or for local testing:

Terminal window
podman pull quay.io/evalhub/community-deepeval:latest

For direct local execution, use the auth-directory and mounted-dataset setup in Local Testing.

To build the image from the eval-hub-contrib repository:

Terminal window
# From the eval-hub-contrib repo root
make image-deepeval
# Push to a registry
make push-deepeval REGISTRY=quay.io/your-org VERSION=v1.0.0

To run the adapter directly without a container, prepare the auth directory in Local Testing and set model.auth.secret_ref in meta/job.json to its absolute file:/// URL. The directory contains api-key and optional judge_api-key / judge_url files:

Terminal window
cd adapters/deepeval
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
export EVALHUB_MODE=local
export EVALHUB_JOB_SPEC_PATH=meta/job.json
.venv/bin/python main.py