Examples
Model-auth Secret for these examples
Section titled “Model-auth Secret for these examples”Create deepeval-model-auth using the shared
Secret example and
DeepEval’s judge mapping.
Keep URLs and API keys out of parameters; the submissions below reference
the Secret by name.
Faithfulness Evaluation
Section titled “Faithfulness Evaluation”Evaluates whether a RAG pipeline’s responses are supported by the retrieved context.
Prerequisites
Section titled “Prerequisites”- EvalHub server running (local or on OpenShift)
- An OpenAI-compatible model endpoint for the judge (e.g. vLLM, Ollama, OpenAI)
- The
eval-hub-sdkCLI installed:pip install eval-hub-sdk
Prepare the Dataset
Section titled “Prepare the Dataset”-
Create a CSV dataset with the three required columns for faithfulness:
input,actual_output,retrieval_context"What is the boiling point of water?","Water boils at 100°C at sea level.","Water boils at 100 degrees Celsius (212°F) at standard atmospheric pressure.""Who invented the telephone?","Alexander Graham Bell invented the telephone in 1876.","Alexander Graham Bell is credited with inventing the first practical telephone, patented in 1876.""What planet is closest to the Sun?","Mercury is the closest planet to the Sun.","Mercury is the innermost planet of the Solar System and the closest to the Sun." -
Upload the dataset to an S3 bucket accessible from your cluster, or mount it at
/test_data/data.csv.
Submit a Faithfulness Evaluation Job
Section titled “Submit a Faithfulness Evaluation Job”Save this API request as deepeval.json. Create the model-auth Secret and the
separate dataset S3 credential Secret in the tenant namespace first; see
Using Custom Data.
{ "name": "deepeval-secret-routing", "model": { "url": "https://target.example/v1", "name": "target-model", "auth": { "secret_ref": "deepeval-model-auth" } }, "benchmarks": [ { "id": "faithfulness", "provider_id": "deepeval", "parameters": { "eval_model_name": "judge-model", "threshold": 0.7, "dataset_format": "csv" }, "test_data_ref": { "s3": { "bucket": "my-bucket", "key": "deepeval-datasets/faithfulness.csv", "secret_ref": "evalhub-s3-credentials" } } } ]}evalhub eval run --config deepeval.json --waitSubmit the same JSON body with the Python SDK or REST API if preferred.
Retrieve Results
Section titled “Retrieve Results”# Check job statusevalhub eval status <job-id>
# Get resultsevalhub eval results <job-id>Results include per-test-case scores and an aggregate:
{ "id": "deepeval-faithfulness-001", "benchmark_id": "faithfulness", "overall_score": 0.89, "results": [ { "metric_name": "faithfulness_score", "metric_value": 0.89, "num_samples": 3 }, { "metric_name": "claims_count", "metric_value": 9, "num_samples": 3 }, { "metric_name": "supported_claims_count", "metric_value": 8, "num_samples": 3 } ], "num_examples_evaluated": 3}Hallucination Detection
Section titled “Hallucination Detection”Detects hallucinated content that is not grounded in the provided context.
{ "id": "deepeval-hallucination-001", "provider_id": "deepeval", "benchmark_id": "hallucination", "model": { "url": "http://vllm-service:8000/v1", "name": "mistral-7b-instruct", "auth": { "secret_ref": "deepeval-model-auth" } }, "parameters": { "eval_model_name": "gpt-4o", "threshold": 0.5, "dataset_format": "csv" }}Dataset (CSV):
input,actual_output,context"What year was the Eiffel Tower built?","The Eiffel Tower was built in 1889.","The Eiffel Tower, located in Paris, was constructed between 1887 and 1889.""What is the speed of light?","Light travels at 300,000 km/s in a vacuum.","The speed of light in a vacuum is approximately 299,792 km/s."Use case: Safety evaluation for production RAG pipelines where factual grounding is critical, such as medical or legal question-answering systems.
Multi-Turn: Role Adherence
Section titled “Multi-Turn: Role Adherence”Tests whether a chatbot maintains its assigned persona throughout a conversation. This example evaluates a customer support bot.
{ "id": "deepeval-role-adherence-001", "provider_id": "deepeval", "benchmark_id": "role-adherence", "model": { "url": "http://vllm-service:8000/v1", "name": "llama-3-8b-instruct", "auth": { "secret_ref": "deepeval-model-auth" } }, "parameters": { "eval_model_name": "gpt-4o", "threshold": 0.6, "dataset_format": "jsonl", "chatbot_role": "friendly and professional customer support agent for a software company" }}Dataset (JSONL — role_adherence_dataset.jsonl):
{"turns": [{"role": "user", "content": "I need help with my account."}, {"role": "assistant", "content": "I'd be happy to help you with your account! What seems to be the issue?"}], "chatbot_role": "friendly and professional customer support agent for a software company"}{"turns": [{"role": "user", "content": "Can you write me a poem instead?"}, {"role": "assistant", "content": "I'm here to help with account and software questions, but I can certainly try to brighten your day! However, my main focus is on resolving your support needs."}], "chatbot_role": "friendly and professional customer support agent for a software company"}Use case: Evaluate persona-constrained assistants to ensure they do not break character when users attempt to redirect the conversation.
Multi-Turn: Conversation Completeness
Section titled “Multi-Turn: Conversation Completeness”Evaluates whether the chatbot addresses all user needs across a multi-turn support conversation.
{ "id": "deepeval-conv-completeness-001", "provider_id": "deepeval", "benchmark_id": "conversation-completeness", "model": { "url": "http://vllm-service:8000/v1", "name": "llama-3-8b-instruct", "auth": { "secret_ref": "deepeval-model-auth" } }, "parameters": { "eval_model_name": "gpt-4o", "threshold": 0.6, "dataset_format": "jsonl" }}Dataset (JSONL):
{"turns": [{"role": "user", "content": "I need to cancel my subscription and get a refund."}, {"role": "assistant", "content": "I can help with that. I'll process the cancellation now."}, {"role": "user", "content": "What about the refund?"}, {"role": "assistant", "content": "The refund will be processed within 5–7 business days."}]}Use case: Identify conversations where the model resolves part of the user’s request but silently drops other tasks or sub-goals.
Local Testing
Section titled “Local Testing”Run the adapter locally without Kubernetes for fast iteration on datasets and job configurations.
cd eval-hub-contrib/adapters/deepeval
# Set up a virtual environmentpython3 -m venv .venv && .venv/bin/pip install -r requirements.txt
# Place your test datamkdir -p test_datacp /path/to/my_dataset.csv test_data/data.csv
# Configure the job spec (edit meta/job.json or set EVALHUB_JOB_SPEC_PATH)export EVALHUB_MODE=localexport EVALHUB_JOB_SPEC_PATH=meta/job.json# Configure model.auth.secret_ref as file:///absolute/path/to/model-auth.# That directory contains api-key and optional judge_api-key / judge_url files.
# Runpython main.pyThe adapter will print lifecycle phase transitions (INITIALIZING, LOADING_DATA, RUNNING_EVALUATION, POST_PROCESSING, PERSISTING_ARTIFACTS) and a JSON summary of results to stdout.
MLflow-Tracked Evaluation
Section titled “MLflow-Tracked Evaluation”Track evaluation results in MLflow for comparison across model versions or prompt changes.
JobSpec with experiment_name:
{ "id": "deepeval-faithfulness-mlflow-001", "provider_id": "deepeval", "benchmark_id": "faithfulness", "experiment_name": "rag-pipeline-v2-faithfulness", "model": { "url": "http://vllm-service:8000/v1", "name": "mistral-7b-instruct", "auth": { "secret_ref": "deepeval-model-auth" } }, "parameters": { "eval_model_name": "gpt-4o", "threshold": 0.7, "dataset_format": "csv" }}Submit the job:
evalhub eval run \ --name rag-pipeline-v2-faithfulness \ --provider deepeval \ --benchmark faithfulness \ --experiment rag-pipeline-v2-faithfulness \ --model-name "mistral-7b-instruct" \ --model-url "http://vllm-service:8000/v1" \ --model-auth-secret deepeval-model-auth \ --param eval_model_name=gpt-4o \ --test-data-s3-bucket my-bucket \ --test-data-s3-key deepeval-datasets/faithfulness.csv \ --test-data-s3-secret evalhub-s3-credentialsAfter the job completes, the run appears in MLflow under the rag-pipeline-v2-faithfulness experiment with the primary benchmark score, pass/fail status, and job metadata logged as metrics and parameters.
See the MLflow guide for setting up the MLflow server and configuring experiment tracking across multiple providers.