Examples
Model-auth Secret
Section titled “Model-auth Secret”Create ragas-model-auth using the shared
multi-model Secret example.
The submissions below keep URLs/API keys out of parameters and select the
judge and embedding models using the Ragas mapping.
The embedding endpoint must implement /v1/embeddings.
End-to-End RAG Evaluation
Section titled “End-to-End RAG Evaluation”This example walks through evaluating a RAG pipeline from dataset preparation to results retrieval using the EvalHub SDK CLI.
Prerequisites
Section titled “Prerequisites”- EvalHub server running (local or on OpenShift)
- An OpenAI-compatible model endpoint serving a judge model (e.g. vLLM, Ollama)
- An embeddings endpoint (can be the same server if it supports
/v1/embeddings) - The
eval-hub-sdkCLI installed:pip install eval-hub-sdk - For S3 test data, create the
evalhub-s3-credentialsSecret in the evaluation job’s namespace with the required keys documented in Using Custom Data.
Configure Model Connections
Section titled “Configure Model Connections”Create the model-auth Secret in the evaluation job’s
namespace before submitting the job. Keep judge_model and embedding_model
in benchmark parameters; their connections come from the Secret. See the shared
multi-model setup
for the manifest and matching URL/API-key requirements.
Prepare the Dataset
Section titled “Prepare the Dataset”-
Create a JSONL dataset with your RAG pipeline outputs. Each line must contain the four RAGAS columns:
{"user_input": "What is the capital of France?", "response": "The capital of France is Paris.", "retrieved_contexts": ["Paris is the capital and largest city of France. It is situated on the River Seine, in northern France."], "reference": "The capital of France is Paris."}{"user_input": "Who wrote Romeo and Juliet?", "response": "Romeo and Juliet was written by William Shakespeare.", "retrieved_contexts": ["Romeo and Juliet is a tragedy written by William Shakespeare early in his career about the romance between two Italian youths from feuding families."], "reference": "William Shakespeare wrote Romeo and Juliet."}{"user_input": "What is photosynthesis?", "response": "Photosynthesis is the process by which green plants convert sunlight into chemical energy.", "retrieved_contexts": ["Photosynthesis is a process used by plants and other organisms to convert light energy into chemical energy that can be stored and later released to fuel the organism's activities."], "reference": "Photosynthesis is the process by which plants convert light energy into chemical energy stored in glucose."} -
Upload the dataset to an S3 bucket accessible from your cluster, or mount it as a volume at
/data/dataset.jsonl. -
If your dataset uses different column names, prepare a
column_map:{"column_map": {"question": "user_input","answer": "response","contexts": "retrieved_contexts","ground_truth": "reference"}}
Submit a RAGAS Evaluation Job
Section titled “Submit a RAGAS Evaluation Job”Use your deployment’s OpenAI-compatible API base path, such as
https://target.example/v1. The /v1 suffix does not guarantee compatibility;
the server must implement the APIs used by its role.
Save this API request as ragas.json. Create the model-auth Secret and the
separate dataset S3 credential Secret in the tenant namespace first; see
Using Custom Data.
{ "name": "ragas-secret-routing", "model": { "url": "https://target.example/v1", "name": "target-model", "auth": { "secret_ref": "ragas-model-auth" } }, "benchmarks": [ { "id": "ragas_rag_default", "provider_id": "ragas", "parameters": { "judge_model": "judge-model", "embedding_model": "embedding-model", "max_tokens": 512, "temperature": 0.1 }, "test_data_ref": { "s3": { "bucket": "my-bucket", "key": "ragas-dataset/dataset.jsonl", "secret_ref": "evalhub-s3-credentials" } } } ]}evalhub eval run --config ragas.json --waitSubmit the same JSON body with the Python SDK or REST API if preferred.
Retrieve Results
Section titled “Retrieve Results”# Check job statusevalhub eval status <job-id>
# Get resultsevalhub eval results <job-id>Results include per-metric scores:
{ "id": "ragas-rag-eval-001", "benchmark_id": "ragas_rag_default", "overall_score": 0.87, "results": [ {"metric_name": "faithfulness", "metric_value": 0.92, "num_samples": 3}, {"metric_name": "answer_relevancy", "metric_value": 0.88, "num_samples": 3}, {"metric_name": "context_precision", "metric_value": 0.85, "num_samples": 3}, {"metric_name": "context_recall", "metric_value": 0.83, "num_samples": 3} ], "num_examples_evaluated": 3, "duration_seconds": 45.2}Running the Full Metric Suite
Section titled “Running the Full Metric Suite”To evaluate with all 11 metrics, use the ragas_rag_full benchmark:
Copy ragas.json to ragas-full.json, change the benchmark ID to
ragas_rag_full, and keep the same model-auth Secret:
evalhub eval run --config ragas-full.json --waitEvaluation Collection with RAGAS
Section titled “Evaluation Collection with RAGAS”For an existing collection containing ragas_rag_default, use the same
model-auth Secret and set only model identifiers in its benchmark override.
Replace your-rag-collection with the registered collection ID:
{ "name": "ragas-collection-routing", "model": { "url": "https://target.example/v1", "name": "target-model", "auth": { "secret_ref": "ragas-model-auth" } }, "collection": { "id": "your-rag-collection", "benchmarks": [ { "id": "ragas_rag_default", "provider_id": "ragas", "parameters": { "judge_model": "judge-model", "embedding_model": "embedding-model" }, "test_data_ref": { "s3": { "bucket": "my-bucket", "key": "ragas-dataset/dataset.jsonl", "secret_ref": "evalhub-s3-credentials" } } } ] }}Run the Collection
Section titled “Run the Collection”Use a job configuration file to supply S3 data for a collection benchmark.
The S3 bucket, key, and secret flags apply to evalhub eval run with inline
benchmark flags; evalhub collections run does not accept them.
Save the request as ragas-collection.json and submit it with
evalhub eval run --config ragas-collection.json --wait. See
Collections to register or choose a collection.
Local Development Example
Section titled “Local Development Example”For local testing without Kubernetes:
# Start a model server (e.g. Ollama)ollama serve &ollama pull qwen2.5:1.5b
# Prepare a small test datasetcat > /tmp/dataset.jsonl << 'EOF'{"user_input": "What is machine learning?", "response": "Machine learning is a subset of AI that enables systems to learn from data.", "retrieved_contexts": ["Machine learning is a branch of artificial intelligence that focuses on building systems that learn from data."], "reference": "Machine learning is a subset of artificial intelligence focused on learning from data."}EOF
# Run the adapter locallycd adapters/ragasexport EVALHUB_MODE=localexport EVALHUB_JOB_SPEC_PATH=meta/job.jsonpython main.pyFor local model authentication, set model.auth.secret_ref in the adapter JobSpec
to an absolute file:/// auth directory containing the same key filenames and
real values. The Kubernetes examples use a Secret name instead and receive
projected references; do not copy :ref tokens into a direct local auth directory.