Skip to content

Examples

Create ragas-model-auth using the shared multi-model Secret example. The submissions below keep URLs/API keys out of parameters and select the judge and embedding models using the Ragas mapping. The embedding endpoint must implement /v1/embeddings.

This example walks through evaluating a RAG pipeline from dataset preparation to results retrieval using the EvalHub SDK CLI.

  • EvalHub server running (local or on OpenShift)
  • An OpenAI-compatible model endpoint serving a judge model (e.g. vLLM, Ollama)
  • An embeddings endpoint (can be the same server if it supports /v1/embeddings)
  • The eval-hub-sdk CLI installed: pip install eval-hub-sdk
  • For S3 test data, create the evalhub-s3-credentials Secret in the evaluation job’s namespace with the required keys documented in Using Custom Data.

Create the model-auth Secret in the evaluation job’s namespace before submitting the job. Keep judge_model and embedding_model in benchmark parameters; their connections come from the Secret. See the shared multi-model setup for the manifest and matching URL/API-key requirements.

  1. Create a JSONL dataset with your RAG pipeline outputs. Each line must contain the four RAGAS columns:

    {"user_input": "What is the capital of France?", "response": "The capital of France is Paris.", "retrieved_contexts": ["Paris is the capital and largest city of France. It is situated on the River Seine, in northern France."], "reference": "The capital of France is Paris."}
    {"user_input": "Who wrote Romeo and Juliet?", "response": "Romeo and Juliet was written by William Shakespeare.", "retrieved_contexts": ["Romeo and Juliet is a tragedy written by William Shakespeare early in his career about the romance between two Italian youths from feuding families."], "reference": "William Shakespeare wrote Romeo and Juliet."}
    {"user_input": "What is photosynthesis?", "response": "Photosynthesis is the process by which green plants convert sunlight into chemical energy.", "retrieved_contexts": ["Photosynthesis is a process used by plants and other organisms to convert light energy into chemical energy that can be stored and later released to fuel the organism's activities."], "reference": "Photosynthesis is the process by which plants convert light energy into chemical energy stored in glucose."}
  2. Upload the dataset to an S3 bucket accessible from your cluster, or mount it as a volume at /data/dataset.jsonl.

  3. If your dataset uses different column names, prepare a column_map:

    {
    "column_map": {
    "question": "user_input",
    "answer": "response",
    "contexts": "retrieved_contexts",
    "ground_truth": "reference"
    }
    }

Use your deployment’s OpenAI-compatible API base path, such as https://target.example/v1. The /v1 suffix does not guarantee compatibility; the server must implement the APIs used by its role.

Save this API request as ragas.json. Create the model-auth Secret and the separate dataset S3 credential Secret in the tenant namespace first; see Using Custom Data.

{
"name": "ragas-secret-routing",
"model": {
"url": "https://target.example/v1",
"name": "target-model",
"auth": {
"secret_ref": "ragas-model-auth"
}
},
"benchmarks": [
{
"id": "ragas_rag_default",
"provider_id": "ragas",
"parameters": {
"judge_model": "judge-model",
"embedding_model": "embedding-model",
"max_tokens": 512,
"temperature": 0.1
},
"test_data_ref": {
"s3": {
"bucket": "my-bucket",
"key": "ragas-dataset/dataset.jsonl",
"secret_ref": "evalhub-s3-credentials"
}
}
}
]
}
Terminal window
evalhub eval run --config ragas.json --wait

Submit the same JSON body with the Python SDK or REST API if preferred.

Terminal window
# Check job status
evalhub eval status <job-id>
# Get results
evalhub eval results <job-id>

Results include per-metric scores:

{
"id": "ragas-rag-eval-001",
"benchmark_id": "ragas_rag_default",
"overall_score": 0.87,
"results": [
{"metric_name": "faithfulness", "metric_value": 0.92, "num_samples": 3},
{"metric_name": "answer_relevancy", "metric_value": 0.88, "num_samples": 3},
{"metric_name": "context_precision", "metric_value": 0.85, "num_samples": 3},
{"metric_name": "context_recall", "metric_value": 0.83, "num_samples": 3}
],
"num_examples_evaluated": 3,
"duration_seconds": 45.2
}

To evaluate with all 11 metrics, use the ragas_rag_full benchmark:

Copy ragas.json to ragas-full.json, change the benchmark ID to ragas_rag_full, and keep the same model-auth Secret:

Terminal window
evalhub eval run --config ragas-full.json --wait

For an existing collection containing ragas_rag_default, use the same model-auth Secret and set only model identifiers in its benchmark override. Replace your-rag-collection with the registered collection ID:

{
"name": "ragas-collection-routing",
"model": {
"url": "https://target.example/v1",
"name": "target-model",
"auth": {
"secret_ref": "ragas-model-auth"
}
},
"collection": {
"id": "your-rag-collection",
"benchmarks": [
{
"id": "ragas_rag_default",
"provider_id": "ragas",
"parameters": {
"judge_model": "judge-model",
"embedding_model": "embedding-model"
},
"test_data_ref": {
"s3": {
"bucket": "my-bucket",
"key": "ragas-dataset/dataset.jsonl",
"secret_ref": "evalhub-s3-credentials"
}
}
}
]
}
}

Use a job configuration file to supply S3 data for a collection benchmark. The S3 bucket, key, and secret flags apply to evalhub eval run with inline benchmark flags; evalhub collections run does not accept them.

Save the request as ragas-collection.json and submit it with evalhub eval run --config ragas-collection.json --wait. See Collections to register or choose a collection.

For local testing without Kubernetes:

Terminal window
# Start a model server (e.g. Ollama)
ollama serve &
ollama pull qwen2.5:1.5b
# Prepare a small test dataset
cat > /tmp/dataset.jsonl << 'EOF'
{"user_input": "What is machine learning?", "response": "Machine learning is a subset of AI that enables systems to learn from data.", "retrieved_contexts": ["Machine learning is a branch of artificial intelligence that focuses on building systems that learn from data."], "reference": "Machine learning is a subset of artificial intelligence focused on learning from data."}
EOF
# Run the adapter locally
cd adapters/ragas
export EVALHUB_MODE=local
export EVALHUB_JOB_SPEC_PATH=meta/job.json
python main.py

For local model authentication, set model.auth.secret_ref in the adapter JobSpec to an absolute file:/// auth directory containing the same key filenames and real values. The Kubernetes examples use a Secret name instead and receive projected references; do not copy :ref tokens into a direct local auth directory.