Skip to content
EvalHub mascot EvalHub
Docs Blog

Configuration Examples

Complete examples for common ATIF trajectory evaluation scenarios.

Quick Test

Section titled “Quick Test”

Minimal auto-mode job scoring a single trajectory file:

{
  "id": "atif-quick-001",
  "provider_id": "atif",
  "benchmark_id": "atif_trajectory_default",
  "benchmark_index": 0,
  "experiment_name": "atif-quick-test",
  "model": {
    "name": "microsoft-phi-4",
    "url": "http://127.0.0.1:8000/v1"
  },
  "callback_url": "http://localhost:8081",
  "parameters": {
    "trajectory_path": "/test_data/trajectory.json",
    "scoring_mode": "auto",
    "concurrency_limit": 2,
    "max_judge_requests": 24
  }
}

Use case: Quick validation, CI smoke tests, small fixture collections.

Benchmark Rubric

Section titled “Benchmark Rubric”

Score a collection of terminal-agent trajectories with the built-in rubric:

{
  "id": "atif-terminal-001",
  "provider_id": "atif",
  "benchmark_id": "atif_trajectory_default",
  "benchmark_index": 0,
  "experiment_name": "terminal-agent-regressions",
  "model": {
    "name": "microsoft-phi-4",
    "url": "http://127.0.0.1:8000/v1"
  },
  "callback_url": "http://localhost:8081",
  "parameters": {
    "trajectory_path": "/test_data",
    "scoring_mode": "benchmark",
    "benchmark_name": "terminal-bench",
    "partial_result_policy": "skip_failed_trajectory"
  }
}

Use case: Regression-tracking a corpus of recorded agent runs against a fixed rubric.

Reference Registry

Section titled “Reference Registry”

Score against expected answers from a registry file. First write the registry:

{
  "rubrics": {
    "default": {
      "criteria": [
        {
          "name": "correctness",
          "description": "Matches the reference",
          "weight": 1
        }
      ],
      "references": {
        "t1": { "answer": "The server on port 8080 must be restarted" },
        "t2": { "answer": "Rotate the credential and update the secret" }
      }
    }
  }
}

Then reference it from the job:

{
  "id": "atif-reference-001",
  "provider_id": "atif",
  "benchmark_id": "atif_trajectory_default",
  "benchmark_index": 0,
  "experiment_name": "atif-reference-test",
  "model": {
    "name": "microsoft-phi-4",
    "url": "http://127.0.0.1:8000/v1"
  },
  "callback_url": "http://localhost:8081",
  "parameters": {
    "trajectory_path": "/test_data",
    "scoring_mode": "reference",
    "reference_registry_path": "/rubrics/registry.json",
    "reference_rubric": "default"
  }
}

Use case: Tasks with known-good outcomes. Reference selection uses each trajectory’s extra.reference_id, then its trajectory_id, then the default reference.

Custom Rubric

Section titled “Custom Rubric”

Score with an inline rubric and explicit aggregation. provider_params.rubric accepts a JSON object directly:

{
  "id": "atif-custom-001",
  "provider_id": "atif",
  "benchmark_id": "atif_trajectory_default",
  "benchmark_index": 0,
  "experiment_name": "atif-custom-test",
  "model": {
    "name": "microsoft-phi-4",
    "url": "http://127.0.0.1:8000/v1"
  },
  "callback_url": "http://localhost:8081",
  "parameters": {
    "trajectory_path": "/test_data",
    "scoring_mode": "custom",
    "provider_params": {
      "rubric": {
        "name": "answer_quality",
        "aggregation": "weighted_mean",
        "criteria": [
          {
            "name": "correctness",
            "description": "Addresses the requested task",
            "weight": 2
          },
          {
            "name": "clarity",
            "description": "Explains the result clearly",
            "weight": 1
          }
        ]
      }
    }
  }
}

The rubric can also be supplied as a YAML or JSON document string, which is useful when mounting or generating rubrics as text:

"provider_params": {
  "rubric": "name: answer_quality\naggregation: weighted_mean\ncriteria:\n  - name: correctness\n    description: Addresses the requested task\n    weight: 2\n  - name: clarity\n    description: Explains the result clearly\n    weight: 1\n"
}

Legacy custom_rubric and custom_rubric_path parameters still work — supply exactly one source, and keep either form under 16 KiB.

Use case: Team-specific quality bars, weighted criteria, mean/minimum aggregation.

Training-Data Selection

Section titled “Training-Data Selection”

Add a training_threshold to produce eligible/ineligible cohorts with source-addressable manifests:

{
  "id": "atif-training-001",
  "provider_id": "atif",
  "benchmark_id": "atif_trajectory_default",
  "benchmark_index": 0,
  "experiment_name": "atif-training-selection",
  "model": {
    "name": "microsoft-phi-4",
    "url": "http://127.0.0.1:8000/v1"
  },
  "callback_url": "http://localhost:8081",
  "parameters": {
    "trajectory_path": "/test_data",
    "scoring_mode": "auto",
    "subagent_aggregation": "hierarchical",
    "training_threshold": 0.8,
    "completion_threshold": 0.5
  }
}

Retrieve atif/report.json, atif/training/eligible_trajectories.json, and atif/training/ineligible_trajectories.json from the MLflow run. See Artifacts for the manifest contract.

Use case: Curating high-scoring, completely-scored trajectories for fine-tuning.

Running Locally Against a Private Judge

Section titled “Running Locally Against a Private Judge”
Terminal window
# A directory of ATIF JSON files (or a single file)
mkdir -p test_data
cp trajectory.json test_data/


# Judge auth directory: api-key file, optional ca_cert.
# Must be a real directory with no symlinks anywhere in its path.
install -d -m 750 ~/.config/atif-judge-auth
install -m 640 /path/to/judge-key ~/.config/atif-judge-auth/api-key

The job spec’s model.auth.secret_ref must point at the mounted copy of this directory:

{
  "model": {
    "name": "microsoft-phi-4",
    "url": "https://judge.internal.example.com/v1",
    "auth": { "secret_ref": "file:///mnt/judge-auth" }
  }
}