Coding v1
Curatedcoding-v1
Evaluates code generation capability of instruction-tuned chat LLMs on execution-based programming problems.
Benchmarks (1)
-
lcb:codegeneration_v6lighteval
Collection Catalog
Browse benchmark collections from EvalHub, synced at site build time. Curated collections appear first, ordered by their curation priority, followed by non-curated collections.
Use a collection ID to run its benchmarks together. Open its definition for the full description, parameters, and scoring criteria.
Showing 19 of 19 collections
Curated collections
Fairness and Bias v1
Curatedfairness-bias-v1
Evaluates selected social-bias diagnostics using Winogender pronoun resolution, CrowS-Pairs stereotypical sentence preference, and BBQ disambiguated-context accuracy.
Benchmarks (3)
-
winogenderlm_evaluation_harness -
crows_pairs_englishlm_evaluation_harness -
bbqlm_evaluation_harness
Instruction and Output Reliability v1
Curatedinstruction-output-v1
Evaluates bare-model instruction and output reliability across explicit instruction following, unconstrained JSON/schema compliance, and compound constraint satisfaction.
Benchmarks (3)
-
ifbenchifbench -
jsonschema_bench_easylm_evaluation_harness -
followbenchfollowbench
Knowledge and Reasoning v1
Curatedknowledge-reasoning-v1
Evaluates closed-book, text-only knowledge and reasoning across difficult scientific, academic and mathematical problems using GPQA Diamond, HLE and MATH-500.
Benchmarks (3)
-
gpqa:diamondlighteval · 0-shot -
inspect/hleinspect -
math_500lighteval · 0-shot
Long-Context Document Comprehension v1
Curateddocument-understanding-v1
Evaluates bare-model understanding of supplied long-context documents and passages using RULER and LongBench v2.
Benchmarks (7)
-
common-words-extractionruler -
frequency-words-extractionruler -
qa-squadruler -
qa-hotpotqaruler -
niah-single-essayruler -
variable-trackingruler -
longbenchv2lm_evaluation_harness · 0-shot
Multimodal Document VQA v1
Curatedmultimodal-v1
Evaluates vision-language understanding on scanned-document visual question answering using DocVQA.
Benchmarks (1)
-
inspect/docvqainspect
Safety and Ethical Principles v1
Curatedsafety-ethical-principles-v1
Evaluates toxicity classification, commonsense moral judgment, and preferences for helpful, honest, and harmless responses using ToxiGen, ETHICS, and HHH multiple-choice tasks.
Benchmarks (3)
-
toxigenlm_evaluation_harness -
ethics_cmlm_evaluation_harness -
bigbench_hhh_alignment_multiple_choicelm_evaluation_harness
Tool Use v1
Curatedtool-use-v1
Evaluates single-turn function-calling capability using supplied function schemas, including function selection, argument generation and structural correctness, together with relevance and no-call detection.
Benchmarks (2)
-
inspect/bfclinspect -
inspect/bfclinspect
Trustworthiness v1
Curatedtrustworthiness-v1
Evaluates selected trustworthiness dimensions for bare models, including prompt-injection resistance and multiple-choice misconception resistance.
Benchmarks (2)
-
inspect/cyberseceval-2-piinspect -
truthfulqa_mc1lm_evaluation_harness
No curated collections match your search.
Non-curated collections
evaluate-model-accuracy
evaluate-model-accuracy
Evaluates model accuracy using standard industry research benchmarks in inspectai.
Benchmarks (11)
-
inspect_evals/gsm8kinspect · inspect/custom · 0-shot -
inspect_evals/gsm8kinspect · inspect/custom · 5-shot -
inspect_evals/mmlu_proinspect · inspect/custom · 0-shot -
inspect_evals/mmlu_proinspect · inspect/custom · 5-shot -
inspect_evals/mmlu_5_shotinspect · inspect/custom -
inspect_evals/gpqa_diamondinspect · inspect/custom -
inspect_evals/humanevalinspect · inspect/custom -
inspect_evals/mbppinspect · inspect/custom -
inspect_evals/ifevalinspect · inspect/custom -
inspect_evals/bfclinspect · inspect/custom -
inspect_evals/niahinspect · inspect/custom · 0-shot
Instruction Following v1
instruction-following-v1
Evaluates how well instruction-tuned chat LLMs follow explicit instructions using IFEval's verifiable constraints for length, keywords, formatting, JSON structure, and language.
Benchmarks (1)
-
ifevallm_evaluation_harness · 0-shot
Model Validation
model-validation
Evaluates model safety using context-aware vulnerability scanning with custom intent typology and probes of increasing complexity.
Benchmarks (1)
-
intentsgarak-kfp
Open LLM Leaderboard v2
leaderboard-v2
Comprehensive evaluation suite for general-purpose language models.
Benchmarks (6)
-
leaderboard_ifevallm_evaluation_harness -
leaderboard_bbhlm_evaluation_harness -
leaderboard_gpqalm_evaluation_harness -
leaderboard_mmlu_prolm_evaluation_harness -
leaderboard_musrlm_evaluation_harness -
leaderboard_math_hardlm_evaluation_harness
Open-Telco v1
open-telco-v1
GSMA Open-Telco suite for evaluating LLMs on telecom-domain tasks.
Benchmarks (4)
-
telemathinspect -
teleqnainspect -
telelogsinspect -
3gpp-tsginspect
Reasoning v1
reasoning-v1
Evaluates multi-step and chain-of-thought reasoning for instruction-tuned chat LLMs across advanced multidomain Q&A, competition mathematics, and expert-level science.
Benchmarks (4)
-
leaderboard_mmlu_prolm_evaluation_harness · 0-shot -
math_500lighteval · 0-shot -
aime25lighteval · 0-shot -
gpqa:diamondlighteval · 0-shot
Safety and Fairness
safety-and-fairness-v1
Evaluates model safety, bias, and fairness across diverse scenarios.
Benchmarks (6)
-
truthfulqa_mc1lm_evaluation_harness · 0-shot -
toxigenlm_evaluation_harness · 0-shot -
winogenderlm_evaluation_harness · 0-shot -
crows_pairs_englishlm_evaluation_harness · 0-shot -
bbqlm_evaluation_harness · 0-shot -
ethics_cmlm_evaluation_harness · 0-shot
Software v1
software-v1
Evaluates non-agentic software capabilities across code generation and code understanding.
Benchmarks (3)
-
lcb:codegeneration_v6lighteval · 0-shot -
inspect/bigcodebenchinspect -
cruxeval_outputlm_evaluation_harness · 0-shot
Standard LLM Evaluation Suite v1
standard-llm-evals-v1
Standard evaluation protocol for instruction-tuned chat LLMs covering instruction following, reasoning, math, and coding tasks.
Benchmarks (6)
-
leaderboard_mmlu_prolm_evaluation_harness · 5-shot -
ifevallm_evaluation_harness · 0-shot -
math_500lighteval · 0-shot -
aime25lighteval · 0-shot -
gpqa:diamondlighteval · 0-shot -
lcb:codegeneration_v6lighteval · 0-shot
Toxicity and Ethical Principles
toxicity-and-ethical-principles
End-to-end safety assessment covering three dimensions: toxic content generation targeting individuals or groups, tendency to produce false or misleading information, and alignment with ethical principles of…
Benchmarks (3)
-
toxigenlm_evaluation_harness · 0-shot -
truthfulqa_mc1lm_evaluation_harness · 0-shot -
bigbench_hhh_alignment_multiple_choicelm_evaluation_harness · 0-shot
No non-curated collections match your search.