---
title: Benchmark — AI Glossary
description: A benchmark is a standardized dataset and scoring method used to measure and compare model capabilities on a defined task, such as reasoning, coding, or question answering. Shared benchmarks make progress comparable across models, though results can be inflated by data contamination or narrow test design.
canonical: https://prompeteer.ai/glossary/benchmark
---

# Benchmark

A benchmark is a standardized dataset and scoring method used to measure and compare model capabilities on a defined task, such as reasoning, coding, or question answering. Shared benchmarks make progress comparable across models, though results can be inflated by data contamination or narrow test design.

## Definition

Benchmarks provide a common yardstick: everyone runs the same inputs and scores them the same way, so models can be ranked on a capability. Well-known examples include MMLU for broad knowledge and GSM8K for grade-school math reasoning. They are essential for tracking field-wide progress and for choosing models.

Benchmarks have limitations. A model may have seen test data during training (contamination), a benchmark may not reflect real-world tasks, and optimizing to a benchmark can overstate general ability. For this reason teams complement public benchmarks with their own task-specific evals.

## Sources

- [Hendrycks et al. (2021), Measuring Massive Multitask Language Understanding (MMLU)](https://arxiv.org/abs/2009.03300)
- [Cobbe et al. (2021), Training Verifiers to Solve Math Word Problems (GSM8K)](https://arxiv.org/abs/2110.14168)

## Related terms

- [evaluation](https://prompeteer.ai/glossary/evaluation)
- [elo-rating](https://prompeteer.ai/glossary/elo-rating)
- [hallucination](https://prompeteer.ai/glossary/hallucination)