Benchmarks are standard test sets used to compare models.
MMLU, GSM8K, HumanEval and SWE-bench are widely cited examples.
They allow rough comparison across models and versions.
Contamination is a real problem, since benchmarks leak into training data.
A public benchmark score rarely predicts performance on your specific task.
Benchmarks are standard test sets used to compare models against each other.
MMLU covers general knowledge across many subjects.
GSM8K covers grade-school maths, and HumanEval covers writing small programs.
SWE-bench uses real GitHub issues and is much closer to real work.
They allow rough comparison, and every model release leans on them heavily.
Contamination is the central problem.
Benchmarks are public, so they end up in the training data of the next model.
A model may have memorised the answers rather than solved anything.
Labs check for this with varying rigour and cannot check what they did not think to look for.
Benchmarks also saturate, since once everyone scores 90 per cent the test stops distinguishing.
They measure narrow capabilities that may not resemble your task at all.
A model topping a coding benchmark may be worse on your codebase.
Held-out and private benchmarks such as arena rankings resist contamination better.
Read them as a rough filter for which models to try.
Then test on your own data, which is the only benchmark that predicts your results.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.