Why Now Daily.

Published

AI Evaluations: How Model Performance Is Tested

An AI evaluation is a structured test of a defined property, task or risk under stated conditions. Benchmarks make some comparisons repeatable, but scores depend on the test set, grading method, system configuration and statistical uncertainty, so no single result establishes broad reliability or safety.

Timeline

  1. 2024-02-09: The UK AI Safety Institute published its approach and emphasized that evaluations are preliminary, limited and do not designate a system as safe.
  2. 2026-01: NIST released an initial public draft of practices for automated language-model benchmark evaluations.
  3. 2026-08-07: NIST announced the draft TEVV-Athlon framework for customized assessment of AI systems and real-world outcomes.
  4. 2026-10-12: Current NIST, UK AISI and OpenAI evaluation guidance was reviewed for this guide.

An AI evaluation is a planned measurement exercise, not simply a public leaderboard. The evaluator begins with a claim or decision: for example, whether a system answers a defined class of questions accurately, whether one configuration outperforms another, or whether a safeguard resists a stated attack. NIST says test, evaluation, verification and validation provide evidence about whether systems meet goals while minimizing negative impacts. The evidence is meaningful only when the objective, tested system and conditions are described. [1][2]

A benchmark is one common evaluation tool. It combines a set of test items with a metric or scoring method so systems can be assessed in a repeatable way. Some tasks have exact answers and can be graded automatically; others need rubrics, expert review, user preferences or measurements of task completion. NIST guidance says the benchmark must fit the evaluation objective: a multiple-choice science test, a conversation preference test and an agent-security test measure different constructs even if each reports a percentage. [2][3]

The system being tested includes more than a model name. Prompt instructions, available tools, retrieval data, sampling settings, time or token budget and software that manages the interaction can all alter performance. For agents, this surrounding setup is often called a harness. OpenAI’s evaluation guidance notes that a harness may preserve state, retry failed actions or help a model recover, changing whether a multi-step capability appears. A fair comparison therefore holds relevant conditions constant or clearly explains why they differ. [5]

A pass rate is usually the share of scored items or trials that meet the chosen success rule, but the denominator and grading rule matter. One attempt per item measures something different from success within several retries. Strict exact-match grading differs from human judgment or an automated model judge. Reports should disclose item count, number of trials, scoring criteria, exclusions and any human or automated graders. NIST also warns that language-model outputs can be nondeterministic, so a reported metric is an estimate with uncertainty rather than an immutable trait. [2][3]

Test quality can fail in several ways. Items may be ambiguous, broken or too narrow for the claim. Training data may contain benchmark questions or close variants, producing contamination. A model may exploit shortcuts in the scorer without demonstrating the intended skill, refuse in a way that masks capability or behave differently when it detects a test. OpenAI’s third-party evaluation guidance lists these validity threats, while NIST recommends inspecting benchmark construction, coverage, difficulty, item format and grading before interpreting results. [2][5]

A higher score does not automatically predict better performance everywhere. NIST distinguishes performance on a fixed benchmark from generalized performance on a wider population of related tasks and notes that gains on a benchmark have not always transferred to similar work. Real use adds changing inputs, users, tools and consequences. Strong evidence therefore combines appropriate benchmarks with representative task studies, error analysis, human baselines where relevant and monitoring after deployment rather than stretching one score beyond its tested scope. [1][3]

Safety evaluations are also scoped tests, not certificates. The UK AI Safety Institute says its work measures particular safety-relevant capabilities and risks, is subject to scientific limitations and does not designate a system as safe. Red teaming can search for failures under adversarial conditions; safeguard evaluations can measure resistance to specified attacks; and impact studies can examine outcomes for people or organizations. Passing one suite does not rule out untested hazards, future attacks or failures caused by a different deployment configuration. [4][5]

To read an evaluation responsibly, ask five questions: What exact claim was the test designed to support? Which model version and system configuration were tested? Do the tasks and users resemble the intended deployment? How were answers scored, repeated and summarized, including uncertainty? What limitations, missing cases and validity checks were reported? A useful evaluation narrows uncertainty about a decision. It does not turn a complex system into one universal number, and conclusions should remain no broader than the evidence collected. [1][2][3][4][5]

Sources

  1. NIST — The TEVV-Athlon Framework for Evaluating AI Systems
  2. NIST — Practices for Automated Benchmark Evaluations of Language Models
  3. NIST — Expanding the AI Evaluation Toolbox with Statistical Models
  4. UK AI Safety Institute — Our Approach to Evaluations
  5. OpenAI — A Shared Playbook for Trustworthy Third-Party Evaluations

Related stories