How Many Evals Are Enough?

You change a prompt, rerun your eval suite, and the score goes from 84% to 86%. Ship it?

It depends almost entirely on how many examples are in your eval set. With 50 examples, that 2-point jump is one extra correct answer, which is well within noise. With 2,000 examples, it is probably real.

The quick math

For a pass/fail metric, the standard error of your score is roughly:

SE ≈ sqrt(p × (1 − p) / n)

At p = 0.85:

Eval set size (n)	Standard error	95% CI (±)
50	5.0pp	±9.9pp
200	2.5pp	±4.9pp
1,000	1.1pp	±2.2pp
5,000	0.5pp	±1.0pp

To detect a 2-point improvement with any confidence, you need around a thousand examples, or a smarter comparison.

Use paired comparisons

Most teams compare two independent scores. But you ran both versions on the same examples, so compare them pairwise: count the examples where the new version wins, loses, or ties. A paired test (McNemar's test for pass/fail) ignores the examples both versions get right or both get wrong. It is far more sensitive at the same n.

Precision is not coverage

A large eval set gives you a precise number. It does not guarantee you are testing the right things. Three separate questions need three separate answers:

1. Precision: is the score stable enough to compare versions? (sample size)
2. Coverage: does the set represent real traffic, including the rare but important slices? (stratified sampling)
3. Discovery: are you still finding new failure modes? (saturation)

A 5,000-example set made entirely of easy cases will tell you, very precisely, the wrong thing.

A practical starting point

• Start with 100–200 hand-labeled examples sampled from real traffic, stratified by the slices you care about.
• Report a confidence interval next to every score.
• Compare versions pairwise, on the same examples.
• Add examples whenever production surfaces a new failure, and track how often new failure modes still appear.

I go deeper on each of these, with runnable code, in the LLM Evals: Foundations track.