How to size an LLM eval suite without losing a quarter to it

A good LLM eval suite is the difference between shipping an LLM product and shipping a science project. A bad one is worse than nothing because it gives false confidence. After building dozens, here is what we have learned about sizing them correctly.

What to actually measure

Four metrics matter: correctness on a representative task distribution, refusal rate on out-of-scope queries, cost per successful task, and p95 latency. Everything else is a vanity metric. We have seen teams build 50-metric dashboards that nobody reads while shipping obvious quality regressions.

The 200-case minimum

We start every project with 200 hand-curated cases that represent the production task distribution. 200 is enough to detect a 5 percentage point regression with statistical confidence. Below 200, the noise floor swallows the signal.

Golden sets versus synthetic data

Golden sets are cases written by humans who know the answer. Synthetic data is cases generated by another model or by templating. Golden sets are precious and slow. Synthetic data is cheap and lossy. We use golden sets for release gates and synthetic data for ongoing regression detection and for growing the suite to cover long-tail cases.

LLM-as-judge pitfalls

A model grading its own outputs tends to give itself high marks. A model grading another vendor's outputs tends to prefer its own vendor. A model grading factual correctness without ground truth hallucinates plausible-sounding rationales. We use LLM-as-judge for style and consistency, never for factual release gates. The judge must have access to the same source of truth the candidate did.

CI gates that do not lie

Every PR runs the eval suite. We break the build on regressions of more than 2 percentage points on the golden set or any failure of the freshness check (the fixture of questions whose answers should change). Engineers cannot merge when the suite is red; the only acceptable override is a written postmortem.

Keeping the suite trustworthy

A stale suite is a misleading suite. Every production failure case becomes an eval case within 48 hours. Every new feature ships with its eval cases. Every quarter we prune cases that no longer represent production traffic. A suite you do not maintain is a number you do not trust.