11
Evals That Decide What Ships
A replayable eval suite with a sealed holdout and a regression gate, scoring the same 158 invoices across models.
What it measured
Haiku 4.5 and Haiku 5.5 both decide 155 of 158 test invoices correctly with zero false approvals. Haiku 5.5 costs $0.19 per 100 invoices against $1.69, and takes 8.7 seconds per invoice against 14.9. Fixed rules alone get 125 right and approve four they should not.
Measured 2026-10-12 · the source is private, so this cannot be checked yet
Done when
Continuous integration replays every committed baseline on each push, the regression gate blocks a change that loses accuracy or safety, and the source is public.
Every model call in an eval run is recorded. The recordings are committed, so a run replays for free and gives the same answer every time. Only a deliberate live run spends money.
The test split was never used to tune anything. A third set, the holdout, is sealed: a test fails if any file in it changes, and nobody reads it until the final review.
The gate checks safety before accuracy. False approvals, red-team misses, personal data in stored output, and broken audit chains must all be zero. Accuracy may fall by at most two points. One gateway serves four providers (Anthropic, OpenAI, Google, and a local model through Ollama), so swapping the model is a config change and a replay.
What the numbers show: on this split the two Haiku models score the same, so accuracy can no longer separate them. Cost and time can.