10
An Accounts-Payable Agent
Reads a supplier's invoice, finds its purchase order in a messy legacy ERP, and proposes approve, flag, or escalate.
What it measured
Claude Haiku 5.5 decided 155 of 158 held-out test invoices correctly, with zero false approvals. The three misses are scanned PDFs, which never reach a model and go to a person.
Measured 2026-10-12 · the source is private, so this cannot be checked yet
Done when
A fresh clone reaches a working demo in under five minutes with no API keys, and the source is public.
The agent runs a hand-written tool-use loop against a SQLite “ERP” that is messy on purpose: one vendor entered three ways, amounts stored as text with currency symbols, dates in three formats, purchase-order numbers missing their prefix. It proposes an outcome and cites the evidence for it. A policy engine, not the model, decides what happens next. Most invoices go to a person, and a wrong approval counts as far worse than a needless escalation.
The failure worth recording came from vendor matching. A fuzzy name score cannot tell “the same company, entered twice” from “a different company with a close name”: both score above 80. A test set of twenty lookalike vendors produced eleven false approvals. The fix was a rule in code, not a better prompt: an approval now requires a shared address as well as a close name. The same set then produced none.
The layers around the agent each have their own row: the evals, the controls, review in Slack, the MCP tools, and the knowledge layer.