Agent engineering · in public

I build AI agents that know when to ask a person.

Agents that do real work with real tools. Evals decide whether a change ships, and audit logs, redaction, and red-team cases are part of the build. I came to agents from security, so a wrong approval counts as far worse than a needless question.

See the work →devheadx@gmail.comOpen to AI agent engineering roles
Test invoices right
155/158
False approvals
0
Cost per 100 invoices
$0.19
Median time per invoice
8.7 s
An Accounts-Payable Agent · measured 2026-10-12 · source private for now

Build log

  1. ap-agentClaude Haiku 5.5 decided 155 of 158 held-out test invoices correctly, with zero false approvals.
  2. evalsHaiku 4.5 and Haiku 5.5 both decide 155 of 158 test invoices correctly with zero false approvals.
  3. audit-controlsAll 18 red-team invoices in the test split end in their required outcome.
  4. mcp-toolsThe full test split, replayed with every ERP tool served over MCP, hit all 158 recordings and scored the same 155 of 158, with no row different from the in-process run.
  5. knowledge-layerOn 42 test invoices whose answer depends on the firm's contracts and notes, the agent went from 12 right without knowledge to 36 with it, with zero false approvals.
  6. voice-receptioniststarted
  7. evalsstarted
  8. knowledge-layerstarted
  9. slack-reviewTested live in a real workspace.
  10. slack-reviewstarted
  11. audit-controlsstarted
  12. mcp-toolsstarted

Agent engineering

Most of this work is one product, an accounts-payable agent, built a layer at a time. Its source is private for now, so every row is marked building: you can read what it measured and when, but you cannot clone it yet.

Agents & Tool Use

10

DO · Agents & Tool Use

An Accounts-Payable Agent

building

Reads a supplier's invoice, finds its purchase order in a messy legacy ERP, and proposes approve, flag, or escalate.

Done when

A fresh clone reaches a working demo in under five minutes with no API keys, and the source is public.

  • python
  • claude-api
  • tool-use
  • sqlite
Open the writeup →Source private · Measured 2026-10-12
15

DO · Agents & Tool Use

Knowledge The Agent Can Cite

building

Contracts searched and cited by chunk, prices read from tables, and memories that stay off until a second person approves.

Done when

The six extra misses are fixed on the development split and the no-harm check passes on test. The source is public.

  • hybrid-search
  • bm25
  • fastembed
  • sqlite-fts5
Open the writeup →Source private · Measured 2026-10-09

Evals

11

DO · Evals

Evals That Decide What Ships

building

A replayable eval suite with a sealed holdout and a regression gate, scoring the same 158 invoices across models.

Done when

Continuous integration replays every committed baseline on each push, the regression gate blocks a change that loses accuracy or safety, and the source is public.

  • python
  • recorded-replay
  • anthropic
  • openai
  • gemini
  • ollama
Open the writeup →Source private · Measured 2026-10-12

Guardrails & Audit

12

DO · Guardrails & Audit

An Agent You Can Audit

building

A hash-chained audit log, personal data redacted before any model call, and a policy the model cannot overrule.

Done when

A reader can clone the repository, change one audit record, and watch verification fail. The source is public.

  • hash-chain
  • pii-redaction
  • policy-engine
  • red-team
Open the writeup →Source private · Measured 2026-10-12

Integrations

13

DO · Integrations

Human Review In Slack

building

Escalated invoices arrive in Slack with approve and reject buttons, behind the same rules as the command line.

Done when

The manual test checklist passes on a fresh workspace, and the source is public.

  • slack-bolt
  • socket-mode
  • block-kit
Open the writeup →Source private · Measured 2026-10-07
14

DO · Integrations

ERP Tools Over MCP

building

The agent's six ERP tools served over the Model Context Protocol, read-only, with results redacted before they leave.

Done when

Any MCP client can connect to the server from a fresh clone, and the source is public.

  • mcp
  • stdio
  • python
Open the writeup →Source private · Measured 2026-10-12
16

DO · Integrations

A Receptionist On The Phone

building

A phone agent that books an appointment only after the caller confirms an exact read-back. Synthetic, not production.

Done when

A live call books one appointment end to end, and a caller who interrupts the read-back is never booked.

  • typescript
  • twilio
  • sqlite
Open the writeup →Source private

Security

Where I started. The lab is in build, and 8 more projects are planned, each with the bar it has to clear.

Detection & Response

01

DO · Detection & Response

Home SOC Lab

building

One command starts a SIEM, a victim host, and an attacker host. Nothing is click-built.

Done when

A replayed attack raises the matching Wazuh alert, with the alert as evidence.

  • wazuh
  • docker
  • atomic-red-team
  • ubuntu
Open the writeup →

All security work →