In this article, you will learn how to catch seven common ways AI agents fail in production, using a small regression check for each that makes the failure show up before a release.
Picture an invoice-review agent, a hypothetical one, on a Tuesday afternoon. Its job for tenant-A is invoice INV-17: pull the amount out of doc-17, get an authorized person to approve it, record a review receipt. The extraction call comes back with amount_cents: 129900 in clean JSON. The agent writes two tidy paragraphs ending in “Review complete for INV-17. Done.” and the run goes green. On Friday, finance asks for the receipt. There isn’t one. Every tool call in the transcript succeeded. Does a successful tool response prove a successful task?
A successful response proves that one call returned. The seven failures below sit at seven boundaries of the same run, from the shape of a tool call to the moment the agent should have stopped. Each hides well behind a green status. Each also leaves a symptom in production, and each admits a small regression case, run before a release, that makes it fail on purpose. Anthropic’s January 9, 2026 agent evaluation guidance draws the line this article follows: a trial checks a task outcome, and the outcome is read from the world rather than from the transcript. Every section below uses the same invented fixture: INV-17, source document doc-17 at version v2, page 2, amount 129,900 cents, to be reviewed by an authorized person before a receipt is recorded. No payment moves and no real record exists.
None of the seven checks works without one piece of plumbing: a run identifier that joins everything the run did. Each tool call, each validation or authorization decision, the source version the run read, any receipt an external system returned and the final task status all carry the same ID. Then “did INV-17 get reviewed” has an answer in the record, whatever the agent’s last message said. OpenTelemetry’s correlated traces and spans give you the structure: a trace for the run, a span for each step, every child span naming its parent, so the span holding the receipt points back at the span holding the call. The agent engineering overview distinguishes production observation from offline evaluation; these checks need both.
Source: Machine Learning Mastery — Read the full article on the original site ↗
