When AI Agents Fail in Production: From Traces to Automated Fixes
An AI agent can return a successful HTTP response and still completely fail its task. The difficult part isn't detecting that the model generated text — it's understanding whether the system actually accomplished what it was supposed to do.
- AI Agents
- TypeScript
- LLMs
- Observability
- Evals
- Production Engineering
- Agent Reliability
The worst failures in software are the ones that return 200. A charging service that double-bills and reports success. A sync job that drops a row and logs 'completed'. An AI agent that replies 'done' without doing the thing. The first two are loud — an exception, an alert, a customer email. The last one is quiet, which is why it is the one I want to write about.
Two kinds of failure
Application failure and agent failure are different species. An application failure is a deviation from a spec: a crash, a timeout, an exception, a 500. It is loud by design. You instrument it, alert on it, page someone, and the trace shows you the exact line. An agent failure is different: every model call succeeded, every tool call executed, every request returned 200 — and the task still failed. The agent selected the wrong tool, passed a malformed argument, abandoned the task after a token budget cut, or concluded a refund was issued when the API response was actually an error message. There is no exception to catch, because the system did exactly what it was built to do. It just built the wrong thing.
What a trace has to show
Because agent failure is silent, diagnosis cannot rely on errors — it has to rely on traces. A useful agent trace records more than the HTTP layer. It records the span tree: every model call with its input and output, every tool call with its arguments and result, the intermediate state between steps, retries, latency, token usage, and the final outcome the agent believed it achieved. Think of each event as a frame in a film. You cannot say where the film goes wrong until you can watch it frame by frame.
In TypeScript, the events in a trace can be modeled as a discriminated union. This is the shape I keep coming back to:
type Span = {
spanId: string;
parentId?: string;
kind: 'model' | 'tool' | 'retry' | 'guard';
startedAt: string;
endedAt?: string;
durationMs?: number;
tokensIn?: number;
tokensOut?: number;
};
type ToolCall = {
tool: string;
arguments: unknown;
result: string;
error?: string;
};
type AgentEvent =
| { type: 'model'; model: string; input: string; output: string }
| { type: 'tool'; call: ToolCall }
| { type: 'state'; snapshot: unknown }
| { type: 'finish'; outcome: 'task_complete' | 'aborted' | 'gave_up' };Two details matter here. First, tool arguments are stored raw — you cannot debug what the agent actually attempted if you only store the outcome. Second, the 'state' event captures the intermediate world view the agent was operating on. Most agent bugs live in the gap between what the agent believed and what was true.
The failure modes that repeat
Read enough production traces and a taxonomy of agent failure emerges. Incorrect tool selection — the agent reaches for the wrong function because the description is ambiguous. Malformed tool arguments — the model passes a string where a number belongs, and the schema rejects it. Hallucinated tool results — the agent invents a response instead of reading the API reply. Context loss — a long conversation truncates and the agent forgets a constraint from step two. Infinite loops — the agent repeats the same failed tool call, each retry amplifying the damage. Unnecessary tool calls — the agent burns budget and time verifying what it already knows. Premature completion — the agent declares victory at the first plausible checkpoint. Failure to verify tool results — a payment gateway rejects the charge and the agent reports success. And, quietly the worst, prompt regressions: the same codebase ships a new prompt and a previously reliable flow starts failing at a 5% clip. None of these raise an alert. All of them produce a confident, well-formed answer.
Reading a trace like a post-mortem
Once you have a trace, the first question is: where did this run diverge from a good run? The cleanest technique is diffing the failed trace against a passing one — step by step, event by event — until the sequences no longer match. That first divergence is your root-cause candidate. It is the same discipline as a code review: you are looking for the one frame where the film cuts.
function firstDivergence(
passed: AgentEvent[],
failed: AgentEvent[]
): number {
const n = Math.min(passed.length, failed.length);
for (let i = 0; i < n; i++) {
if (passed[i].type !== failed[i].type) return i;
if (
passed[i].type === 'tool' &&
failed[i].type === 'tool' &&
passed[i].call.tool !== failed[i].call.tool
) {
return i;
}
}
return n;
}The divergence usually lands in one of three places: the tool call that should not have happened, the argument that was malformed, or the model call where the agent lost the plot. From there you can ask the question that matters — was this a prompt problem, a tool-description problem, a context-management problem, or a guardrails gap? The answer determines whether the fix is a string change or an architectural change, which is exactly why the trace has to exist before you start guessing.
Turning a failure into a test
A reproduced production failure is an asset. The step that separates teams that get good at agents from teams that don't is this: they convert every confirmed failure into a regression evaluation before writing the fix. The eval must capture two things — the deterministic facts (the correct tool was selected, the arguments were valid, the state was unchanged) and the semantic question (did the agent actually satisfy the user's request?). Deterministic checks are cheap and precise. The semantic check usually needs a model-based judge, which is powerful and imperfect: it can hallucinate correctness, it drifts when the judge model updates, and it will sometimes disagree with itself on the same trace. The mitigation is calibration — run every new judge against a set of known-good traces so you can measure its false-positive and false-negative rates before you trust it in CI.
evals.register({
name: 'refund-window-enforced',
case: {
input: 'User asks to refund a purchase from 8 months ago.',
expected: 'Agent refuses and explains the 90-day policy.',
},
facts: [
(trace) => trace.some((e) => e.type === 'tool' && e.call.tool === 'refund'),
(trace) =>
trace.some((e) =>
e.type === 'model' && /90-?day|window/.test(e.output)
),
],
judge: 'semantic',
});The fix loop
Now the loop closes. Production trace → detect failure → reproduce → diagnose → create regression eval → patch → test → review → deploy → observe again. The order is the discipline. The eval comes before the patch, not after: it encodes the bug as a test, so the patch is done when it passes, not when someone feels confident. The patch itself should be boring — a prompt change, a stricter tool description, a guard, a validation layer. Then the human review, because agents amplify whatever you ship: a subtle prompt regression at 2% per run becomes a slow, compounding disaster you will discover in a revenue report, not in an alert.
Guarding the loop
None of this works without hard guardrails around the loop itself. The agent needs a security boundary: it cannot reach production systems directly, its tool surface is an allowlist, and irreversible actions — sending mail, deleting records, moving money — require human approval. Tool calls should be validated against a schema at runtime, not trusted because the model said so. The loop itself needs a budget: a maximum step count and a token budget so a runaway loop terminates before it can do damage, with a sentinel that kills a run that exceeds its allowed latency. And retries must be idempotent — every side-effecting tool call carries an idempotency key, so a retried request cannot double-charge or double-send.
const MAX_STEPS = 12;
const TOKEN_BUDGET = 32_000;
for (let step = 0; step < MAX_STEPS; step++) {
if (tokensUsed > TOKEN_BUDGET) {
await emit({ type: 'finish', outcome: 'aborted' });
break;
}
const next = await agent.step();
if (next.done) break;
}
const blocked = new Set(['send_email', 'delete_record']);
function guardToolCall(call: ToolCall): ToolCall | null {
if (blocked.has(call.tool)) return null;
if (!schemas[call.tool]?.safeParse(call.arguments).success) return null;
return call;
}Reliable agents are not a prompt-engineering achievement. They are a systems achievement, built at the intersection of four disciplines that predate LLMs: software engineering, because the agent is a program with control flow; observability, because you cannot fix what you cannot see; evaluation, because you cannot ship what you cannot test; and developer tooling, because the loop only closes when turning a failure into a regression eval is a ten-minute chore rather than a two-day project. The model generates text. The system is what you build around the model — and the system is where the engineering happens.
An agent you cannot trace cannot be trusted. An agent you cannot test cannot be shipped. And an agent you cannot patch and redeploy within the hour is not software — it is a demo. The good news is that the discipline is old. We knew how to do this before anyone said 'agent.' We just had to learn to apply it to a system whose output is prose and whose failure is a confident, well-typed sentence.
Written by Rajat Yadav. If you enjoyed this, say hello in the guestbook or read another essay.