Why logs don't help when an agent fails
- Agentic AI
- Production
The agent acted on the wrong machine. First instinct: check the logs.
The log shows the action happened. It shows which parameters it used. It shows the result. The one thing it does not show is why those parameters were chosen - and that is the entire fault.
A log records an event; an agent makes a series of decisions
On an ordinary service call a log line is enough, because input and output are deterministically related. Same input, same result.
An agent flow stuffs a pile of decisions in between: which tool was chosen, which documents were fetched, which output failed validation, on which attempt it passed. Each one shapes the outcome and none of them fits in a log line.
What you are left with is "spooler restarted on APP-014". The question you are asking is "why APP-014" and the answer is not in the record.
The decision points are what need recording
We collected each agent run under a single trace, with every decision point as a step of it.
What a step holds: its input and which prior step it came from; the identity rather than a summary of what went into the context (which documents, which tool definitions); the structured output the model produced; the validation result and, on failure, which layer rejected it; and the attempt count.
With those together, "why APP-014" becomes answerable. Classification filed the incident under the wrong category, retrieval pulled a document from that category, and the planner picked the wrong target as a direct consequence. You had to walk back three steps.
Where we couldn't sample
Sampling traces is standard practice at volume - keep one percent, not all of them.
In agent systems it fails for two reasons. The cases you care about are the rare ones, and sampling misses precisely those. There is also this: if an agent's decision cannot be reconstructed afterwards, responsibility for it cannot be taken.
We drew the line this way. Sampling applies to reversible actions that ran automatically. Anything escalated to a person or anything irreversible keeps a full trace. The first group carries the volume, the second carries the weight.
A trace that hangs off nothing is useless
On the infrastructure side, tracing is solved. What is missing on the agent side is tying the trace to something in the business domain.
Every trace carries an incident identifier. From an action on a machine you can walk back to the incident, the plan and the step behind it. Or walk forward from an incident and see what it set off.
Without that link, tracing stays a technical curiosity instead of becoming an operational tool.
The three queries we kept running
Once tracing was in place, three queries got opened constantly. Which steps does the validation layer reject most - that is where the schema is vague. Which steps need more than one attempt - that is where the model struggles. And traces of incidents that were auto-closed and then reopened.
The third turned up something we had not expected: a class of failure that produces no error message at all, and therefore could not have been seen any other way.
Observability in agent systems is not about writing more log lines. It is about building something that connects decisions to each other.
Logs answer "what happened". Traces answer "why". When an agent fails, the question is always the second one.