TL;DR
Traces aren't just for speed and cost — they're your debugger. Most agent failures show up as a specific signature in the trace: an error span the answer hid, a retrieval that returned the wrong chunks, a tool given bad arguments, or a runaway loop. Match the symptom to the span and attribute that explains it.
The answer lies; the trace doesn't
A wrong answer tells you that something failed, not what. The model often papers over a failure — a tool errored, retrieval came back empty — and produces a confident answer anyway. The trace is where the real story lives: every span's status, inputs, and outputs are recorded even when the final reply hides them. Debugging an agent is mostly reading the trace for the step that went wrong, not staring at the answer.
Symptom → where to look
Each common failure has a trace signature. Learn the mapping and debugging becomes lookup, not guesswork:
| Symptom | Where to look | What you'll find |
|---|---|---|
| Confident but wrong answer | The tool / retrieval span's output | Wrong chunk retrieved, or a tool result the model misread |
| Answer ignores your documents | The retrieval span | Empty or low-similarity results — nothing useful was retrieved |
| Tool 'didn't work' | The tool.execute span's status + input | An error status, or arguments the model filled in wrong |
| Run is slow | The longest bar's duration | A big LLM call with bloated input (Module 6 L4) |
| Run never finishes / loops | Repeated tool.execute spans | The model calling the same tool over and over — hit the max-iterations cap |
That's the checkpoint: when every span shows OK but the answer is wrong, the failure isn't an error — it's bad data. Open the retrieval and tool spans and read their outputs. A retrieval that returned the wrong chunk, or a tool result the model misinterpreted, is a successful span that still produced a wrong answer.
Analogy
A wrong answer is a patient saying "I feel awful." The trace is the chart — vitals, labs, history. You don't diagnose from the complaint; you read the chart for the one number that's off. Each symptom above points at which number to read first.
The debugging loop
- Reproduce the bad run and open its trace.
- Classify the symptom — wrong, ignored-docs, tool-failed, slow, looping.
- Jump to the signature span from the table and read its attributes (status, input, output, tokens).
- Fix upstream — the chunking/K (Module 4), the tool description (Module 2), or the prompt — then re-run and compare traces.
Common mistake
Debugging by re-prompting the model ("try harder," "be accurate") instead of reading the trace. If retrieval returned the wrong chunk, no amount of prompt-tweaking fixes it — the model never had the right facts. The trace tells you whether the problem is input (retrieval/tool) or reasoning (the model), and they need completely different fixes.
Key takeaways
Key takeaways
- Traces are a debugger: failures have specific span signatures.
- A wrong answer with all-OK spans means bad data — read tool/retrieval outputs.
- Match symptom → span → attribute instead of guessing.
- Fix upstream (retrieval, tool, prompt); re-run and compare traces.
Go deeper: from one trace to many — observability at scale
Reading one trace debugs one run. The same span data, aggregated across many runs, is what "observability" really buys you:
- Aggregate metrics. Average latency, p95 latency, cost per run, error rate — computed by rolling up span attributes across thousands of runs. A single slow run might be noise; a p95 that doubled is a regression.
- Filtering and search. "Show me every run where a tool span errored" or "every run over $0.10" turns a pile of traces into a worklist of things to fix.
- Regressions over time. Watching cost-per-run or error-rate as you change prompts, models, or retrieval settings tells you whether a change helped or hurt — which is the bridge to the next module, Evals, where you measure quality deliberately instead of waiting for a bad run to surface.
A trace is the unit; observability is the aggregate. You learned to read the unit here so the aggregates mean something.