TL;DR
Across almost every run, both latency (how long) and cost (how much) are driven by the same thing: the LLM calls. Tools and retrieval are fast and free in the trace's accounting. Within a model call, the bill is mostly input tokens — everything you sent. Slow and expensive almost always means: too much went into the model.
The LLM call dominates both
Two numbers matter for almost every run, and one kind of span drives both:
- Latency. Tool calls and retrievals are usually fast — milliseconds to a fraction of a second. The model call is where the seconds go, because the model has to read your whole prompt and then generate a response token by token.
- Cost. You only pay for the model; tools and retrieval are free in the trace's accounting. And within a model call, the bill is mostly the input tokens — everything you sent: the system prompt, the conversation history, and any retrieved context you stuffed in.
So a run that feels slow and expensive is usually telling you the same story: too much went into the model.
Analogy
The model call is the one expensive ingredient in the recipe; everything else is salt and water. If the dish costs too much, you don't scrutinize the salt — you look at how much of the expensive ingredient went in. Input tokens are that ingredient.
Cost is a stacked bar
The demo breaks one run's cost down by span. Notice the model call is the entire bill — the tool call and retrieval contribute nothing. That's the shape of nearly every run, and it's why "open the LLM span" is almost always the right first move.
How to actually reduce it
When you want a run cheaper or faster, the trace points straight at the lever — and it's the checkpoint:
- Open the longest / most expensive span — it'll be an
llm.complete. - Read its
input_tokens. A big number means a bloated prompt: too much retrieved context, a giant system prompt, or a long conversation history. - Trim the input. Fewer retrieved chunks (lower K, Module 4), a tighter system prompt, or summarising old turns all cut input tokens — which cuts both cost and latency at once.
That's the whole loop: read the timeline, find the dominant LLM call, shrink what you feed it.
Common mistake
Optimising the cheap spans. Shaving 20ms off a tool call feels productive, but if the model call is 3 seconds and $0.049 of a $0.05 run, you've optimised the salt. Always confirm where the cost and time actually are in the trace before changing anything — the dominant LLM span is nearly always the only one worth touching.
Key takeaways
Key takeaways
- LLM calls dominate both latency and cost; tools/retrieval are fast and free.
- Within a model call, input tokens are most of the bill.
- To cut cost/latency: open the LLM span, read input tokens, trim the prompt.
- Cutting input tokens reduces cost and latency together.
Go deeper: input vs output tokens, and the multi-call multiplier
Two refinements once you're optimising for real:
- Input vs output pricing. Most providers price output tokens higher per token than input — but agents usually send far more input than they generate, so input still dominates the total. The exception is long-generation tasks (writing an essay, large code edits), where output can lead. Read both token counts; don't assume.
- The multi-call multiplier. A tool loop (Module 2) re-sends the whole growing conversation on every cycle. A 4-cycle run isn't one prompt — it's four, each longer than the last. That's why tool-heavy agents cost more than single-shot calls, and why keeping cycles few (and tool results lean) is a cost lever, not just a tidiness one.
- Caching. Some providers let you cache a stable prefix (a big system prompt or fixed context) so repeated calls don't re-pay for it. When a long, unchanging prefix dominates your input tokens, prompt caching can cut cost sharply without changing behaviour.
The trace shows you the per-span token counts; these refinements tell you which knob those numbers point to.