TL;DR
LLMs are stateless — they remember nothing between calls. To keep a conversation going, the agent re-sends the entire history every turn. History is measured in tokens, and you pay per token, so longer threads cost more and eventually hit the model's context window (its maximum size).
The idea
A token is roughly three-quarters of an English word. Both your prompt and the model's response are measured in tokens, and you pay for both.
Here's the part that surprises everyone: the model is stateless. It has no memory of your previous message. So how does a chat "remember" anything? The agent does the remembering — on every turn it re-sends the whole conversation so far back to the model. The "memory" you experience is the app dutifully replaying the transcript each time.
Every turn re-sends the full transcript, so the tokens you pay for grow as the conversation gets longer.
Analogy
Imagine a brilliant consultant with total amnesia. Every time you walk in, they've forgotten everything — so before each question you hand them the entire notebook of your past conversation to re-read. The longer your history, the thicker the notebook, and the more it costs to have them read it again.
Example: why a one-word reply can still be expensive
Say each turn of a chat adds about 100 tokens. Because the whole history is re-sent, the tokens sent climb every turn even though the user typed a short message:
| Turn | New message | Total history re-sent | Relative cost |
|---|---|---|---|
| 1 | ~100 tokens | ~100 tokens | × 1 |
| 5 | ~100 tokens | ~900 tokens | × 9 |
| 10 | ~100 tokens | ~1,900 tokens | × 19 |
This is exactly why a long Claude Code session — which keeps a large working context of your files, commands, and prior steps — sends far more tokens per step than a quick one-line question. The agent is re-sending its accumulated context so the model stays coherent.
The ceiling: the context window
Re-sending the whole history can't grow forever. Every model has a context window — a maximum number of tokens it can read at once (Anthropic's Sonnet models support a 200,000-token window today: large, but not infinite). Two things follow from that ceiling:
- Something has to give near the limit. Exceed the window and older history must be dropped or summarized to make room.
- Cost and latency rise with every token. More tokens per turn means more money and a slightly slower response. Long conversations feel heavier for a reason.
Try it yourself
This demo breaks a real agent run's cost down step by step. Notice that the LLM calls carry essentially the entire bill — that's the token cost in action. Tools and wrappers are nearly free; the tokens sent to and from the model are what you pay for.
Common mistake
Don't assume a short user message means a cheap call. On turn 50, a one-word "thanks" still drags the entire prior conversation along with it. Managing what goes into the context window is a real engineering job (you'll see it again in the knowledge-base and observability modules).
Key takeaways
Key takeaways
- LLMs are stateless — the agent re-sends the full history every turn.
- You pay per token, for both the prompt and the response.
- Longer conversations send more tokens, so they cost more and run slower.
- The context window caps how much can be in flight at once.
Go deeper: what counts as a token, and how to tame long chats
- Tokenization. Text is split into tokens before the model sees it. Common
words are a single token; rarer words split into pieces ("tokenization" →
token+ization). Punctuation and spaces count too. As a rule of thumb, ~750 words ≈ ~1,000 tokens for English. - Keeping long chats affordable. Real agents don't blindly re-send forever. Common tactics: summarize older turns into a short recap, trim the least-relevant messages, or retrieve only the relevant history on demand (the idea behind knowledge bases in Module 3). Each trades a little fidelity for a lot of saved tokens.
- Why output tokens often cost more than input tokens. Generating text is more expensive than reading it, so most providers price output tokens higher — another reason concise system prompts and responses pay off.