When I first started building small AI agents (the “give a model some tools and let it loop” pattern), my mental model for cost was: cost ≈ the tokens in my prompt + the tokens in the answer. It isn’t. In an agent loop the biggest line item is usually the conversation you resend every single turn.
Here’s the mechanism, because it surprised me. Each turn, the model gets the entire history back: your instructions, every tool call it made, and — the expensive part — every tool result. A tool that dumps 4 KB of JSON doesn’t cost you 4 KB once; it costs you 4 KB on that turn, then again on the next turn, and again on every turn after, until the conversation ends or gets trimmed. A 20-step task re-reads early outputs ~20 times. I’ve started calling it the “context tax”: you pay rent on old output, not just the price of new output.
A quick way to see it for yourself if your provider returns usage stats: log input_tokens per turn and plot it. In a healthy short task it stays flat-ish; in a tool-heavy loop it climbs roughly linearly and can dominate the total before the task even finishes. That single graph changed how I write agents.
Two things that actually moved the number for me:
- Batch independent tool calls into one turn. If step A and step B don’t depend on each other, don’t take two turns to do them — fewer turns means fewer re-reads of everything before them.
- Summarize fat tool outputs before they re-enter history. If a tool returns a 200-line blob and the model only needs three fields, extract those three fields and drop the blob. You keep the signal and stop paying rent on the noise.
None of this is exotic — it’s just that the cost model is counterintuitive until you watch the per-turn tokens climb. Curious whether others here have measured this on their own agent loops, and what you did about it. Did trimming history hurt answer quality in your case, or was it free?