DEV Community

Walker Miller
Walker Miller

Posted on Edited on Originally published at loopandretry.github.io

Your token bill is the cheap part: dimensioning the real cost of an agent

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

The token bill is the cost you can see, because the provider mails you an invoice for it every month. So that's the number that gets optimized: people switch models, trim prompts, cache prefixes, and celebrate a 30% drop in spend. Meanwhile the same agent is holding a worker process open for ninety seconds per task, paging a human for one review in five, and polling a queue that's empty 95% of the time — and none of that is on the invoice. For a surprising number of real workloads, the tokens are the cheapest thing the agent consumes.

This post is a checklist and a model. The checklist is the six axes an agent actually spends across. The model sums them into one number per task, so you can see which axis dominates your workload instead of assuming it's the one with the invoice attached. My retry-budgets post worked out one axis — token cost under retry — in detail. This one zooms out to the other five. If you're focused on tokens alone and ignoring the other axes, you're optimizing the wrong variable; the incident teardown in the $200 postmortem shows how a bill can be nearly 100% non-token.

The six axes

Here they are, roughly in the order people discover they exist:

  1. Tokens. Input (prefill) plus output, priced separately, multiplied by retries and context growth. The one with an invoice.
  2. Latency-as-cost. Wall-clock time isn't free even when the compute is. A task that takes 90 seconds instead of 9 ties up a worker, delays a user, or misses an SLA. If a human is waiting on the result, latency converts directly into wages.
  3. Orchestration and infra. The queue, the worker pool, the state store, the vector database you query for retrieval, the egress on every tool call. This runs whether or not any agent is doing useful work.
  4. Tool-call fees. Every external API the agent calls may bill per request: search APIs, enrichment services, code execution sandboxes, other paid models. An agent that makes twelve tool calls per task can spend more on those than on the model driving them.
  5. Human-in-the-loop. Review, approval, correction, and the escalations the agent kicks up when it's stuck. A human minute is the most expensive resource in the whole system by one to two orders of magnitude, and agents are very good at generating them.
  6. Idle and polling. The cost of being ready. Workers held warm, connections kept alive, queues polled on an interval. This scales with uptime, not with throughput, so it's the cost that's largest exactly when the agent is doing the least.

The trap is that axis 1 is the only one the provider itemizes for you, so it's the only one that gets a budget. The other five are smeared across your cloud bill, your team's calendar, and your latency dashboards, where nobody adds them up per task.


Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article

Top comments (0)