Tokens are not the cost

Ask a team to estimate the cost of an LLM feature and they will open a pricing page, multiply tokens by requests, and produce a number that looks like a rounding error. Per-token prices keep falling, so the number keeps looking better. It is also close to irrelevant, because API spend is routinely the smallest line item in what an AI feature actually costs. The real ledger looks different.

The first hidden line is retries and cascades. Production LLM calls fail in ways the estimate never models: malformed output that needs a re-ask, validation failures that trigger fallbacks to a larger model, timeout retries, and agentic loops where one user action fans out into a dozen calls. Real cost per user action lands at a multiple of the naive per-call estimate. The multiplier is discoverable through measurement and almost never discovered before launch.

The second is context. Costs scale with tokens processed, and context windows invite abuse. Whole documents pasted where a section would do, chat histories dragged along forever, retrieval configured to stuff twenty chunks where five would answer. Context discipline (truncation, summarization, retrieval tuning, caching) is unglamorous engineering that often cuts spend by half or more. Skipping it means paying rent on tokens the model did not need.

The third is evaluation and iteration, which is to say: engineering time. The feature that "worked" in week one gets tuned for weeks against quality complaints. Without an eval harness this tuning is blind, slow, and endless, and senior engineering weeks cost more than most annual API budgets. This is the line item that dwarfs the others, and it is the one no pricing page mentions.

The fourth is the failure tail. What does a wrong answer cost? For an internal drafting tool, a shrug. For anything customer-facing that touches money, health, or legal exposure, single failures can cost more than a year of API spend. This is why review workflows and confidence thresholds are not enterprise theater; they are how you cap the price of being wrong. The cost of a mistake, times its probability, belongs in the budget next to the token math.

And the fifth is switching costs held too long. Model economics shift every few months. Teams that hard-wire one provider into their code cannot chase better price-quality frontiers without a rewrite, so they quietly overpay for quarters at a time. A thin abstraction over the model call, plus an eval set to verify quality after a swap, keeps that option liquid. The eval set, again. It keeps appearing in every honest cost discussion, because measurement is what converts all five of these from surprises into decisions.

The practical takeaway is not that AI features are secretly unaffordable. Most are excellent investments. It is that the token estimate answers the wrong question. Budget the system: calls at observed multipliers, context discipline, evaluation as a permanent line, a priced failure tail, and preserved freedom to switch. Teams that budget this way ship features that stay shipped. Teams that budget from the pricing page get a very educational second quarter.