Nobody asks me about token budgets in the sales call. Everyone asks me about them three weeks after launch, usually right after finance forwards an API bill that's four times what anyone modeled. This post is about the parts of shipping an AI feature that don't make it into the proposal deck, because they're not exciting, and they're also the reason half of "we added AI" launches quietly get walked back within a year.
Nobody budgets for the retry storm
Here's a failure mode that's obvious in hindsight and invisible until it happens: your LLM call times out or returns a malformed response, so your code retries. Reasonable. Except now you're retrying at whatever concurrency your traffic is running at, each retry is a full-price API call, and if the underlying cause is the provider having a slow afternoon rather than a one-off blip, you're not recovering from a transient error — you're multiplying your bill by however many retries you configured, for as long as the slowdown lasts.
We had exactly this happen on an early project, before we'd built proper circuit breakers into our LLM client wrapper. A provider-side slowdown turned a normal Tuesday into a bill spike that took us most of a day to fully explain to the client. Now every LLM integration we ship gets:
None of that shows up in a demo. All of it shows up in a bill.
Cost per query is a product decision, not a finance afterthought
Every LLM feature has a cost-per-query number, and almost nobody calculates it before launch. It's usually not hard math — input tokens plus output tokens, multiplied by the provider's rate, times expected query volume — but the number tends to be uncomfortable, and uncomfortable numbers get skipped in planning meetings that are running long.
Here's the shape of the tradeoff we walk clients through on basically every project:
The point of that chart is the middle dot, not the right-hand one. Teams reach for the most capable, most expensive model by default because it's the one that impressed everyone in the demo, and then discover in production that a smaller model with well-built retrieval gets 90% of the quality at a fraction of the cost — for most query types. The frontier model earns its keep on a genuinely small slice of hard queries. Routing most traffic to a cheaper model and escalating only the queries that need it is unglamorous engineering, and it's routinely the single biggest lever on unit economics.
Prompts are code. Treat them like it
This one seems obvious once you say it and is still skipped constantly: prompts change behavior in production exactly the way code does, and yet a lot of teams manage them as strings pasted directly into a function, edited in place, with no version history and no way to know which prompt version produced a given historical response.
What we actually do:
- Prompts live in version control, reviewed the same way a code change is reviewed
- Every production response gets tagged with the prompt version that generated it — this matters enormously when a user complains about an answer from three weeks ago and you need to know exactly what was running
- Changes go through the eval set before they ship — a prompt tweak that improves one type of query can silently regress another, and the only way to catch that is running the full eval set, not just eyeballing the specific case you were trying to fix
I've seen a one-line prompt "fix" — adding a clarifying instruction to handle an edge case a user hit — quietly drop accuracy on an unrelated query type by double digits, because the extra instruction changed how the model weighted competing parts of the context. We only caught it because the eval set flagged the regression before it shipped. Without that harness, it ships, someone notices weeks later that "the AI got worse," and nobody can point to why, because there's no record of what changed.
Guardrails aren't a compliance checkbox
The last piece that gets treated as an afterthought is output guardrails — checking what the model actually said before it reaches a user, not just trusting the prompt to have handled it. We run every production response through a lightweight classification pass before it's shown to anyone: does this contain something it shouldn't, does it match the expected format, does it stay within the scope we've grounded it in.
This isn't paranoia. It's the same instinct as input validation on a web form — you don't trust user input, and you shouldn't fully trust model output either, especially not for anything customer-facing or anything that triggers an automated action downstream. The cost of the extra check is a few hundred milliseconds and a small API call. The cost of skipping it is the one time in a thousand it matters, which is exactly the kind of failure rate that feels safe to ignore until it very publicly isn't.
None of this is the part anyone gets excited about in a kickoff meeting. It's also the entire difference between an AI feature that's still running, quietly, a year later, and one that got switched off after an expensive quarter nobody wants to talk about.
— Kalyan