Somewhere around 2023, "we're adding AI to the product" became a sentence founders said in board meetings the way they used to say "we're moving to the cloud." It sounded like a decision. Most of the time it wasn't one yet — it was a hope, wearing a decision's clothes.
I run the AI/ML side of delivery at Alminds, which mostly means I spend my week doing two things: building systems that use large language models, and talking clients out of building the wrong one first. This post is about the second part, because it's the part nobody puts in the pitch deck.
The gap between a demo and a product
An LLM demo is easy. You write a prompt, you paste in some context, you get a genuinely impressive answer, and everyone in the room says "ship it." I've watched this happen a dozen times and I understand the instinct completely — the first time GPT-4-class output surprises you, it feels like the hard part is over.
It isn't. The hard part starts the moment real users start typing things you didn't anticipate, in a context you didn't test, at a volume your budget didn't model. A demo has one input. A product has an unbounded set of them, forever, and about 5% of that set is going to be adversarial, malformed, or just weird in a way your prompt never accounted for.
We built a support-ticket triage system for one of our logistics clients last year — routing incoming tickets to the right team, tagging urgency, drafting a first response. In the demo, it was 95% accurate on a curated set of sample tickets. In week one of production, accuracy on real tickets was closer to 78%, because real tickets contain things like three unrelated issues jammed into one email, screenshots with no caption, and customers writing in a mix of English and Bengali mid-sentence. None of that was in our test set, because we wrote the test set, and we don't think like an anxious customer at 11pm.
What actually closes that gap
That eval set becomes the thing every prompt change, every model swap, every RAG tweak gets measured against — not vibes, not "it feels better," an actual number that goes up or down. This sounds obvious written down. It is routinely skipped in practice, because building an eval set is unglamorous and doesn't produce a demo-able output, and teams under deadline pressure gravitate toward the visible work.
Here's roughly the pipeline shape we've converged on for retrieval-augmented systems, which is most of what we build:
Every box in that diagram has a failure mode we've hit in production: retrieval that returns technically-relevant-but-useless chunks, re-ranking that's too aggressive and throws away the one document that mattered, guardrails that are too loose and let the model confidently answer a question it has no grounding for. The eval harness is what tells you which box broke, instead of just "the answer was wrong" — which is not, on its own, an actionable bug report.
Hallucination isn't a bug you fix once
Clients often ask us how we "solve" hallucination, as if it's a checkbox. It isn't. It's a rate you manage, the same way you manage error rates in any other system — through architecture, not through a single clever prompt.
The things that actually move the needle, in rough order of impact:
- Constrain the model to only answer from retrieved context, and make it say "I don't know" explicitly when nothing relevant comes back — this alone eliminates most of the worst hallucinations, because most of them happen when the model is asked something outside its grounding and tries to be helpful anyway
- Cite sources in the response, so a human can verify in two seconds instead of taking the answer on faith
- Route low-confidence answers to a human rather than shipping them — for our logistics client, anything below a confidence threshold gets flagged for manual review instead of auto-sent
- Re-run the eval set on every model version change — a "better" model from the provider's benchmarks can genuinely regress on your specific task, and you won't know unless you're measuring your task, not theirs
None of this is exotic. It's mostly just applying the same engineering discipline — testing, monitoring, graceful degradation — that we'd apply to any other system, to a component that happens to be non-deterministic. The teams that get burned by AI in production are almost always the ones that treated it as magic instead of as a component with a failure rate, and skipped the boring infrastructure around it accordingly.
If you're looking at an AI feature and the plan is "we'll prompt-engineer it until it works," that's the point to slow down and build the eval set first. It's less exciting than the demo. It's also the entire difference between a feature that survives contact with real users and one that gets quietly turned off three months later.
— Kalyan