The demo-to-production gap in AI features

Every LLM demo works. That is precisely the problem.

A language model is a machine for producing plausible output, which means the happy path is nearly free. Wire a model to your data, ask it a friendly question in front of stakeholders, and it will perform. The demo is not evidence that the feature works. It is evidence that the model can act well in front of an audience of people who want it to succeed.

Production is a different audience. Production is a user pasting in a 40-page contract at 11pm, a customer typing in Turkish because your form didn't say English only, a prompt that contains the words "ignore previous instructions" because someone read a blog post. The gap between these two audiences is where most AI features die, and the size of that gap is consistently underestimated because the demo felt so close to done.

In our experience the gap has four parts, and they are boring, which is why they get skipped.

Evaluation. If you cannot score the quality of your feature's output, you do not know whether yesterday's prompt change made things better or worse. You are doing vibes-driven development. A test set of even fifty realistic cases with expected outcomes, scored automatically, changes the entire engineering conversation. It turns "I think it's better now" into "we went from 71% to 84% on the hard set."

Failure design. The model will produce wrong answers. The engineering question is never how to prevent that entirely, it is what happens next. Does the wrong answer flow silently into a database? Does a human review the low-confidence cases? Can the user see why the system answered as it did? A feature with a 90% accuracy rate and a good review loop is shippable. The same accuracy with silent failures is a lawsuit generator.

Adversarial input. Users are not adversarial, but their inputs occasionally are, and some users genuinely are. Prompt injection, data exfiltration through crafted documents, and plain abuse need answers before launch, not after the incident. This is not paranoia. It is the same posture we already accept for SQL injection, applied to a new interface.

Cost under load. The demo cost eleven cents. The feature at real usage may cost eleven thousand euros a month, and the difference only becomes visible when you model tokens per request against actual traffic. Model choice, caching, and truncation strategy are engineering decisions with budget consequences, and they deserve the same rigor as any infrastructure sizing exercise.

None of this is glamorous. All of it is the actual work. The teams that ship AI features that survive are not the ones with the cleverest prompts; they are the ones who treated the model as an unreliable component in an otherwise disciplined system, because that is what it is.

The demo is the first five percent. Budget accordingly.