Designing LLM applications that survive contact with production

Most LLM applications we get called in to fix share a single root cause: they were built around the demo, not the deployment. The demo was 92% accurate on three handpicked questions. The deployment is 71% accurate on a stream of 50,000 customer queries a day and the failures are catastrophic. Here is what we do differently on every project, in order of how often they are missed.

1. Treat prompts and configs like code

Prompts live in version control. Every change has a review. Every change runs through an eval suite before it ships. If your prompt is in a CMS, you do not have a LLM product; you have a science project.

2. The eval suite comes before the product

We do not write features until we have a regression suite of at least 200 golden cases that runs in under 10 minutes on PR. Without it, every release is a guess. We have seen teams ship three months of work and find out they regressed a critical customer flow the day an enterprise deal closed.

3. Retrieval has a freshness problem

The hardest thing about RAG is not retrieval accuracy; it is retrieval freshness. Document indexing pipelines silently rot. Build a freshness check into your eval suite: a fixture of questions whose answers should change every week, and an assertion that the system picks them up. We have caught customer-facing production systems serving month-old data because the indexing cron job died six weeks earlier and nobody noticed.

4. Cost is a P95 number, not an average

Average token cost is a vanity metric. P95 prompt length times P95 tool-call depth times the number of concurrent sessions is the cost that ends your runway. We instrument per-session cost ceilings and break the build if any single regression exceeds them.

5. Latency budgets force architecture

If you have a 1.5 second latency budget end-to-end, you do not have time for a 400 ms retriever plus a 1.2 s LLM. The budget forces you to choose small models for first-token, escalate to larger models only on hard queries, and aggressively cache.

6. Guardrails belong outside the model

Models should not be the last line of defense for policy enforcement. They are flaky at it. Run input classifiers and output validators as separate services. The model gets to suggest; the system gets to decide.

7. Observability beats logging

Structured traces that capture the prompt, the retrieved chunks, the tool calls, the model output, the cost, the latency, and the user feedback for every session are the difference between a team that ships and a team that debugs in the dark. We use Langfuse or Helicone or self-hosted OpenTelemetry; what matters is that the trace is a first-class artifact, not a thing you grep logs for.

8. Fallback models are not optional

Every provider has an outage. Every provider has a quality regression. We design for graceful degradation: if the primary model fails, retry once, then fall back to a secondary model with a simpler prompt, then return a graceful error. The user should not see your provider's problems.

9. Ship to 10% of traffic first

Production AI products are won or lost on the first two weeks of shadow traffic. We always ship to 10% with the new model in shadow mode, comparing outputs and costs against the baseline, before flipping the switch.

When not to use an LLM

If your task is deterministic and the rules are well known, an LLM is the wrong tool. We have talked clients out of LLM projects when a 200-line script would do. The right model for the right job is the boring one.