The demo took two weeks. Production never came. If that sentence describes a GenAI initiative inside your company, you're in the majority. Industry surveys consistently put the share of enterprise GenAI pilots that never reach production at 70–90%, and our own engagement data sits squarely in that range: of the pilots clients bring to us for "rescue," roughly four in five stalled for reasons that had nothing to do with the model.
That last clause matters. Teams instinctively blame the model — not smart enough, hallucinating too much, too slow. Then a new model ships, the pilot gets re-run, and it stalls again. After a year of taking GenAI systems to production across financial services and manufacturing, we see three failure patterns account for almost every stall. Each has a fix.
Fix 1: Replace vibe checks with evaluation discipline
Most pilots are judged the way demos are judged: a stakeholder types five questions, likes four answers, and the project gets a green light it can't sustain. The first time the assistant confidently misquotes a contract clause in front of a regulator-adjacent team, trust collapses — and trust, once lost, doesn't come back with a patch.
Production teams treat quality as an engineering measurement, not an opinion:
- A golden dataset — 100–300 real questions with verified answers, sourced from the people who will actually use the system, refreshed quarterly.
- Regression gates in CI — every prompt change, retrieval tweak or model upgrade runs against the golden set before it ships. Score drops block the release, exactly like failing unit tests.
- LLM-as-judge with human calibration — automated grading for scale, spot-checked by domain experts so the judge itself stays honest.
The teams that ship don't have better models. They have better tests.
Fix 2: Fix the data before the prompt
Retrieval-augmented generation is only as good as what it retrieves. In stalled pilots we routinely find the assistant grounded on a document dump: three versions of the same policy, expired contracts mixed with active ones, scanned PDFs that OCR mangled into noise. The model then does exactly what it was asked — it summarizes garbage fluently.
The fix is unglamorous and decisive: treat the knowledge corpus like a governed data product. Deduplicate and version documents so only the current truth is retrievable. Attach metadata — owner, effective date, jurisdiction — and filter on it at query time. Set freshness SLAs so the corpus is re-synced on a schedule, not "whenever someone remembers." On a recent financial-services engagement, corpus governance alone moved answer accuracy more than any prompt or model change we tested — and it's why we insist data platform and governance work precedes the AI build, not follows it.
Fix 3: Put the answer where the work happens
The third stall is the quietest: the pilot works, accuracy is fine, and nobody uses it. A standalone chat window asks busy people to change their habits — to leave the CRM, the ticketing queue, the ERP screen — to go ask a bot. Adoption curves for "destination" assistants sag within weeks.
Assistants that stick are embedded: the answer appears inside the support ticket, the contract-review pane, the planner's console — with the source cited and an action button next to it. That's also where ROI becomes measurable, because usage maps to a workflow with a known cost. One client's support copilot resolves the majority of internal queries today, not because the model is exotic, but because agents never leave their queue to use it.
Key takeaways
- Most GenAI stalls are process failures, not model failures — upgrading the model rarely fixes them.
- Build a golden dataset and wire evaluation into CI before scaling beyond a demo.
- Govern the retrieval corpus like a data product: deduplicated, versioned, metadata-rich, fresh.
- Embed answers in the tools people already use; standalone chatbots quietly die of neglect.
- Sequence matters: data readiness → evaluation harness → workflow integration → scale.
The 8-week path we run instead
Our production GenAI engagements follow a fixed sequence: two weeks of corpus audit and golden-dataset construction, two weeks building the retrieval and evaluation harness, two weeks embedding into the target workflow with guardrails and audit logging, and two weeks of supervised rollout with the metrics dashboard live from day one. Eight weeks, one workflow, measured outcomes — then scale to the next workflow with the harness you already trust.
If you have a stalled pilot — or want to skip the stall entirely — tell us what you're trying to ship. We'll give you an honest read on what it would take.