VentureBeat reports that 80% of enterprise applications now ship with an AI agent built in, and only 25% of those reach full-scale production[1]. Most of that gap is decided at the pilot stage, by how the pilot was designed.
If you’ve run an AI proof of concept that everyone agreed was promising and then nothing happened, you’re in the majority. The pilot proved the thing was possible. It didn’t show whether it was worth doing at scale, who would own it once live, or how it would cope with your real data and workflows. So it stalled, someone moved on to another priority, and the prototype sat on a shared drive.
Why most AI pilots stall before production
Most pilots stall because the hard questions get left until after the demo. A firm picks an interesting use case, builds something that works in a controlled setting, presents it, gets a good reaction, and loses momentum before anyone answers what matters.
The hard questions are: whose job changes if this goes live? What happens when the output is wrong? Who signs off before it reaches a client? What does it cost to run, not just to build?
A pilot that skips those produces a demo, not a system. In an advice or wealth firm, a system that can’t answer them won’t be deployed, because the compliance and operational risk is too high. Yet most AI tutorials and vendor demos focus on prompts and models, and treat architecture, data governance and human review as afterthoughts[2]. For a regulated firm that’s the wrong way round.
The four phases of a well-designed pilot
A pilot worth running has four phases, and the work in the first decides whether you ever reach the last[3].
Phase 1: define the problem. Most pilots rush this. Before touching any technology, answer three things: what specific task will this handle, what does success look like in measurable terms, and what does a wrong output cost? If you can’t answer the third before starting, you’re not ready to start.
In an advice firm that might read: the task is first-draft meeting notes from a transcript. Success means a qualified paraplanner can approve or amend the draft in under four minutes, against fifteen today. A wrong output is one that misstates the client’s risk appetite or misses an action from the review. That’s specific enough to design around.
Phase 2: set the scope and architecture. Keep the scope narrow enough to test and wide enough to mean something. A pilot on three synthetic cases tells you almost nothing. One on thirty real cases from the last six months, checked against what was actually produced, tells you something you can act on.
Build the pilot on the infrastructure production would use. A system that works when documents are loaded into memory can fail completely against a document store of any real size[4]. Test on a simplified set-up and you’re testing the wrong thing.
Phase 3: build and test. This is where human review checkpoints belong. Every AI-assisted step that touches a regulated output needs a defined human checkpoint before that output is used, partly to catch errors and partly so the firm can show a qualified person reviewed it. Build the checkpoint into the pilot, not into some later production phase.
Test the awkward cases as well as the typical ones. What happens with a poor-quality transcript? With a client situation the system hasn’t seen? When it isn’t sure? If the pilot data can’t answer those, the pilot isn’t finished.
Phase 4: evaluate formally. This is where pilots become projects or end. Do it properly even in a small firm. Compare the output against the phase 1 success criteria, list the failure modes you found, and state the real cost of running it at scale: staff time, compute and maintenance. Then decide explicitly: proceed, redesign or stop.
Showing that something is possible is a long way short of showing it’s worth doing.
Three things that kill pilots in regulated firms
The data problem. Systems that do well on clean, structured data often struggle with real client data, which is spread across systems, inconsistently labelled and stored in formats never meant for machines. If the pilot uses clean test data and production uses real client records, that gap will sink the rollout.
Nobody owns the output. When an AI system produces something a person then acts on, someone has to own that chain: who reviews it, what they check, and what they do when unsure. Firms that leave this open tend to find the system quietly falls out of use, because it creates more uncertainty than it removes.
No governance record. AI governance is now a regulatory obligation for financial services firms, not just good practice. A pilot without a basic framework (who approved it, how errors are logged, how it’s monitored once live) isn’t ready for production in a regulated firm, however well the technology works. Our guide to automation governance for regulated firms covers what that framework needs.
A checklist before your next pilot
1. Write the success criteria before the first prompt. Make them something a sceptic could check. “Reduces review time” isn’t a criterion. “A qualified paraplanner can review and approve the output in under five minutes” is. On my own builds, every requirement gets its test written before the build starts, and a pilot deserves the same discipline.
2. Name the failure modes in advance. What’s the worst thing this could output, and who would catch it, and how? If you can’t say before the pilot starts, it isn’t ready.
3. Use real data in the real environment. Synthetic cases help during development but tell you little at evaluation. If production data has access or privacy constraints, solve them as part of the pilot.
4. Put the human review step in the pilot. It’s part of the system and needs testing with everything else.
5. Make the end decision explicit. Finish with a meeting that has one agenda item: proceed, redesign or stop. A clear decision to stop is more useful than a prototype nobody mentions again.
Match the pilot’s architecture to production early
The architecture used for a pilot needs to match what you’d use in production. Experienced builders treat that as obvious, and most firms find it out the hard way.
Document retrieval is a good example. As volume grows, you have to move from in-memory processing to a vector database[4]. A pilot that holds thirty documents in memory can work perfectly and then fail against the ten thousand in a real client file store. The AI didn’t fail there; the pilot was designed for the wrong conditions.
Not every pilot needs an enterprise-grade build. It needs to be honest about what it’s testing. If the question is whether AI can do the task, a simple build is fine. If the question is whether it’s ready for production, the build has to reflect production.
How to tell a pilot is ready to become a project
A pilot is ready when you can confirm all of these:
- It met the phase 1 success criteria on real data, edge cases included.
- You’ve identified the failure modes and put human review checkpoints around them.
- You know what it costs to run at scale, in pounds and staff time.
- It has a named owner once live.
- There’s a basic governance record: who approved it, how errors are logged, how it’s monitored.
Anything unanswered is the work still to do. Do it, then go ahead. The firms that get pilots into production are the ones that decide up front what the pilot has to prove, and don’t move on until it has.
This article is for informational purposes only and does not constitute regulated financial advice or a compliance opinion. Consult a qualified compliance professional for advice specific to your firm.
Sources
[1] VentureBeat (2026) ‘Orchestration is the new challenge for CX in the age of AI agents’. Available at: https://venturebeat.com/orchestration/orchestration-is-the-new-challenge-for-cx-in-the-age-of-ai-agents. [Cited for the 80%/25% deployment gap statistic.]
[2] van Riel, Z. (2026) ‘AI system design patterns 2026’, Zen van Riel, AI Engineer Blog. Available at: https://zenvanriel.com/ai-engineer-blog/ai-system-design-patterns-2026. [Cited for the observation that architecture matters more than prompts and models for production success.]
[3] van Riel, Z. (2026) ‘AI proof of concept template’, Zen van Riel, AI Engineer Blog. Available at: https://zenvanriel.com/ai-engineer-blog/ai-proof-of-concept-template. [Cited for the four-phase proof of concept framework.]
[4] van Riel, Z. (2026) ‘How to scale AI document retrieval from memory to database’, Zen van Riel, AI Engineer Blog. Available at: https://zenvanriel.com/ai-engineer-blog/how-to-scale-ai-document-retrieval-from-memory-to-database. [Cited for the in-memory to vector database scaling requirement.]