Eighty percent of enterprise applications now ship with an AI agent built in. Only 25% of those ever reach full-scale production[1]. That gap is not a technology problem. It is a design problem. And it starts at the pilot stage.
If you have run an AI proof of concept that everyone agreed was promising and then nothing happened, you are in the majority. The pilot worked in a narrow sense: it demonstrated the thing was possible. What it did not demonstrate was whether it was worth doing at scale, who would own it when it was live, or how it would behave in contact with your actual data and workflows. So it stalled. Someone moved on to another priority. The prototype sat in a shared drive.
This article is about how to run a pilot that does not do that.
Why most AI pilots stall before production
The failure mode is nearly always the same. A firm picks an interesting use case, builds something that works in a controlled demo, presents it to stakeholders, gets a positive response, and then loses momentum before the hard questions are answered.
The hard questions are: whose job changes if this goes live? What happens when the output is wrong? Who signs off before it touches a client? What does it cost to run, not just to build?
Pilots that skip those questions produce interesting demos. They do not produce systems. And in a financial advice or wealth management context, a system that cannot answer those questions does not get deployed, because the compliance and operational risk of deploying it is too high.
Most AI tutorials and vendor demonstrations focus on prompts and models. Architecture, data governance, and human review design are treated as afterthoughts[2]. For a professional services firm in a regulated environment, those are exactly the wrong priorities.
What a well-designed pilot looks like
A pilot worth running has four phases, and the work in phase one determines whether you ever reach phase four[3].
Phase one: problem definition. This is where most pilots go wrong by going too fast. Before you touch any technology, you need a precise answer to three questions. What is the specific task this will handle? What does success look like, in measurable terms? And what is the cost of a wrong output? A pilot that cannot answer question three before it starts is not ready to start.
For a financial advice firm, this might look like: the task is first-draft meeting notes from a transcript. Success means a qualified paraplanner can approve or amend the output in under four minutes, where the current process takes fifteen. A wrong output is one that mis-states a client’s stated risk appetite or omits an action item from the review meeting. That framing is specific enough to design around.
Phase two: scope and architecture. Keep the scope narrow enough to be testable and wide enough to be meaningful. A pilot that runs on three synthetic cases tells you almost nothing. A pilot that runs on thirty real cases from the past six months, with the output reviewed against what was actually produced, tells you something you can act on.
Architecture matters here more than most firms realise. A system that works fine when documents are loaded into memory can fail entirely when you scale it to a document store of any real size[4]. Build the pilot on the infrastructure that production would use, not a simplified version of it. Otherwise you are testing the wrong thing.
Phase three: implementation and testing. This is where human review gates earn their place in the design. Every AI-assisted step that touches a regulated output needs a defined human checkpoint before that output is used. Not because the system will necessarily get it wrong, but because the firm needs to be able to demonstrate that a qualified person reviewed the output. Build that checkpoint into the pilot itself, not as an afterthought when you move to production.
Test against edge cases, not just typical cases. What happens when the transcript is poor quality? What happens when the client has a complex situation the system has not seen before? What does the system do when it is uncertain? If you cannot answer those questions from the pilot data, the pilot has not finished.
Phase four: evaluation. The evaluation step is where pilots either turn into projects or die. Do it formally, even if the firm is small. Compare the pilot output against your success criteria from phase one. Identify the failure modes you found. State what it would cost to run this at scale, in real terms: people time, compute costs, maintenance. Then make an explicit decision: proceed, redesign, or stop.
A pilot that demonstrates the thing is possible is not the same as a pilot that demonstrates the thing is worth doing.
The specific things that kill pilots in regulated environments
In a financial advice or wealth management firm, three failure modes show up repeatedly.
The data problem. AI systems that work well on clean, structured data often struggle with real client data, which is fragmented across systems, inconsistently labelled, and held in formats that were not designed for machine reading. If your pilot uses clean test data and your production environment uses real client data, the gap between them will kill your rollout.
The accountability gap. When an AI system produces an output that a human then acts on, someone needs to own that chain. Who reviews the output? What are they checking for? What do they do when they are not sure? Firms that do not answer these questions before going live tend to find that nobody owns them, and the system quietly stops being used because it creates more uncertainty than it resolves.
The governance gap. AI governance has moved from internal best practice to a regulatory obligation for financial services firms. A pilot that does not include a basic governance framework (who approved deployment, how errors are logged, how the system is monitored once live) is not ready for production in a regulated environment, even if the technology works perfectly.
What to do before you start your next pilot
This is the practical checklist. It is not long, but skipping any item tends to produce a familiar result.
1. Write the success criteria before you write a single prompt. Define what success looks like in terms a sceptic could verify. “Reduces review time” is not a criterion. “A qualified paraplanner can review and approve the output in under five minutes” is.
2. Identify the failure modes in advance. What is the worst thing this system could output? Who would catch it? How? If you cannot answer that before the pilot starts, you are not ready to start.
3. Run the pilot on real data, in the real environment. Synthetic test cases are useful for development. They are not useful for evaluation. If production data has access or privacy constraints, solve those constraints as part of the pilot, not after.
4. Include a human review step in the pilot design. Do not plan to add it later. The review step is part of the system, and it needs to be tested along with everything else.
5. Make the evaluation decision explicit. At the end of the pilot, hold a meeting with a clear agenda item: proceed, redesign, or stop. Do not let the pilot fade into ambiguity. An explicit decision, even a decision to stop, is more useful than a stalled prototype.
The architecture question you should ask early
One detail that experienced practitioners treat as obvious but most firms discover the hard way: the architecture you use for a pilot needs to match the architecture you would use in production.
Scaling AI document retrieval, for example, requires moving from in-memory processing to a vector database as document volume grows[4]. A pilot that uses in-memory processing for thirty documents may work perfectly, and then fail when it meets the ten thousand documents in a real client file store. That is not a failure of the AI. It is a failure of pilot design.
This does not mean every pilot needs to be an enterprise-grade build. It means the pilot needs to be honest about what it is testing. If you are testing whether the AI can do the task, a simple build is fine. If you are testing whether this is ready for production, the build needs to reflect production conditions.
How to tell if your pilot is ready to become a project
The checklist is short. A pilot is ready to become a project when you can answer all of these:
- The system performed within the success criteria you defined in phase one, on real data, including edge cases.
- You have identified the failure modes and designed human review gates around them.
- You know what it costs to run at scale, in pounds and in people time.
- You have a named owner for the system once it is live.
- You have a basic governance record: who approved it, how errors will be logged, how it will be monitored.
If any of those are unanswered, that is the work still to do, not a reason to delay. Do that work, then proceed.
The firms that turn pilots into production systems are not the ones with the best technology. They are the ones that are honest about what the pilot is for and disciplined about what it needs to demonstrate before they move on.
If you are working through a pilot right now and want to think through the design for your specific situation, a discovery call with Cordrey Consulting is a good place to start.
This article is for informational purposes only and does not constitute regulated financial advice or a compliance opinion. Consult a qualified compliance professional for advice specific to your firm.
Sources
[1] VentureBeat (2026) ‘Orchestration is the new challenge for CX in the age of AI agents’. Available at: https://venturebeat.com/orchestration/orchestration-is-the-new-challenge-for-cx-in-the-age-of-ai-agents. [Cited for the 80%/25% deployment gap statistic.]
[2] van Riel, Z. (2026) ‘AI system design patterns 2026’, Zen van Riel, AI Engineer Blog. Available at: https://zenvanriel.com/ai-engineer-blog/ai-system-design-patterns-2026. [Cited for the observation that architecture matters more than prompts and models for production success.]
[3] van Riel, Z. (2026) ‘AI proof of concept template’, Zen van Riel, AI Engineer Blog. Available at: https://zenvanriel.com/ai-engineer-blog/ai-proof-of-concept-template. [Cited for the four-phase proof of concept framework.]
[4] van Riel, Z. (2026) ‘How to scale AI document retrieval from memory to database’, Zen van Riel, AI Engineer Blog. Available at: https://zenvanriel.com/ai-engineer-blog/how-to-scale-ai-document-retrieval-from-memory-to-database. [Cited for the in-memory to vector database scaling requirement.]