Every AI product demo is a performance, rehearsed on data chosen to behave. That’s not an accusation, it’s how demos work, and your job as the buyer is to see what happens off the script. This guide covers the two requests that do that fastest, then the questions to ask and what good and evasive answers sound like.
The two requests
Ask for these two things in every demo, before anything else.
Run it on our data, now. Bring a realistic, non-sensitive sample and ask to see it processed live. Listen for the hedge: setup required, needs configuration, our engineers would tune that. Some setup is legitimate. If it can’t touch unfamiliar data at all, you’re watching a video with a salesperson attached.
Show us the log. Ask what the system did for a real customer last week: the actions, the errors, the times a person stepped in. Real products have logs, and vendors who are proud of theirs will open them, suitably anonymised. No logs is a finding that outranks everything else on the screen, and for agentic products it rules them out, as the governance guide explains.
The questions, and what the answers sound like
Each question has a good answer and an evasive one, and the difference is usually easy to hear:
- What happens when it’s wrong? Good: a specific error path, a review step, and a real story. Evasive: accuracy percentages with no denominator, or “it learns”.
- What exactly is the AI here? Plenty of solid products are ordinary software with a model at one step, and that’s fine. The AI-washing guide covers the ones where the label is the product.
- Which model is underneath, and what happens when it changes? You’re listening for a plan, not a brand name.
- What does the bill look like at our volume? Have your numbers ready and ask them to work it out live. Metered products tend to get expensive precisely when they work.
- Where does our data go? The contract-reading guide has the seven clauses this answer needs to stand up to.
- What do we take with us if we leave? Data, configurations and logs. Watch for the pause before the answer.
After the demo
Three things to do once the call ends:
- Write down the promises while they’re fresh, because the contract won’t contain them.
- Set up any trial with numbers and an end date, borrowing the scorecard from the agent job description template.
- Score the meeting itself with the due-diligence sheet. Vague answers are findings, not gaps.
If you haven’t yet listed the processes you’re shopping for, the process inventory worksheet is the place to start, ideally before the next demo goes in the diary.