Every AI product demo is a performance, rehearsed on data chosen to behave. That is not an accusation, it is stagecraft, and the buyer’s job is to get backstage. This is the field guide: the two requests that end most theatre, then the questions and the sounds of good and evasive answers.
The two requests
Run it on our data, now. Bring a realistic, non-sensitive sample and ask to see it processed live. Watch for the hedge: setup required, needs configuration, our engineers would tune that. Some setup is legitimate; total inability to touch unfamiliar data means you are watching a video with a salary.
Show us the log. What did the system do for a real customer last week: the actions, the errors, the human interventions. Real products have logs and proud vendors open them, suitably anonymised. No logs is a finding that outranks everything else on the screen, and for agentic products it is disqualifying, per the governance guide.
The questions, and the sounds of answers
- What happens when it is wrong? Good: a specific error path, review step, and a story. Evasive: accuracy percentages without denominators, or “it learns”.
- What exactly is the AI here? Plenty of solid products are ordinary software with a model at one step, which is fine; the AI-washing guide covers the ones where the label is the product.
- Which model underneath, and what happens when it changes? You are listening for a plan, not a brand.
- What does the bill look like at our volume? Have your numbers ready and make them model it live; metered products fail by succeeding.
- Where does our data go? The contract-reading guide holds the seven clauses this answer must survive.
- What do we take with us if we leave? Data, configurations, logs; watch the pause before the answer.
After the demo
- Write down the promises while they are fresh, because the contract will not contain them.
- Structure any trial with numbers and an end date, borrowing the scorecard from the agent job description template.
- Score the meeting itself with the due-diligence sheet: vague answers are findings, not gaps.
Where I fit in
I sit on the buyer’s side of these meetings for clients: asking the questions above without needing the seller’s goodwill, translating the answers, and structuring trials that produce evidence. It is a small, cheap engagement that routinely pays for itself in one avoided subscription; the Automation Audit includes it wherever tool decisions are live.