Benchmark tables usually measure exam questions. Firms buy AI for drafting, so we tested drafting: five tasks of the kind a UK service firm runs daily, one attempt per model, prompts published in full, misses reported alongside wins. It is a small test, and that is deliberate: any firm can copy the method with its own tasks in an afternoon.
The models tested and the method
We tested four models on five tasks, one attempt each. Claude Sonnet (Anthropic, via API), Gemini Pro (Google, via API), and two open-weight models served through Groq: gpt-oss-120b (OpenAI’s open-weight release) and Qwen 27B (Alibaba). One attempt per task, no retries, same day in September 2026. The five tasks: rewrite a blunt late-report apology email to a client; compress a regulatory paragraph into exactly three plain-English bullets for a client letter; extract structured JSON from messy meeting notes; explain pound cost averaging to a nervous first-timer in about 120 words with one everyday analogy; and write a cautious emergency-fund recommendation paragraph using only supplied facts.
Results, task by task
Gemini won three tasks outright or jointly, Claude won two, and no model swept the board.
| Task | Best output | The detail that decided it |
|---|---|---|
| Client apology email | Gemini | The only model that honoured “late again” from the source, apologising for a continued delay. Claude a close second on tightness. |
| Three-bullet summary | Claude | Right audience voice for a client letter, all bullets inside the word cap, no accuracy drift. Qwen quietly narrowed “fair value” to “overpriced”. |
| JSON extraction | Gemini / Qwen | All four extracted correctly and normalised the date; Claude alone wrapped the JSON in code fences despite “output only valid JSON”. |
| Plain-English explainer | Gemini | Petrol-pump analogy that fits the mechanism, plus an unprompted risk caveat. gpt-oss chose a newspaper analogy that breaks the very point being explained. |
| Facts-only recommendation | Claude | The strictest reading of “do not invent figures”: no imported conventions, no derived targets. gpt-oss produced the most practically useful paragraph but imported a three-to-six-month rule the prompt never gave. |
What the results mean for choosing an AI model
Three findings hold up despite the small sample:
- Fluency is solved everywhere. No model produced an unusable draft, including a 27-billion-parameter open-weight you can run on your own hardware, which supports the open-weight guide’s claim that the gap is smallest exactly where business volume lives.
- The differences are character, not quality. Claude read constraints most conservatively, which is what regulated drafting wants, and still logged the day’s clearest miss with those code fences; Gemini showed the best ear for audience and the small human details; the open-weights padded slightly and imported outside conventions under pressure.
- Every model missed something. That is why the human-approves discipline applies to all of them. It reflects how one-attempt use works in a real firm, not doubt about any one vendor.
These findings match the picks in the best-AI-by-task table, which this test now part-evidences, and they will be re-run at each quarterly review of this page.
The prompts, verbatim
These are the five prompts exactly as sent, so you can check the test or copy it:
- Email. “Rewrite this email to a client so it is professional, warm and honest. Keep it under 120 words, UK English, no exclamation marks. Do not invent facts”, over a deliberately sloppy original.
- Bullets. “Summarise the following for a client letter as exactly three plain-English bullet points, each under 25 words”, over a fair-value regulation paragraph.
- Extraction. “Extract to JSON with keys client_name, meeting_date, amounts, next_action. Output only valid JSON”, over messy meeting notes.
- Explainer. “Explain pound cost averaging to a nervous first-time investor in about 120 words, plain language, no promises about returns, one everyday analogy.”
- Recommendation. A facts list with the instruction “Do not invent figures.”
The full task file and raw outputs are retained and available on request.