Benchmark tables usually measure exam questions. Firms buy AI for drafting, so we tested drafting: five tasks of the kind a UK service firm runs daily, one attempt per model, prompts published in full, misses reported as gladly as wins. It is a small test, honestly scored, and the small size is the point: this is the method any firm can copy with its own tasks in an afternoon.
The contenders and the method
Claude Sonnet (Anthropic, via API), Gemini Pro (Google, via API), and two open-weight models served through Groq: gpt-oss-120b (OpenAI’s open-weight release) and Qwen 27B (Alibaba). One attempt per task, no retries, same day in September 2026. The five tasks: rewrite a blunt late-report apology email to a client; compress a regulatory paragraph into exactly three plain-English bullets for a client letter; extract structured JSON from messy meeting notes; explain pound cost averaging to a nervous first-timer in about 120 words with one everyday analogy; and write a cautious emergency-fund recommendation paragraph using only supplied facts.
Results, task by task
| Task | Best output | The detail that decided it |
|---|---|---|
| Client apology email | Gemini | The only model that honoured “late again” from the source, apologising for a continued delay. Claude a close second on tightness. |
| Three-bullet summary | Claude | Right audience voice for a client letter, all bullets inside the word cap, no accuracy drift. Qwen quietly narrowed “fair value” to “overpriced”. |
| JSON extraction | Gemini / Qwen | All four extracted correctly and normalised the date; Claude alone wrapped the JSON in code fences despite “output only valid JSON”. |
| Plain-English explainer | Gemini | Petrol-pump analogy that genuinely fits the mechanism, plus an unprompted risk caveat. gpt-oss chose a newspaper analogy that breaks the very point being explained. |
| Facts-only recommendation | Claude | The strictest reading of “do not invent figures”: no imported conventions, no derived targets. gpt-oss produced the most practically useful paragraph but imported a three-to-six-month rule the prompt never gave. |
What it adds up to
Three findings survive the small sample:
- Fluency is solved everywhere. No model produced an unusable draft, including a 27-billion-parameter open-weight you can run on your own hardware, which supports the open-weight guide’s claim that the gap is smallest exactly where business volume lives.
- The differences are character, not quality. Claude read constraints most conservatively, which is what regulated drafting wants, and still logged the day’s clearest miss with those code fences; Gemini showed the best ear for audience and the small human details; the open-weights padded slightly and imported outside conventions under pressure.
- Every model missed something. The durable lesson: the human-approves discipline is not pessimism about any vendor, it is what one-attempt reality requires from all of them.
These findings match the picks in the best-AI-by-task table, which this test now part-evidences, and they will be re-run at each quarterly review of this page.
The prompts, verbatim
Published so the test is checkable and copyable:
- Email. “Rewrite this email to a client so it is professional, warm and honest. Keep it under 120 words, UK English, no exclamation marks. Do not invent facts”, over a deliberately sloppy original.
- Bullets. “Summarise the following for a client letter as exactly three plain-English bullet points, each under 25 words”, over a fair-value regulation paragraph.
- Extraction. “Extract to JSON with keys client_name, meeting_date, amounts, next_action. Output only valid JSON”, over messy meeting notes.
- Explainer. “Explain pound cost averaging to a nervous first-time investor in about 120 words, plain language, no promises about returns, one everyday analogy.”
- Recommendation. A facts list with the instruction “Do not invent figures.”
The full task file and raw outputs are retained and available on request.
Where I fit in
This test is a miniature of how I choose models inside client systems: by task, against written constraints, with the misses recorded, and re-checked as the field moves. If you want the same five-task discipline run on your firm’s actual work, with your documents and your red lines, that is an afternoon inside an Automation Audit, and it ends the which-AI debate with evidence instead of opinions.