What 1,500 graded AI runs taught us about using ChatGPT, Claude and Gemini at work

AI Professions — Wed, 28 Oct 2026 17:15:55 +0000

At AI Professions we test AI models on real tasks from ten professions: accountants, lawyers, real estate agents, healthcare staff, engineers, sales managers, travel agents, event planners, construction managers and interior designers. Our open benchmark now holds 1,500 graded runs: 100 prompts, 5 models, each prompt run 3 times. Every prompt has 5 pass/fail criteria, and we locked them with a hash before any model saw the task.

Here are the five lessons that matter most if you use AI for client work.

1. The most common failure is added content, not wrong content

In round 1, 78 of 110 failed verdicts (71%) came from the model adding something that was not in the input. Only 24 failures were missing or changed facts, and 8 were length problems.

Some examples from the test set:

Patient handout. The notes covered type 2 diabetes, metformin, side effects and follow-up, with no emergency instructions. The model added: "If you feel very sick or think it is an emergency, call 911." The advice sounds harmless, but it was not in the clinician's notes, and the criterion failed in all 6 Claude runs.
Flight disruption. The notes said "same airline, next day" and did not mention a connection. The model wrote about a "comfortable connection". That kind of unsupported detail failed in 8 of 9 runs.
Sales cold email. The input allowed one approved proof point. The model invented an extra benefit claim in 3 of 3 runs.

What to do: when you review an AI draft, don't only check whether the facts are right. Look for sentences you did not give it.

2. A raw score is not a ranking

GPT-6 Astra scored 99% raw in round 1, against 92% for Claude Opus 5.5 and 91% for Claude Sonnet 5. But GPT was graded by a Claude judge, and the Claude models were graded by GPT. When we swapped judges on the same 30 Claude runs, the GPT judge passed 93% of criteria and the Claude judge passed 98%. Most of the raw gap was judge strictness, not model quality.

What to do: be careful with any "Model X wins" headline that doesn't name the judge.

3. Quality drops outside English

The same models lost points in other languages. Under one judge, Claude Opus scored 92% on the English/Spanish set and 79% on the French/Portuguese set. Sonnet fell from 91% to 70%. Gemini 4 Argon held up best, going from 91% to 80%.

What to do: if your clients read French, Portuguese or Spanish, test your prompts in that language. Don't assume the English result carries over.

4. Spreadsheet answers need checking against your requirements

On Excel and Microsoft 365 tasks, scores ranged from 83% to 93%. Older lookup functions like VLOOKUP caused 20 failed verdicts. The formulas could work, but they didn't meet the stated requirement for XLOOKUP or INDEX/MATCH.

What to do: say exactly which functions and output format you need, and check that the answer follows it.

5. Use AI for drafting, but a professional still signs off

Our tests measured whether the outputs met written criteria. They did not measure whether a professional should sign the result. Use AI for first drafts, summaries and checklists. Keep confidential client data out of tools whose data terms you haven't checked.

The data is open

All prompts, rubrics, transcripts and verdicts are free to reuse under CC BY 4.0:

Full results and methodology: AI Model Test Results
How we test: Methodology
Dataset: GitHub · Zenodo DOI 10.5281/zenodo.23577235
Tested prompts and guides for your profession: aiprofessions.ai

AI Professions is an independent portal that tests AI tools on real professional tasks in English, Spanish, French and Portuguese.

Disclosure: this post was drafted with AI assistance and checked by the AI Professions team against the published benchmark data.