Key Takeaways
- A “council” of five GPT-4 models achieved 97% accuracy on USMLE Step 1 in an October 2025 study, well above the 60% passing threshold that was itself considered a milestone in 2022.
- A June 2025 meta-analysis found individual models such as GPT-o1 reached 95.4% accuracy across medical licensing exams, closing to within 2 points of the multi-agent result. When five instances of OpenAI‘s GPT-4 were configured to deliberate together rather than answer independently, they hit 97% on USMLE Step 1 in October 2025. That is 37 points above the passing threshold that Google’s Med-PaLM first crossed in late 2022and it arrived in under three years.
The Council Approach
The architecture behind the 97% result is worth examining. Five GPT-4 instances were configured to deliberate before producing a final answer rather than each responding in isolation. Multi-agent frameworks have drawn attention in security research for their failure modes, safe individual models can behave unexpectedly when they coordinate but in a closed, exam-style evaluation with well-defined correct answers, deliberation appears to work in the opposite direction, suppressing individual errors rather than compounding them.
Individual Models Close the Gap
A June 2025 meta-analysis found that GPT-o1, running without any multi-agent scaffolding, reached 95.4% accuracy across a range of medical licensing exams. That is 2 points behind the five-model council. The gap between a single advanced model and a coordinated configuration of five is now smaller than the gap between today’s best individual models and where Med-PaLM started.
What that trajectory does not settle is whether higher scores reflect deeper clinical understanding or increasingly sophisticated pattern-matching on training data that happens to include large volumes of clinical vignette material. Benchmark performance and real-world diagnostic reliability are different things, and none of the studies reviewed here tested the latter.
Beyond Fact Recall
GPT-4o has also demonstrated strong performance on clinical vignette questions, identifying conditions and suggesting plausible treatment paths. Whether this constitutes genuine clinical reasoning remains a live question that exam scores alone cannot answer.
A November 2025 study explored using GPT-4o to generate, score and return feedback on Script Concordance Tests, which are designed to evaluate clinical reasoning under uncertainty rather than factual recall. Tools including iatroX, BMJ OnExamination’s AI-PACES and Geeky Medics’ AI virtual patients are already offering OSCE-style simulation with rubric-linked feedback, some using spaced repetition and knowledge-tracing to personalise preparation. Evidence of their effect on actual exam outcomes, as distinct from simulated ones, remains limited.
What Medical Educators Face Now
The performance gap between individual models and collaborative configurations carries a practical signal for anyone building on these systems: architecture choices, not just model selection, drive results. GPT-4’s documented performance on professional licensing-style assessments now spans more than one high-stakes domain, and the multi-agent approach suggests that how a model is configured may matter as much as which model is chosen.
Medical educators face a harder version of that question. If a model scores 95% on the same exam used to certify physicians, what does the exam actually measure, and what should it measure instead? None of the studies reviewed here answer that, but the data makes it harder to defer asking.
Originally published at https://autonainews.com/ai-excels-on-medical-exams-collaborative-gpt-4-hits-97-gpt-o1-954/
Top comments (0)