ChatGPT masters USMLE: a real win for AI, but a long way from the art of medicine

DR GPT™ perspective

A benchmark result that felt enormous at the time and is unremarkable now. That is the point I keep making about benchmarks. A model that scores well on an exam can still miss a patient it has never seen.

Why exam scores do not equal clinical skill

Passing a licensing exam shows a model can recognize patterns in well-written questions. Real patients rarely arrive as well-written questions. They arrive scared, with half the history, three medicines they cannot name and a symptom they describe in their own words.

What an exam does not test:

• Uncertainty. Knowing when you do not have enough information, and what to do next.
• Examination. What the abdomen feels like, how the patient looks across the room.
• Values. Fitting a plan to what this patient wants and can afford.
• Accountability. Standing behind a decision and following up.

Benchmarks are useful for tracking progress. They are a poor guide to whether a tool is ready for a specific clinical job.

Questions people ask

Did ChatGPT pass the medical licensing exam?

Studies in 2023 reported that ChatGPT performed at or near the passing threshold on USMLE-style questions. That is a benchmark result, not proof of clinical competence.

Can AI replace doctors?

I do not think so. AI can take on tasks, but medicine is judgment, relationship and accountability under uncertainty.

Originally published on KevinMD. Read the full essay, or see how I build these arguments for a room as a speaker.