How Do You Measure Clinical-Grade AI?

Every healthcare AI vendor says its product is “clinical grade.” Almost nobody says what that means, or how it is measured. In this Healthcare AI Update, with no guest and just the two besties, Dr. Junaid Kalia and DR GPT™ Harvey Castro take the question apart from both ends: the questions a health system should ask, and the numbers a builder should be able to show.

“People, for some reason, demand a lot higher from AI as compared to what we have been doing.” Dr. Junaid Kalia

Harvey starts with a framework he now uses every month: six questions adapted from WHO and NIST. Trust and safety: is this helping me book an appointment, or is it touching neurosurgery? Governance and accountability: who is responsible? In Texas, for now, it is the physicians, because they hold the license. Culture and workflow, which he calls the rate-limiting step: having AI does not mean people use it, and clinicians may be using tools their institution never approved. Privacy and security: if I build a digital twin on my phone, who sees it? Equity and access: who can use it and who can afford it? And finally, outcomes and value: what actually improves?

Then he applies the last question to the evidence. In a cluster-randomized primary-care study in Kenya, AI made physicians’ documentation more efficient but showed no benefit for the patient. In another trial, the headline odds ratio looked good until you noticed its range included “no difference,” and in some cases worse. His warning to anyone reading a study from ten thousand feet: a better score does not automatically mean better health. He leaves hospital CEOs with three questions: what outcome was actually measured, who responds and when, and what would justify continuing the AI tool or stopping it. Then he adds a forecast. With 60% confidence, by this time next year, an AI used in a hospital system will show lower morbidity and mortality than the standard of care.

Junaid pushes on the legal side. If an AI flags a breast mass and the radiologist disagrees, who is medically and legally liable? And at what point does the question flip to “why didn’t you use AI?” Harvey’s answer is the episode’s most quotable scenario. Picture two hospitals across the street from each other, one using AI and one not. If the AI hospital consistently gets better outcomes, AI becomes the regional standard of care. The day a patient has a bad outcome at the other hospital, a lawsuit follows, and the culture changes overnight.

“Just because we get a special number, does that mean that technically gives me better health?” DR GPT™ Harvey Castro

The episode closes on a live audience question from MaryAnne Rizk, Ph.D.: how do you measure responsible AI for clinical-grade use? Harvey hands it to Junaid as the builder on the front lines, and Junaid answers in three steps. First, reset the expectations. Even clinical trials are not 100%: drugs pass trials with thousands of patients and are still pulled from the market after post-market surveillance. Second, there is no single “clinical-grade” number. Every model gets the KPI that fits its job: Dice for a vision model segmenting a liver or a liver mass on CT, F1 for a medical-coding model, and job and task completion rates for a voice agent making prior-authorization calls. Third, keep a golden dataset, physician- and multi-clinician-verified ground truth. Every time a new model or version arrives, it is re-scored against that dataset before it ships.

Harvey’s close ties it together: it is all about human plus AI. Your clinical gestalt, your secret sauce, plus the tool, measured honestly.

Connect with us

Dr. Junaid Kalia, Neurocritical Care Specialist, Founder of Savelife.AI™

DR GPT™ Harvey Castro, ER Physician, #DrGPT

Edward Marx, CEO, Advisor

Originally published on my Substack. Planning a session on evaluating clinical AI? Explore my healthcare AI keynotes.