Clinical AI Is Not a Technology Problem. It Is Trust, Workflow, and Judgment

By Harvey Castro, MD, MBA

A hospital can buy an excellent model and still get a bad outcome. I have watched it happen more than once, and the postmortem almost never lands on the algorithm.

It lands on one of three things. The clinicians did not trust the output. The output arrived in the wrong place at the wrong moment. Or the clinicians trusted it more than they should have, and stopped thinking.

Those are three different failures with three different fixes. Health systems that treat them as one problem, usually filed under "change management," end up solving none of them.

Failure one: trust

Trust is not a feeling. In medicine it is a track record.

The Epic Sepsis Model gives the cleanest example. Deployed at hundreds of US hospitals before independent evaluation, it showed a hospitalization-level area under the curve of 0.63 when Michigan Medicine studied it across 38,455 hospitalizations. It missed 67 percent of sepsis cases and would have flagged 18 percent of hospitalizations at the evaluated threshold. The analysis was retrospective, used a score threshold of at least 6 for these misses and hypothetical flags, and excluded hospitalizations eligible for actual alerts. According to PubMed, that is Wong A, Otles E, Donnelly JP, et al., JAMA Internal Medicine, 2021 (DOI).

The fix is upstream. Validate locally before go-live, with clinicians involved in choosing the metric. Publish the local result to the people who will use it, including the parts that look bad. Then commit to a monitoring cadence and tell the staff what will trigger a change.

Trust survives a tool that is imperfect and honest. It does not survive a tool that is oversold.

Failure two: workflow

The second failure has nothing to do with accuracy. A perfectly calibrated model delivered into the wrong moment of a clinical day is still a bad product.

Ask where the output lands. An overnight risk score that appears in a chart nobody opens until morning rounds has zero clinical value regardless of its performance. A prompt that requires a nurse to leave the medication administration screen, read a paragraph, and click twice will get dismissed, because the nurse is holding a syringe.

In a voluntary, uncontrolled 30-day pre/post quality-improvement study of 263 ambulatory clinicians across six health systems, self-reported after-hours documentation decreased by a mean of 0.90 hours per week. Adjusted burnout estimates decreased from 51.9 percent to 38.8 percent in the smaller burnout-analysis sample. These findings describe an association and do not establish causation (Olson et al., JAMA Network Open, 2025). A separate three-month Stanford quality-improvement pilot enrolled 48 physicians; paired survey analyses of 38 respondents found reductions in task load and burnout (Shah et al., JAMIA, 2025).

The fix is observation, not survey. Before deployment, watch three shifts. Count the clicks. Find the moment where the information would actually change a decision, and put it there. If no such moment exists, do not deploy the tool.

One more workflow metric belongs on every dashboard: alerts per 100 admissions. When that number climbs, clinical attention is being spent, and attention is the scarcest resource in the building.

Failure three: judgment

The third failure is the one that scares me, because it looks like success.

Automation bias is well described in medicine. A confident output on a screen shifts a clinician's threshold, particularly when the clinician is tired, junior, or covering an unfamiliar service. The model says low risk, the patient goes home, and the chart shows a clinician who agreed with a computer at 4 a.m.

Two design choices make this worse. Presenting a score without any indication of uncertainty. And building a workflow where agreeing takes one click and disagreeing takes four.

The fix is friction placed correctly. Show the drivers behind the output, not only the number. Show when the model is operating outside its validated range. And make the override path shorter than the agreement path for any decision with real downside.

Then measure it. Override rate is a vital sign. Very low override rates warrant review; they do not alone prove automation bias.

What all three fixes look like together

The clearest example I can point to is breast cancer screening in Sweden.

The MASAI trial randomized 105,934 women at four screening sites to AI-supported screening or standard double reading. AI-supported screening detected 6.4 cancers per 1,000 participants against 5.0 per 1,000 in the control group, a 29 percent increase, with more small node-negative invasive cancers found. There was no statistically significant increase in the false-positive rate. Screen-reading workload fell 44.2 percent. According to PubMed, that is Hernström V, Josefsson V, Sartor H, et al., Lancet Digital Health, 2025 (DOI).

Look at how that design handles all three failures at once. Trust is earned through a randomized trial inside a real national screening program rather than a vendor claim. Workflow is respected, because the AI triages examinations to single or double reading instead of adding a step to an already full day. And judgment stays with the radiologist, who reads the case and decides, with the model highlighting findings rather than issuing a verdict.

Notice also what the trial measured. Not accuracy alone. Detection, false positives, cancer type and stage, and reading workload. That is what an honest evaluation looks like, and it is the shape of evaluation most hospitals never demand from a vendor.

What separates the systems that get this right

The health systems I would hold up as examples share three habits.

They pilot narrowly and measure honestly, with a stopping rule written before go-live. They put a practicing clinician in the design conversation from the first meeting rather than the last. And they have retired at least one tool, publicly, which tells everyone in the organization that evaluation is real.

That third habit is rare. Most AI portfolios only grow.

The scoreboard I would use

Four numbers, reviewed quarterly, for every clinical AI tool in the building.

Local performance on current patients, not the vendor's original cohort. Alerts per 100 admissions. Override rate and its direction over time. And performance stratified by language, payer, and site, because a model that works on one population and fails on another is a quality problem before it becomes a legal one.

If a tool cannot produce those four numbers, the organization does not know what it owns. In a late-2025 Black Book survey of 182 hospital leaders, only 22 percent were highly confident they could produce a complete AI audit trail within 30 days (Black Book Research release).

Back to the bedside

The best clinical AI I have used did not feel impressive. It felt like the note was already written when I sat down, so I got to look at the patient instead of the screen.

That is the standard. Not accuracy on a slide. Whether the tool gives a clinician back the one thing the job actually runs on, which is attention.

Harvey Castro, MD, MBA (DR GPT™) is a board-certified emergency physician, 5x TEDx speaker, Advisor to the CEO of Phantom Space, and author of books on AI, medicine and leadership.

Related DR GPT™ reading

• AI Governance in Healthcare: A Physician's Framework for Boards and Executives
• The Physician's Role in Responsible AI Adoption

More from DR GPT™: who DR GPT™ is, healthcare AI keynotes, board advisory, speaking, TEDx talks, books, and the media kit.

Book Harvey Castro, MD, MBA, known as DR GPT™, for a healthcare AI keynote, board retreat, or executive session. Start the conversation.