The Physician's Role in Responsible AI Adoption

By Harvey Castro, MD, MBA

The most expensive AI mistakes I have seen in healthcare were not technical. They were organizational. Somebody bought a tool, somebody configured it, somebody announced it at a town hall, and no practicing clinician was in the room for any of those three moments.

Then the tool met the fifth floor at 3 a.m.

Physicians tend to enter this conversation late and in the wrong posture. Either as skeptics who want the thing gone, or as enthusiasts who want a demo. Both roles are weak. The strong role is different, and it is one no committee can outsource.

Why the clinician has to be there

Clinical AI fails in ways that only show up at the bedside.

Consider the Epic Sepsis Model, which reached hundreds of US hospitals before independent evaluation. Michigan Medicine researchers studied it across 38,455 hospitalizations and found a hospitalization-level area under the curve of 0.63, with 67 percent of sepsis cases missed and 18 percent of hospitalizations hypothetically flagged. The analysis was retrospective, used a score threshold of at least 6 for these misses and hypothetical flags, and excluded hospitalizations eligible for actual alerts. According to PubMed, that is Wong A, Otles E, Donnelly JP, et al., JAMA Internal Medicine, 2021 (DOI).

That gap between the metric and the experience is the physician's territory. Nobody else can close it.

Five jobs only a practicing clinician can do

One: define the problem in clinical language

Most AI projects start with a capability looking for a use. Someone has a model, so the organization goes hunting for a place to put it.

Reverse that. A physician can state the problem precisely: patients with abdominal pain wait too long for a CT read overnight, or discharge summaries take 20 minutes each and are frequently late. A problem stated that way tells you what to buy, what to measure, and when to stop.

Two: pressure-test the validation

You do not need a statistics degree to ask the right question. You need to know your own patients.

Ask which population the model was trained on, then say out loud how your population differs. My patients in a Dallas emergency department differ from an academic center's cohort in payer mix, language, comorbidity burden, and how late in an illness they present. A physician can name those differences in thirty seconds. A vendor cannot.

Three: design the workflow, not just approve the tool

Where does the output appear. Who sees it first. What is the clinician expected to do within how many minutes. What happens when it is wrong.

In a voluntary, uncontrolled 30-day pre/post quality-improvement study of 263 ambulatory clinicians across six health systems, self-reported after-hours documentation decreased by a mean of 0.90 hours per week. Adjusted burnout estimates decreased from 51.9 percent to 38.8 percent in the smaller burnout-analysis sample. These findings describe an association and do not establish causation (Olson et al., JAMA Network Open, 2025).

Compare that with a risk score dropped into an unchanged workflow. Same category of technology. Opposite result. The difference is design, and design is a clinical act.

Four: protect the right to disagree

Human oversight only counts if disagreeing is cheap.

I ask three questions when I audit this. Can a clinician override without filing an incident report. Does overriding cost her time she does not have. Does any dashboard, quality metric, or compensation formula punish her for it.

If the answer to any of those goes the wrong way, the organization has automation with a witness, not oversight.

Five: be willing to say no in public

This is the job physicians avoid, and it is the one that matters most.

Somebody has to be able to stand in a leadership meeting and say the tool is not working, here is the evidence, and it should come out. That takes standing, and it takes a culture where retiring a tool is treated as a normal outcome instead of a failed initiative.

Recommend pausing or retiring a tool when the evidence warrants it, and explain the decision to the people who use and oversee it.

What physicians should stop doing

Stop treating governance as somebody else's committee. In a Black Book Research survey of 182 hospital leaders in late 2025, 33 percent said unclear ownership across IT, quality, safety, and compliance slowed governance progress (Black Book Research release). Ambiguity like that gets resolved by whoever shows up.

Stop asking for a demo. Ask for the validation data and the alert volume per 100 admissions.

Stop framing the choice as adoption versus resistance. The useful question is which specific tool, for which patients, with what monitoring, owned by whom.

A 30-day plan for the physician who just got handed this

Many of you will be volunteered for an AI committee without warning and without protected time. Here is what I would do in the first month.

Week one, build the inventory. Ask a simple question in writing: what AI is running in this organization today, and who approved each one. If the answer is incomplete, document what is missing. In the Black Book survey, 29 percent reported enforced policies covering AI model inventory, lineage, and sign-offs; that finding does not establish how many organizations could produce a complete inventory.

Week two, pick one tool and go see it. Not a demo. Go to the unit, stand next to the person using it, and count. How many alerts per shift. How many acted on. How many silenced without reading. Bring numbers back, not impressions.

Week three, read one contract. Ask legal for the agreement behind that tool and look for four things: local validation, performance monitoring obligations, notification when the vendor updates the model, and data portability on exit. Note which are missing.

Week four, propose one decision. Something concrete and small. Retune a threshold, add a monitoring metric, or sunset a tool nobody uses. Get it approved and executed.

At the end of that month you will know more about your organization's AI risk than most of the people who bought it. That is not an exaggeration, and it is not a compliment to the industry.

The regulatory floor is rising, and it names clinicians

On June 1, 2026, The Joint Commission launched a voluntary Responsible Use of AI in Healthcare certification across five domains including governance, monitoring and validation, and transparency with education and training (Joint Commission announcement).

These laws cover different decisions and actors.

Utah's SB 319, effective January 1, 2027, addresses adverse preauthorization determinations involving clinical or medical necessity. The decision-maker must know the condition or consult a knowledgeable specialist, exercise independent medical judgment subject to the specified specialist-consultation exception, and not rely solely on another source's recommendations. Separate provisions govern licensed review of specified appeals.

Georgia's SB 444, effective January 1, 2027, requires a qualifying natural person to conduct utilization review with a clinical peer participating before AI or software issues an adverse determination; the software cannot supersede that peer's judgment.

Indiana's HB 1271 added a medical-necessity downcoding provision effective July 1, 2026: an insurer may not use automation, including AI, as the sole basis without an employee or contractor reviewing the covered person's medical record. That clause does not specify a licensed clinician, and the chapter excludes Medicaid and Medicaid managed-care organizations.

Delaware's HB 191, effective April 23, 2026, bars nonhuman entities from specified nursing, physician, and physician-assistant licenses or certification and from using the corresponding protected titles.

I would rather define that role than have it defined for me.

Where I have landed

I do not think physicians should be the brake on healthcare AI. I have written extensively about healthcare AI because I think the upside is real and the workforce math leaves no alternative.

But the tools get safer when a clinician sits at the table from the first meeting, names the problem, checks the validation, designs the workflow, defends the override, and keeps the authority to pull the plug.

That is not resistance. That is the job.

Harvey Castro, MD, MBA (DR GPT™) is a board-certified emergency physician, 5x TEDx speaker, Advisor to the CEO of Phantom Space, and author of books on AI, medicine and leadership.

Related DR GPT™ reading

• AI Governance in Healthcare: A Physician's Framework for Boards and Executives
• Clinical AI Is Not a Technology Problem. It Is Trust, Workflow, and Judgment

More from DR GPT™: who DR GPT™ is, healthcare AI keynotes, board advisory, speaking, TEDx talks, books, and the media kit.

Book Harvey Castro, MD, MBA, known as DR GPT™, for a healthcare AI keynote, board retreat, or executive session. Start the conversation.