The Physician's Role in Responsible AI Adoption

The most expensive AI mistakes I have seen in healthcare were not technical. They were organizational. Somebody bought a tool, somebody configured it, somebody announced it at a town hall, and no practicing clinician was in the room for any of those three moments.

Then the tool met the fifth floor at 3 a.m.

Physicians tend to enter this conversation late and in the wrong posture. Either as skeptics who want the thing gone, or as enthusiasts who want a demo. Both roles are weak. The strong role is different, and it is one no committee can outsource.

Why the clinician has to be there

Clinical AI fails in ways that only show up at the bedside.

Consider the Epic Sepsis Model, which reached hundreds of US hospitals before independent evaluation. Michigan Medicine researchers studied it across 38,455 hospitalizations and found an area under the curve of 0.63, with 67 percent of sepsis cases missed and alerts firing on 18 percent of all hospitalized patients. According to PubMed, that is Wong A, Otles E, Donnelly JP, et al., JAMA Internal Medicine, 2021 (DOI).

A data scientist reads that as a calibration problem. An executive reads it as a vendor problem. A nurse who worked those shifts read it years earlier, in her body, as a pager that stopped meaning anything.

That gap between the metric and the experience is the physician's territory. Nobody else can close it.

Five jobs only a practicing clinician can do

One: define the problem in clinical language

Most AI projects start with a capability looking for a use. Someone has a model, so the organization goes hunting for a place to put it.

Reverse that. A physician can state the problem precisely: patients with abdominal pain wait too long for a CT read overnight, or discharge summaries take 20 minutes each and are frequently late. A problem stated that way tells you what to buy, what to measure, and when to stop.

Two: pressure-test the validation

You do not need a statistics degree to ask the right question. You need to know your own patients.

Ask which population the model was trained on, then say out loud how your population differs. My patients in a Dallas emergency department differ from an academic center's cohort in payer mix, language, comorbidity burden, and how late in an illness they present. A physician can name those differences in thirty seconds. A vendor cannot.

Three: design the workflow, not just approve the tool

Where does the output appear. Who sees it first. What is the clinician expected to do within how many minutes. What happens when it is wrong.

Ambient documentation works in most systems because it fits a workflow clinicians already had. In a multicenter study of 263 ambulatory clinicians across six health systems, burnout fell from 51.9 percent to 38.8 percent after 30 days, with an average of 0.90 fewer hours per day spent documenting after hours. According to PubMed, that is Olson KD, Meeker D, Troup M, et al., JAMA Network Open, 2025 (DOI).

Compare that with a risk score dropped into an unchanged workflow. Same category of technology. Opposite result. The difference is design, and design is a clinical act.

Four: protect the right to disagree

Human oversight only counts if disagreeing is cheap.

I ask three questions when I audit this. Can a clinician override without filing an incident report. Does overriding cost her time she does not have. Does any dashboard, quality metric, or compensation formula punish her for it.

If the answer to any of those goes the wrong way, the organization has automation with a witness, not oversight.

Five: be willing to say no in public

This is the job physicians avoid, and it is the one that matters most.

Somebody has to be able to stand in a leadership meeting and say the tool is not working, here is the evidence, and it should come out. That takes standing, and it takes a culture where retiring a tool is treated as a normal outcome instead of a failed initiative.

Retire one thing publicly in year one. The whole organization recalibrates.

What physicians should stop doing

Stop treating governance as somebody else's committee. In a Black Book Research survey of 182 hospital leaders in late 2025, 33 percent named unclear ownership across IT, quality, safety, and compliance as a primary obstacle to AI audit readiness (Becker's Hospital Review). Ambiguity like that gets resolved by whoever shows up.

Stop asking for a demo. Ask for the validation data and the alert volume per 100 admissions.

Stop framing the choice as adoption versus resistance. The useful question is which specific tool, for which patients, with what monitoring, owned by whom.

A 30-day plan for the physician who just got handed this

Many of you will be volunteered for an AI committee without warning and without protected time. Here is what I would do in the first month.

Week one, build the inventory. Ask a simple question in writing: what AI is running in this organization today, and who approved each one. You will get an incomplete answer. The incompleteness is the finding. Only 29 percent of hospitals in the Black Book survey had enforced policies covering AI model inventory, lineage, and sign-offs, so an unclear answer puts you in the majority, not in trouble.

Week two, pick one tool and go see it. Not a demo. Go to the unit, stand next to the person using it, and count. How many alerts per shift. How many acted on. How many silenced without reading. Bring numbers back, not impressions.

Week three, read one contract. Ask legal for the agreement behind that tool and look for four things: local validation, performance monitoring obligations, notification when the vendor updates the model, and data portability on exit. Note which are missing.

Week four, propose one decision. Something concrete and small. Retune a threshold, add a monitoring metric, or sunset a tool nobody uses. Get it approved and executed.

At the end of that month you will know more about your organization's AI risk than most of the people who bought it. That is not an exaggeration, and it is not a compliment to the industry.

The regulatory floor is rising, and it names clinicians

On June 2, 2026, The Joint Commission launched a voluntary Responsible Use of AI in Healthcare certification across five domains including governance, monitoring and validation, and transparency with education and training (Fierce Healthcare).

State law is more pointed. Utah's SB 319 and Georgia's SB 544, both effective January 1, 2027, require that a licensed professional make an adverse determination independently rather than at a system's direction. Indiana's HB 1271, effective July 1, 2026, bars AI as the sole basis for downgrading a claim without professional review. Delaware's HB 191 bars AI systems from licensure or protected professional titles (Holland & Knight).

Read those statutes carefully and a pattern emerges. Legislators are writing the physician back into the loop by force, because the market did not do it voluntarily.

I would rather define that role than have it defined for me.

Where I have landed

I do not think physicians should be the brake on healthcare AI. I have written more than 30 books on this subject precisely because I think the upside is real and the workforce math leaves no alternative.

But the tools get safer when a clinician sits at the table from the first meeting, names the problem, checks the validation, designs the workflow, defends the override, and keeps the authority to pull the plug.

That is not resistance. That is the job.

Harvey Castro, MD, MBA is a board-certified emergency physician, 5x TEDx speaker, and author of more than 30 books on AI and healthcare, including AI in Emergency Medicine (Wiley). He serves on Singapore's Ministry of Health Regulatory Advisory Panel and advises the Texas Medical Association's Committee on Health Information Technology.

Related DR GPT™ reading

• AI Governance in Healthcare: A Physician's Framework for Boards and Executives
• Clinical AI Is Not a Technology Problem. It Is Trust, Workflow, and Judgment

More from DR GPT™: who DR GPT™ is, healthcare AI keynotes, board advisory, speaking, TEDx talks, books, and the media kit.

Book Harvey Castro, MD, MBA, known as DR GPT™, for a healthcare AI keynote, board retreat, or executive session. Start the conversation.