What Every Hospital CEO Should Know Before Buying AI

I have worked enough overnight shifts to know what a bad alert sounds like. It sounds like nothing, because by 2 a.m. nobody is listening anymore.

That is the part of the AI conversation that rarely makes it into the boardroom. A vendor shows a slide with an impressive number on it. Someone in the room says the word "safety." Someone else says the word "efficiency." Nine months later a nurse on the fifth floor silences the same pop-up for the ninetieth time that week, and the hospital is paying six figures a year for an alarm that has been trained out of the building.

I am a board-certified emergency physician. I have also sat on the buying side of this table as an executive. Here is what I would want to know before I signed.

The vendor's number is almost never your number

Every clinical AI product arrives with a performance figure attached. The question no one asks often enough is where that figure came from.

The clearest example in the literature is the Epic Sepsis Model. It was deployed at hundreds of US hospitals before anyone published an independent evaluation. When researchers at Michigan Medicine finally ran one across 38,455 hospitalizations, the model's area under the curve came in at 0.63. It missed 1,709 of 2,552 sepsis patients, roughly 67 percent, while firing alerts on 18 percent of all hospitalized patients. According to PubMed, that study is Wong A, Otles E, Donnelly JP, et al., published in JAMA Internal Medicine in 2021 (DOI).

Read that again. Hundreds of hospitals. Widespread deployment. Poor discrimination, poor calibration, and a heavy alert burden on the nursing staff.

The model was not fraudulent. It was trained on one population and sold into thousands of others. A prediction tool built on a suburban academic center's patient mix behaves differently in a safety-net hospital in South Texas. My patients in an emergency department in Dallas do not present the way patients present in Rochester.

So the first question is simple. On which population was this validated, and will you validate it on mine before go-live?

If the answer involves a pause, the pause is the answer.

Adoption has run ahead of governance, and the gap is measurable

The buying is happening fast. Among the 2,784 US hospitals on Epic, 62.6 percent had adopted ambient AI documentation tools as of June 2025, according to research published in the American Journal of Managed Care in January 2026 (Yang F, Graetz I). Adoption skewed toward metropolitan hospitals, nonprofit systems, and institutions with healthier operating margins. For-profit adoption sat at 28.8 percent against 70.2 percent for nonprofits.

The oversight is not keeping pace. Black Book Research surveyed 182 hospital leaders between October and November 2025 and found that only 29 percent had implemented and enforced policies covering AI model inventory, lineage, and sign-offs. Only 22 percent were highly confident they could produce a complete AI audit trail within 30 days if a regulator or payer asked. In small hospitals that figure fell to 15 percent. The median share of 2026 IT and quality budgets going to AI governance and safety: 4.2 percent (Becker's Hospital Review).

A CEO reading those two paragraphs together should notice the shape of the risk. Most systems have bought. Fewer than a third can prove what they bought, who approved it, and how it is performing today.

Five questions to ask before you sign

I keep these short on purpose. If a vendor cannot answer them in one meeting, that tells you something about the next three years.

One. What is the performance on my patients, and who pays to find out? Ask for a local validation period with a defined sample, defined metrics, and a defined kill switch. Put the cost of that validation in the contract, not in a side conversation.

Two. What happens when the model drifts? Populations change. Coding practices change. A model that performed in March can quietly decay by November. Ask who monitors, how often, against what threshold, and who is required to tell you when performance falls.

Three. Who is accountable for the clinical decision? Not who is liable in a lawsuit. Who, by name and role inside the hospital, owns the output. If the answer is "the clinician," then the clinician needs the authority to override without friction and without a productivity penalty.

Four. What does this do to my clinicians' day? Ambient documentation gives time back. A poorly tuned prediction model takes time away and adds alarm fatigue. Ask for the alert volume per 100 admissions, then ask a working nurse whether that number is survivable.

Five. Can I get my data out? Model outputs, audit logs, and the underlying data should leave with you if the relationship ends. Say so in writing.

Put the governance in the contract, not the press release

The clauses I would fight for: local validation before go-live, ongoing performance monitoring with reporting obligations on the vendor, a model card or equivalent documentation that a quality committee can actually read, notification requirements when the vendor updates the model, and a clean exit with full data portability.

That fourth one matters more than it sounds. A silent model update is a new device in your hospital that nobody credentialed.

Note that 41 percent of the hospital leaders in the Black Book survey named insufficient vendor documentation as their primary obstacle to audit readiness, and 33 percent named unclear ownership across IT, quality, safety, and compliance. Both of those are contract problems and org-chart problems. Both get solved before the purchase or not at all.

The oversight is arriving on a published schedule

This is no longer a voluntary conversation you can defer to next fiscal year.

In September 2025, The Joint Commission and the Coalition for Health AI released joint guidance on the responsible use of AI in healthcare. On June 2, 2026, The Joint Commission launched a voluntary Responsible Use of AI in Healthcare certification built around five domains: governance, data management, risk and bias reduction, monitoring and validation of safety and effectiveness, and transparency with education and training. Any healthcare organization may apply, and prior accreditation is not required. The program certifies organizational practice, not individual products (Fierce Healthcare).

State law is moving faster than most boards realize. Indiana's HB 1271 took effect July 1, 2026. Maryland's HB 1563 took effect June 1, 2026. Alabama's SB 63 arrives October 1, 2026, and Utah's SB 319 and Georgia's SB 544 follow on January 1, 2027 (Holland & Knight). The common thread across them is that a licensed human, not a model, must own an adverse determination.

If you operate across state lines, your AI policy is now a fifty-state problem.

Buy the workflow, not the model

The strongest business case for clinical AI is not diagnostic accuracy. It is capacity.

The Association of American Medical Colleges projects a shortage of up to 86,000 physicians by 2036, driven by a population over 65 growing 34.1 percent and a population over 75 growing 54.7 percent, against a clinical workforce where 20 percent of physicians are already 65 or older (AAMC). You cannot hire your way out of that curve. You can redesign the work.

That reframes the purchase. The question stops being "how accurate is the model" and becomes "how many minutes does this give back to a clinician, and what happens in those minutes." Ambient documentation passes that test in most systems. A prediction score bolted onto an unchanged workflow usually does not.

What I tell CEOs

Start with one clinical problem you can name, one owner who can be paged, and one number you will measure at 90 days. Validate locally. Write the monitoring into the contract. Give clinicians a real override. Then scale what worked and kill what did not, publicly, so the organization learns that saying no to a tool is a normal outcome rather than a failure.

AI will not fail your hospital. Buying decisions made without a physician in the room will.

Harvey Castro, MD, MBA is a board-certified emergency physician, 5x TEDx speaker, and author of more than 30 books on AI and healthcare, including AI in Emergency Medicine (Wiley). He serves on Singapore's Ministry of Health Regulatory Advisory Panel and advises the Texas Medical Association's Committee on Health Information Technology.

Related DR GPT™ reading

• AI Governance in Healthcare: A Physician's Framework for Boards and Executives
• Healthcare AI Keynote Speaker: What Hospital Leaders Need to Hear in 2026

More from DR GPT™: who DR GPT™ is, healthcare AI keynotes, speaking, TEDx talks, books, and the media kit.

Book Harvey Castro, MD, MBA, known as DR GPT™, for your board, leadership retreat, or healthcare AI event. Start the conversation.