What Every Hospital CEO Should Know Before Buying AI

By Harvey Castro, MD, MBA

I have worked enough overnight shifts to know what a bad alert sounds like. It sounds like nothing, because by 2 a.m. nobody is listening anymore.

That is the part of the AI conversation that rarely makes it into the boardroom. A vendor shows a slide with an impressive number on it. Someone in the room says the word "safety." Someone else says the word "efficiency." Nine months later a nurse on the fifth floor silences the same pop-up for the ninetieth time that week, and the hospital is paying six figures a year for an alarm that has been trained out of the building.

I am a board-certified emergency physician. I have also sat on the buying side of this table as an executive. Here is what I would want to know before I signed.

The vendor's number is almost never your number

Every clinical AI product arrives with a performance figure attached. The question no one asks often enough is where that figure came from.

The clearest example in the literature is the Epic Sepsis Model. It was deployed at hundreds of US hospitals before anyone published an independent evaluation. When researchers at Michigan Medicine finally ran one across 38,455 hospitalizations, the model's hospitalization-level area under the curve came in at 0.63. It failed to identify 1,709 of 2,552 sepsis hospitalizations, roughly 67 percent, while hypothetically flagging 18 percent of hospitalizations at the evaluated threshold. The analysis was retrospective, used a score threshold of at least 6 for these misses and hypothetical flags, and excluded hospitalizations eligible for actual alerts. According to PubMed, that study is Wong A, Otles E, Donnelly JP, et al., published in JAMA Internal Medicine in 2021 (DOI).

Read that again. Hundreds of hospitals. Widespread deployment. Poor discrimination, poor calibration, and a heavy alert burden on the nursing staff.

The paper describes model development and validation using data from three health systems. It does not establish that differences between patient populations alone explain the lower performance in this external evaluation (Wong et al.).

So the first question is simple. On which population was this validated, and will you validate it on mine before go-live?

If the answer involves a pause, the pause is the answer.

What adoption and governance studies measure

Among 2,784 sampled US hospitals using Epic, 62.6 percent were classified as adopting ambient AI documentation as of June 2025. That definition included health systems that had implemented or were implementing a tool. In adjusted analyses, estimated adoption was 28.8 percent for for-profit hospitals and 70.2 percent for nonprofits (Yang and Graetz, American Journal of Managed Care, 2026).

Separately, Black Book Research surveyed 182 hospital leaders from October 15 to November 8, 2025. In that survey, 29 percent reported enforced policies covering AI inventory, lineage, and sign-offs; 22 percent reported high confidence in producing an auditable AI explanation within 30 days, falling to 15 percent in the small-hospital group. These are self-reports from the surveyed group, not audited findings for all hospitals (Black Book Research release).

The adoption study and governance survey cover different samples and measure different things. Read each in its own context; their percentages cannot establish how many health systems have adopted AI without adequate governance.

Five questions to ask before you sign

I keep these short on purpose. If a vendor cannot answer them in one meeting, that tells you something about the next three years.

One. What is the performance on my patients, and who pays to find out? Ask for a local validation period with a defined sample, defined metrics, and a defined kill switch. Put the cost of that validation in the contract, not in a side conversation.

Two. What happens when the model drifts? Populations change. Coding practices change. A model that performed in March can quietly decay by November. Ask who monitors, how often, against what threshold, and who is required to tell you when performance falls.

Three. Who is accountable for the clinical decision? Not who is liable in a lawsuit. Who, by name and role inside the hospital, owns the output. If the answer is "the clinician," then the clinician needs the authority to override without friction and without a productivity penalty.

Four. What does this do to my clinicians' day? Ambient documentation gives time back. A poorly tuned prediction model takes time away and adds alarm fatigue. Ask for the alert volume per 100 admissions, then ask a working nurse whether that number is survivable.

Five. Can I get my data out? Model outputs, audit logs, and the underlying data should leave with you if the relationship ends. Say so in writing.

Put the governance in the contract, not the press release

The clauses I would fight for: local validation before go-live, ongoing performance monitoring with reporting obligations on the vendor, a model card or equivalent documentation that a quality committee can actually read, notification requirements when the vendor updates the model, and a clean exit with full data portability.

That fourth one matters more than it sounds. An unannounced model update can change what the hospital is using. The contract should make changes visible so the organization can decide what reassessment is needed.

Note that 41 percent of the hospital leaders in the Black Book survey named insufficient vendor documentation as their primary obstacle to audit readiness, and 33 percent said unclear ownership across IT, quality, safety, and compliance slowed governance progress. Both of those are contract problems and org-chart problems. Both get solved before the purchase or not at all.

The oversight is arriving on a published schedule

The certification is voluntary. Applicable state-law duties are separate and depend on the jurisdiction, regulated entity, and decision involved.

In September 2025, The Joint Commission and the Coalition for Health AI released joint guidance on the responsible use of AI in healthcare. On June 1, 2026, The Joint Commission launched a voluntary Responsible Use of AI in Healthcare certification built around five domains: governance, data management, risk and bias reduction, monitoring and validation of safety and effectiveness, and transparency with education and training. Prior Joint Commission accreditation is not required. Applicants must be US-based, US-government-operated, or congressionally chartered and meet the program's accreditation or applicable federal-law compliance conditions, with established AI oversight and responsible-use processes. Health systems must comply with applicable federal laws and share AI governance, processes, and policies (official eligibility criteria). The program certifies organizational practice, not individual products (Joint Commission announcement).

These laws cover different decisions and actors.

Indiana's HB 1271 added a medical-necessity downcoding provision effective July 1, 2026: an insurer may not use automation, including AI, as the sole basis without an employee or contractor reviewing the covered person's medical record. That clause does not specify a licensed clinician, and the chapter excludes Medicaid and Medicaid managed-care organizations.

Maryland's HB 1563, effective June 1, 2026, concerns emergency-room coverage decisions and post-acute-care studies; it is not a general AI human-review mandate.

Alabama's SB 63 specifies an October 1, 2026 effective date and requires a licensed physician or other competent health care professional to make medical-necessity decisions to deny, delay, or modify prior-authorization requests.

Utah's SB 319, effective January 1, 2027, addresses adverse preauthorization determinations involving clinical or medical necessity. The decision-maker must know the condition or consult a knowledgeable specialist, exercise independent medical judgment subject to the specified specialist-consultation exception, and not rely solely on another source's recommendations. Separate provisions govern licensed review of specified appeals.

Georgia's SB 444, effective January 1, 2027, requires a qualifying natural person to conduct utilization review with a clinical peer participating before AI or software issues an adverse determination; the software cannot supersede that peer's judgment.

If you operate across state lines, your AI policy is now a fifty-state problem.

Buy the workflow, not the model

The strongest business case for clinical AI is not diagnostic accuracy. It is capacity.

In March 2024, the AAMC projected a national shortage of up to 86,000 physicians by 2036. The report also emphasized investment in physician training (AAMC March 21, 2024 report announcement). That projection alone does not establish which staffing options will work for each health system. Workflow redesign is another option to evaluate.

That reframes the purchase. The question stops being "how accurate is the model" and becomes "how many minutes does this give back to a clinician, and what happens in those minutes." Evaluate documentation tools and prediction systems in the local workflow. Evidence from one tool or organization does not establish benefit in most health systems.

What I tell CEOs

Start with one clinical problem you can name, one owner who can be paged, and one number you will measure at 90 days. Validate locally. Write the monitoring into the contract. Give clinicians a real override. Then scale what worked and kill what did not, publicly, so the organization learns that saying no to a tool is a normal outcome rather than a failure.

AI will not fail your hospital. Buying decisions made without a physician in the room will.

Harvey Castro, MD, MBA (DR GPT™) is a board-certified emergency physician, 5x TEDx speaker, Advisor to the CEO of Phantom Space, and author of books on AI, medicine and leadership.

Related DR GPT™ reading

• AI Governance in Healthcare: A Physician's Framework for Boards and Executives
• Healthcare AI Keynote Speaker: What Hospital Leaders Need to Hear in 2026

More from DR GPT™: who DR GPT™ is, healthcare AI keynotes, speaking, TEDx talks, books, and the media kit.

Book Harvey Castro, MD, MBA, known as DR GPT™, for your board, leadership retreat, or healthcare AI event. Start the conversation.