Who checks the checker?

This week's news has one question running under it: once AI is inside care, who checks it? Regulators, research funders, and Medicare all gave different answers.

1. FDA puts GenAI mental-health chatbots on its priority list

FDA's device center posted its guidance agenda for fiscal year 2027. The only new draft on the priority A-list covers generative AI conversational devices for mental disorders, with a focus on what evidence they need. The broader risk guidance for all GenAI-enabled devices sits on the "under construction" list instead.

So FDA is going one indication at a time. Therapy chatbots may get a written evidence bar before anyone gets a general GenAI rulebook. Keep in mind that an agenda is intent, not policy, and FDA doesn't always hit its list.

Source: FDA CDRH guidance agenda

2. ARPA-H funds an AI that supervises clinical AI

A UC San Diego team won an ARPA-H award to build a system that checks patient-facing heart-failure AI against rules that clinicians write. Some agents defend an answer, others try to refute it, and the rulebook can be swapped out.

This turns after-launch oversight into a piece of software, with clinicians writing the rules. It's a funded build with no data yet, and a supervisor can share the blind spots of the agent it watches. Press reports name Anthropic as a partner. The award page doesn't.

Source: ARPA-H award

3. HHS bets on digital twins and AI agents to shrink clinical trials

HHS and ARPA-H launched SURPASS. It funds digital twins and predictive models for trial design, real-time analysis that could need smaller control groups, and AI agents that handle trial startup and data work.

If simulated control arms earn regulators' trust, the evidence standard itself starts to depend on models. Then someone has to validate the twin. This is a program launch. "Faster, with fewer participants" is a goal, not a result.

Source: HHS launches SURPASS

4. Records from Medicare's AI prior-auth pilot show delays and paid denials

CMS records obtained by EFF through a public records request show prior-authorization requests in the WISeR pilot running past the 72-hour target, and vendor payments tied to denials. EFF later clarified that turnaround covers the whole request pathway, not the vendors alone.

This is the clearest U.S. case of AI shaping access to care through payment, outside device rules. EFF is a party in litigation here, so read the records yourself and treat its interpretation as interpretation. Prior-auth delays also predate AI, and the AI's exact role is still hard to see.

Source: EFF, Medicare's AI prior authorization experiment

5. Federal agencies will say "Super Intelligence" in place of "AI"

Executive Order 14434 tells agencies to use "Super Intelligence" or "SI" instead of "AI" in non-statutory documents. The order defines SI with the existing statutory definition of AI, so the label changed and the scope didn't.

Expect to see "SI" in future FDA and CMS documents, and in vendor pitches. When it shows up, ask what the tool does. The name alone tells you nothing new.

Source: Federal Register, Executive Order 14434

The pattern

Put the five together and the pattern is plain. FDA wants an evidence bar per use. ARPA-H is paying for software that watches software. Trials may lean on models for evidence. Medicare's pilot shows what access looks like when the checking is thin. The tools are arriving faster than anyone's plan for watching them after go-live.

If your hospital is piloting a mental-health chatbot or any patient-facing AI, forward this to the person who owns monitoring after go-live, and ask them who checks the tool once it's live.

Ask DR GPT™: If an AI supervisor flags a problem with a patient-facing tool at 2 a.m., who in your organization gets the alert, and do they have the authority to switch it off?

Educational and futurist context only. Not clinical advice. Not a diagnostic service.

Originally published on my Substack. Planning a session on AI oversight for your board? Explore my healthcare AI keynotes.