Patient trust in AI starts with human accountability

DR GPT™ perspective

Trust is not a property of a model. It is a property of a system, and systems have names attached to them.

The instinct when trust is low is to improve the model. Higher accuracy, better benchmarks, a more impressive demo. That instinct is wrong, and the arithmetic shows why. Consider a hypothetical comparison: a less accurate model with effective error detection might cause less harm than a more accurate model whose errors go unnoticed. The percentages alone cannot establish safety. Evaluate the errors, their consequences and the effectiveness of oversight in the intended setting.

The practical form of that is a named human on every output. Not a committee, not a policy, and not the phrase the algorithm recommended. A person, with three things: a name, the authority to override, and enough time to disagree. Strip any one of those and what is left is not oversight. A clinician with eleven seconds and no authority to push back is liability laundering, and patients feel the difference even when they cannot name it.

Trust also behaves asymmetrically, which is the part institutions underestimate. Deposits are slow and withdrawals are instant. One visible failure can damage an entire category of care for years, well beyond the organization that caused it. That is why caution here is not timidity. It is arithmetic.

Originally published on KevinMD. Read the full essay, or see how I build these arguments for a room as a speaker.