The Autonomy Trap
In 2019, researchers published a landmark study in *Science* (Obermeyer et al.) examining a commercial risk-prediction algorithm — sold by Optum and widely used across the US health system — that helped decide which patients were flagged for extra care programmes. At the time, algorithms of this kind were applied to an estimated 200 million people a year. The algorithm used healthcare spend as a proxy for medical need — and it was systematically deprioritising Black patients: at any given risk score, Black patients were significantly sicker than white patients. Nobody had designed it to discriminate. Race wasn't even an input.
The explanation was structural. Black patients had historically received less care — not because they needed it less, but because they had less access to it. The model interpreted lower historical healthcare spend as lower medical need and allocated resources accordingly. A proxy that seemed technically neutral was socially harmful: the researchers estimated that fixing it would nearly triple the proportion of Black patients flagged for additional care.
The health systems using the algorithm had given it substantial influence over a decision with serious consequences, with no mechanism in place to catch the skew in its outputs. The lesson isn't that AI can't be used in healthcare. It's that the level of autonomy given to any AI system must be proportionate to the stakes of the decision it influences.
What is the Autonomy Trap?
The Autonomy Trap is what happens when an organisation gives AI more decision-making power than it has earned — either because the tool seems impressive, because it reduces headcount, or simply because nobody explicitly designed the oversight structure. Once an AI is deployed autonomously, the instinct to trust its outputs grows over time. Reversing it becomes politically difficult, even when the evidence suggests it should be reversed.
Autonomy is easy to grant and hard to retract. Once an AI makes decisions without human review, the humans who would have reviewed those decisions are typically reassigned or made redundant. If the AI later needs human oversight reintroduced, the institutional knowledge to provide that oversight may no longer exist.
The next lesson introduces the Autonomy Spectrum — a five-level model that gives you a structured way to decide how much autonomy any AI system should have, and what human oversight mechanisms must exist at each level.
