Automation bias — the tendency to favour AI-generated outputs over human judgement — is not a training problem. It is a measurable psychological phenomenon.
Automation bias — the tendency to favour AI-generated outputs over human judgement — is not a training problem. It is a measurable psychological phenomenon that increases with system confidence, time pressure, and cognitive load. Most governance frameworks do not account for it.
The term was first described in aviation research, where investigators found that pilots would follow automated navigation systems even when visual evidence contradicted the system's output. The same phenomenon has since been documented across healthcare, criminal justice, financial services, and every domain where humans interact with automated decision-support systems.
The EU AI Act explicitly recognises automation bias as a risk to human oversight. Article 14 requires that persons assigned to human oversight be 'aware of the possible tendency of automatically relying on or over-relying on the output produced by a high-risk AI system.' Awareness, however, is not immunity.
Research consistently demonstrates that knowing about automation bias does not prevent it. The phenomenon operates at a cognitive level that is largely resistant to awareness-based interventions. People who know about automation bias still exhibit it — particularly under the conditions that characterise real-world AI oversight: time pressure, high stakes, and cognitive load.
This has profound implications for governance design. If awareness does not mitigate automation bias, then governance frameworks that rely on training and awareness programmes to ensure genuine human oversight are structurally inadequate.
The Responsible AI Center's Governance Diagnostic includes constructs designed to assess automation bias susceptibility in oversight-critical roles. We measure not whether people know about automation bias, but the extent to which they are likely to exhibit it in the specific context of their governance responsibilities.
This assessment is grounded in the psychological research on automation bias and calibrated for the AI governance context. It provides organisations with a specific, actionable picture of where automation bias is likely to undermine oversight — and what structural interventions can mitigate it.
Effective mitigation of automation bias requires structural interventions — changes to the conditions under which oversight is exercised, not just the knowledge of the people exercising it. These may include mandatory deliberation periods before accepting high-confidence AI recommendations, structured disagreement protocols that require explicit justification for accepting AI outputs, rotation of oversight responsibilities to prevent habituation, and environmental design that reduces the cognitive conditions under which automation bias is most acute.
These are governance design decisions, not training decisions. They require an evidence base — which is precisely what the Governance Diagnostic provides.
Every engagement begins with a Discovery Conversation. Just an honest exchange about where your organisation stands and whether we are the right fit to help.
Book a Discovery Conversation