I was recently at the Microsoft most valuable professional (MVP) summit in Redmond Seattle and the term automation bias was introduced to me during a talk and it was a concept that really caught my interest. So I decided to explore it in definition, it’s impacts and how we can combat it.

What is automation bias?

‘Automation bias is the propensity for humans to favor suggestions from automated decision-making systems and to ignore contradictory information made without automation, even if it is correct.

To give an example relevant to most, automation bias is when you get ChatGPT to write a report/email/essay for you and just accepting it will be correct with little to no checking of the outputs of the GenAI. To my surprise, it is not unique to the age of GenAI with an older example being people driving into lakes because their GPS said it was the correct route.

What’s the impact?

If we trust AI blindly at the basal level we are choosing to erode our abilities to think critically as we just accept the outputs of automated processes. But if we think more granularly you can break this down to the following impacts:

• reduced vigilance of outputs thus more mistakes are allowed.
• loss of skills as automated systems are solely used to complete tasks, without user applying the skills themselves to verify outputs.
• The duality of inaction and action on false information, where either the results we see don’t suggest a required action, or it does but is incorrect in that suggestion).

Ultimately, if we let automated systems such as GenAI do it’s own thing unchecked we risk spreading misinformation, not acting at the right time or in the right way and the loss of essential skills such as critical thinking required to question and verify whether what we are producing is correct and to what degree and ultimately how to correct it.

How can we correct this?

There are three key strategies spoke about in connection to correcting automation bias and they are usually used in combination; emphasizing system design which puts in place safeguards, training and fostering critical thinking in regard to the results of automated systems.

System design should ensure that automated systems are transparent and give reasoning for the decisions and recommendations. This is already coming through natively in the new wave of reasoning models we are seeing (Deepseek R1 and GPT 01 for example). The systems should also incorporate alerts which can flag potential errors or cases where a human should be prompted to double check the results of the systems. It is also encouraged that process are not fully automated and humans should always have a presence in the systems processes, usually at key decision making points as to maintain vigilance and thus guide the model.

Training is normally concentrated on two areas; knowledge of how the systems work and the importance of human oversight and critical thinking. By stressing these through training you allow users to understand the strengths and weakness’ of the systems and thus they can self prompt themselves in situations where they know the systems is more likely to require human oversight.

Finally by encouraging critical thinking, users should always address the outputs of these systems with skeptism, thus prompting them to analyse and question the system independently. This allows users to recognise the limitations of these systems and maintain a balance between human judgement and system recommendations.

Summary

Automation bias isn’t a new thing but it’s impacts can be massive depending on where systems are being applied, such as in healthcare or finance where errors in information can be catastrophic. But by applying the points above and ultimately always relying on our human skill to think critically we can ensure users stay vigilant and actively engaged thus reducing errors and promoting informed decision making.