Automation bias: when a recommendation ends the inquiry

Automation bias describes excessive reliance on an automated recommendation. The important word is excessive: using a reliable tool can be sensible. The problem appears when its answer receives more weight than the task, evidence or circumstances justify.

NIST includes automation bias among risks in human interaction with generative AI. This is a description of a possible failure, not a claim that everyone who uses an assistant becomes less careful. NIST Generative AI Profile, section 2.7.

When a correct answer builds the wrong habit

Imagine a volunteer who uses a scheduling assistant for a busy hall. For months it catches clashes she would otherwise miss. One afternoon it recommends a room that she knows has a broken lift. She notices the problem but accepts the suggestion, assuming the software must know something she does not.

Her earlier trust was earned. The present assumption needs checking. Perhaps the lift has been repaired; perhaps the assistant has old information. Neither automatic obedience nor automatic rejection resolves that uncertainty. The useful next step is to inspect the relevant record or ask the person responsible.

A system can make that step easy by showing the date and source of its accessibility information. A manager can make it practical by allowing time to check. If every correction requires an explanation while acceptance requires one click, the workflow gives agreement an advantage before anyone evaluates the recommendation.

What this is different from

Sycophancy concerns an AI response bending toward a user's expressed view. Automation bias concerns a person's reliance on an automated output. They can reinforce each other, but they name different parts of the interaction. A person may trust a flattering answer too much; another may trust an unwelcome ranking too much because it appears objective.

Nor is the presence of a human signature enough to establish meaningful oversight. We need to know what the signer could examine and change.

Read the decision

The open sample of Sovereign Mercy puts a driver between a confident instruction and observations from the road. It is an invented emergency, not evidence about the safety of real routing products. Its useful question is what makes contradictory information actionable before the moment has passed.

When reading, ask what would justify continuing to trust the recommendation, what would justify checking it, and whether the person actually has time and authority to do either. That is a more revealing question than deciding in advance whether the human or the machine must be right.