80Complacency and Bias in Human Use of Automation

Raja Parasuraman and Dietrich H. Manzey, Human Factors 52(3), pp. 381–410, 2010

Linked to 21 chapters: supports 11 · contradicts 3 · mechanism 11 · same claim 2 · pairing 3

Parasuraman and Manzey take two terms that grew up in separate literatures and argue that they name the same thing. Complacency came out of aviation, where investigators used it for crews who stopped checking, and it was studied in supervisory control, where an operator watches an automated process while working by hand on something else. Automation bias came out of decision research, where a person is handed an aid that names a fault and recommends a response, and it was studied as a heuristic, a cousin of confirmation bias. Both literatures kept finding the same things: the effect appears in students and in experienced pilots alike, it survives practice, it grows as the machine becomes more reliable, and instructions to cross-check do not remove it. What joins them, the review argues, is attention. Complacency is not a state the operator sinks into; it is an active reallocation of attention away from the automated task and towards the manual ones. The same reallocation, applied to the raw readings that would confirm or refute a decision aid, produces the errors that had been called bias. The paper closes with a model in which a disposition, a task context and a machine's reliability combine into an attentional strategy that two feedback loops push in opposite directions.

Automation complacency

The word entered the literature as a coding item. The Aviation Safety Reporting System, set up in 1976, defined complacency for its incident reports as non-vigilance based on an unjustified assumption of satisfactory system state. Wiener discussed that definition in 1981 alongside others, including complacency as "a low index of suspicion", and argued that the concept would earn its place in safety work only once experiments had given it a mechanism. Three decades later no consensus definition exists, and Parasuraman and Manzey assemble a working one from three features common to the accident analyses and the experiments: an operator is monitoring an automated system; the frequency of that monitoring falls below some optimal value; and the shortfall has an observable consequence, usually a missed malfunction.

The accident record supplied the motive. In a survey of 100 highly experienced airline captains, more than half named complacency a leading factor in accidents, and a 1999 analysis of automated-aircraft accidents put it among the top five factors. Outside aviation the standing case is the grounding of the cruise ship Royal Majesty off Nantucket: an antenna cable frayed, the satellite signal was lost, the radar plotting aid silently reverted to dead reckoning without correcting for tide and wind, and the ship was carried onto a sand bank. Investigators cited overreliance on the aid and insufficient monitoring of the other navigational sources. Dekker and Hollnagel called complacency and situation awareness folk models without scientific standing; the reply is the experiments, which the accident reports had only motivated.

What the multi-task studies found

The first serious attempt to produce the effect in a laboratory failed. Thackray and Touchstone gave participants a simple conflict-detection task with an aid that failed twice in two hours, and found them as accurate with the aid as without it. Parasuraman and Manzey blame the emptiness of the session rather than its length: the participants had nothing else to do.

Parasuraman, Molloy and Singh supplied the missing ingredient in 1993. Participants ran three tasks at once from the Multi-Attribute Task Battery: compensatory tracking and fuel management by hand, and engine monitoring supported by imperfect automation. Following Langer's notion of a premature cognitive commitment — an attitude formed on first meeting a device and confirmed every time it behaves the same way — they also varied whether the automation's reliability held constant or fluctuated. Detection of its failures averaged 82% when reliability varied and 33% when it held constant, a difference of 149%. Under single-task conditions the same malfunctions were easy, detected about 97% of the time, and participants asked only to back up the automation caught nearly all of them.

That pattern is what makes the everyday sense of the word wrong. Complacency is not passivity. The operator is working hard, and the work is going somewhere else.

What does not remove it

Moving the automated display from the periphery to the centre of the screen changed nothing. Cutting the failure rate to one failure in a thirty-minute session changed nothing: alone with the automation most participants caught it, under multiple tasks about half did, and fewer caught a late failure than an early one. In a driving simulator where lateral control was automated and a vehicle merged suddenly into the lane, the system did not react, half the drivers failed to brake and take over, and 14% did not respond in time to avoid the collision. Lowering reliability helped a little: detection varied inversely with reliability, but monitoring stayed worse than manual performance even at low reliability, and no floor appeared below which the cost vanished.

Expertise did not help. Pilots with about 440 hours showed the effect; general aviation pilots using a real cockpit alerting system detected fewer engine malfunctions with it than without it; experienced controllers given a conflict probe that failed once missed the conflict more often than the same conflict handled by hand. An hour of extra practice left the effect intact. Training that addresses attention rather than the task did better: variable-priority training, alternating which task gets 80% of the trainee's effort, improved multitasking and produced a trend towards less complacency.

The most instructive failure was Duley's. He superimposed the automated engine display on the manual tracking task, interleaving the elements so that neither masked the other and every fixation of the tracking task put the automated task in foveal vision too. Tracking was as good as before, and so was the complacency effect. Whatever is being withdrawn is not only the eyes. Eye movements carry part of the story: among controllers who missed the conflict there were significantly fewer fixations of the radar display under automation, and among those who caught it there were none.

Trust is the weakest joint. It is the obvious explanation for why constant reliability breeds inattention, and the direct evidence is thin: one replication reproduced the constancy effect with no accompanying rise in measured trust, another found an inverse relation between trust and monitoring only when two experiments were pooled, and Lee and See found no convincing link at all. Parasuraman and Manzey suspect the measure, and propose that what an operator does to verify a machine indexes trust better than what the operator says. Individual differences are real and poorly characterised: the one scale in use, twenty items on attitudes to everyday automated devices, predicts poor monitoring only above a threshold.

Automation bias

A decision aid does two things: it alerts, and it recommends. Each function has its own failure. Mosier and Skitka defined automation bias as using the aid's output as "a heuristic replacement for vigilant information seeking and processing", which places it among the ordinary decision heuristics, with the qualification that this one comes from a machine.

Three things are held to produce it: cognitive economy, since a recommendation is cheaper than an analysis; imputed authority, since participants asked to predict how well an automated aid would do against a human helper expected the machine to be considerably better before either had done anything; and diffusion of responsibility, the slackening that occurs whenever two parties are given one job.

The evidence runs across four domains. In a simulated engine fire, an automated checklist recommended shutting down the engine that was not on fire, and 75% of pilots complied, against 25% working from a paper checklist. In a full simulator study, when a false engine fire warning appeared with no supporting reading anywhere on the flight deck, every pilot shut the engine down, though in debriefing the same pilots said that such a message alone would not justify it. Two thirds of them reported having seen a second indication supporting the warning, which had not been there: Mosier called it phantom memory. Non-pilots running a three-task simulation missed 41% of the events the automation failed to announce, against about 3% unaided, having been told the gauges were always valid. Radiologists reading mammograms fell from 46% to 21% detection on cancers the aid left unmarked, and in a command-and-control task accuracy fell from 89% unaided to 70% when the aid was wrong — worst where the aid recommended a specific action and where its overall reliability was highest.

The in-flight icing study shows the asymmetry plainly: an aid that named the icing condition, and one that also prescribed pitch, flap and power, both cut stalls when their information was right and raised stalls to 88.89% when it was wrong, against 30.00% for pilots flying on their own sense of the aircraft. Aids that recommend actions produce more commission errors than aids that only analyse information, and an aid that publishes its confidence case by case costs less when wrong.

Two interventions that ought to work do not. Neither a briefing on omission and commission errors nor an explicit prompt to verify the aid changed the error rates, in 181 non-pilots or in 48 experienced glass-cockpit pilots, and a second crew member working redundantly changed nothing. What did change them was accountability: participants who expected to justify their accuracy afterwards made fewer errors of both kinds and more use of the monitor provided for checking the aid.

Toward an integrated model

The two constructs meet first at the omission error, which is complacency under another name: the operator has leaned on the alerting function and stopped watching the environment. The commission error splits. Discounting information that contradicts the aid is a decision bias in the strict sense. Following the aid without looking for the contradiction is not a bias in weighing evidence at all; it is a failure to gather it, which is the same reallocation of attention.

Moray had pressed a sharp objection. Monitoring a machine less often than one's own tasks is rational when the machine is reliable, and misses are the price of a sound strategy; complacency can be diagnosed only against a normative sampling rate. Parasuraman and Manzey accept the demand and doubt the supply. Moray's normative rate comes from the sampling theorem, which needs the bandwidth of the signal, and the bandwidth of most real information sources cannot be computed; nor does the theorem price the cost of a head movement.

The answer came from a process-control microworld simulating a spacecraft life-support system. Bahner, Hüper and Manzey gave operators an aid that diagnosed faults and recommended responses, and mouse access to every tank level, flow rate and history graph needed to check a diagnosis. Because a complete check was enumerable, actual sampling could be set against normative sampling, and complacency measured before any failure had occurred. Every group sampled less than a complete check required; operators merely told that the aid might fail sampled less than operators who had met a few rare failures during training. In a companion experiment, 18 of 24 participants followed the first wrong diagnosis they met; four fifths of them had not checked enough to catch it, and one fifth had looked at every parameter that contradicted it.

That last fraction is what the integration turns on. In a further study, about 43% of operators missed the first wrong diagnosis; half had sampled incompletely and half had sampled everything. Those who caught the error took markedly longer over a parameter that did not fit the diagnosis than over one that did. Those who missed it showed no such increase: they had looked at the contradiction and spent nothing on it.

So a commission error has three sources: an overt reduction in sampling, a covert withdrawal of attention from information that was looked at, and an active discounting of a contradiction that was seen. The first two are attentional and account between them for most of the observed errors. That is the paper's central claim, and it makes complacency and automation bias two readings of one instrument.

The model

Complacency potential — a tendency to deal with a particular system less attentively — is set by that system's reliability and constancy, by the operator's history with it, and by the person. Whether the potential becomes an attentional bias depends on the task context, chiefly workload and how accountable the operator feels, and on the operator's state; fatigued operators, in one study, cross-checked an aid more carefully than alert ones. The attentional bias produces selective processing of information, whose cost is a loss of situation awareness: invisible while the automation works, and an omission or a commission error the moment it does not.

The two loops make the account dynamic rather than a list of factors. Because automation is usually reliable, inattentive operation usually has no consequence, and the absence of consequences raises complacency potential further: the positive loop, which the authors name after learned carelessness in industrial safety. A failure actually experienced runs the other way, which is why the operators who had met failures in training sampled more. The remedies follow the loops: aid the analysis rather than the choice, publish the machine's confidence case by case, raise accountability, and let operators meet failures before the failures matter.

Since 2010

The integration has held. That complacency requires competing tasks, that it survives expertise and practice, and that decision aids produce both omission and commission errors across aviation, medicine, process control and command and control are settled findings, and the attentional reading of automation bias is the standard one. The term has travelled furthest, into clinical decision support, forensic examination and algorithmic risk scoring, where it is often used without the distinction that gives it its content.

The loose joints are the ones the authors named. Trust still does not measure well: self-report scales predict monitoring poorly, and the behavioural indices the paper recommended have been adopted without displacing the questionnaires. Moray's requirement that complacency be diagnosed against a normative sampling rate remains the honest standard and the rare practice; most studies still infer it from missed failures, which is what he objected to. The reliability threshold below which an aid stops paying for itself, put at roughly 70%, has not survived as a constant, and looks instead like a function of workload and of whether the raw data are available at all.

That last qualification has grown teeth. Bahner's design depended on an operator who could enumerate and inspect every reading the aid had used. Aids built since do not always expose one, and where the evidence cannot be sampled the distinction between failing to check and being unable to check collapses, along with the normative model that gave complacency its meaning.

The diagnosis returns to where the field began. Ironies of Automation argued that the more reliable a system, the less its operator monitors and the less able that operator becomes to take over when it fails. Parasuraman and Manzey supply the measured mechanism and correct one thing about it. What the reliable machine takes is not principally skill. It is attention, which it takes quickly, from experts as readily as from novices, and in proportion to how well it has been working.