1Ironies of Automation

Lisanne Bainbridge, Automatica 19(6), pp. 775–779, 1983

Linked to 44 chapters: supports 21 · mechanism 29 · same claim 13 · pairing 17

Figure 1.0 — The paper's four numbered sections and what each contributes. The sentence it is remembered for opens the third.

Bainbridge's subject is the control room of a process plant at the start of the 1980s, where computers had taken over the running of reactors, refineries and power stations and a human operator had been kept on to watch and, if necessary, to take over. Her argument is that this arrangement contradicts itself. The designer automates because the human is thought unreliable, but the designer is also human and builds errors of the same kind into the system, where no one is on duty to catch them. The designer automates what can be specified and leaves the human what cannot: intervening when the automation fails, in an abnormal state, where the most skill is needed. And the automation, by taking the routine work away, removes the daily practice that kept that skill alive, and the running picture of the plant that made intervention possible. The operator is left with the hardest part of the job and the least preparation for it, and is asked in the meantime to stay alert for hours to a process on which nothing happens. Her remedies are practical and she is candid about their cost: hands-on control for a short period in each shift, alarms and displays designed for the moment things go wrong, and a rethinking of the computer's role so that it supports a person who is still making the decisions rather than replacing one who is not.

The two ironies

The paper opens with two dictionary definitions. An irony is a combination of circumstances whose result is the direct opposite of what might be expected. A paradox is a statement that seems absurd but may be well founded. Both words are then put to work. The classic aim of automation, she says, is to replace human control, planning and problem solving with automatic devices and computers. Yet even highly automated systems need people for supervision, adjustment, maintenance and improvement, so automated systems are still human-machine systems, and the more advanced the control system, the more the human operator's contribution may matter. That last claim, which most readers remember as her conclusion, is in fact her opening.

The ironies proper are two. The designer's view of the operator may be that the operator is unreliable and inefficient and should be eliminated from the system. The first irony is that designer errors can be a major source of operating problems. Bainbridge notes drily that the people who have collected data on this are reluctant to publish it. The second irony is that the designer who tries to eliminate the operator still leaves the operator the tasks the designer could not think how to automate. Those tasks are not a random remainder. They are what resisted specification, which means the rare, the abnormal and the poorly understood. The operator is left with an arbitrary collection of them, and, she observes, little thought is usually given to supporting them.

Figure 1.1 — The loop the two ironies close. Each pass removes practice from the tasks that remain, so the next failure is more likely and the premise looks better confirmed.

Tasks after automation

Two kinds of task are left. The operator may be expected to monitor that the automatic system is working correctly, and may be expected to take over when it is not. Taking over and stabilising the process needs manual control skill; diagnosing the fault, as a basis for shutting down or recovering, needs cognitive skill. She takes them in turn, and then returns to monitoring, because the points about takeover bear on it.

Manual control skill

Studies of process operators making a step change show a clear difference between the experienced and the inexperienced. The experienced operator makes the minimum number of actions and the process moves smoothly to the new level. The inexperienced operator's process oscillates around the target. Physical skill deteriorates when it is not used, and what goes first is the fine adjustment of gain and timing. So a formerly experienced operator who has spent months monitoring an automated process may now be an inexperienced one. If that operator takes over, the process may be set into oscillation; the operator will have to wait for feedback instead of acting open-loop, will find it hard to tell whether the feedback means the plant is faulty or the last action was misjudged, and will have to make further actions to correct the first ones, adding to the load. Takeover is needed when something is already wrong. The operator, she concludes, needs to be more skilled and less loaded than average at exactly the moment the system has made them less skilled and more loaded.

Figure 1.2 — What the operator has against what the moment asks for. Demand stays low while the automation holds, then steps up at takeover; skill has been falling the whole time. The dashed line is her remedy, hands-on control for a short period each shift, drawn as she describes it rather than as measured.

Cognitive skill

Manual skill is the visible loss. Bainbridge spends more of the paper on two invisible ones.

The first is long-term knowledge. An operator generates strategies for unusual situations from a stock of knowledge about how the process can behave. Two things go wrong with this for what she calls machine-minding operators. Retrieval from long-term memory depends on frequency of use; she invites the reader to consider any subject passed at school and not thought about since. And the knowledge develops only through use and feedback on whether it worked, so classroom instruction without practice produces people who neither understand much of it nor remember much of it. She adds a warning that was prescient in 1983: the automated systems of the day were being monitored by former manual operators and were riding on their skills, which the next generation of operators could not be expected to have.

The second is working storage: the operator's knowledge of the current state of the process. This is not raw data but the results of predictions and decisions the operator has already made, held ready for the next ones. It takes time to build. Manual operators, she notes, come into the control room a quarter to half an hour before their shift to get the feel of what the process is doing. An operator taking over from automatic control has had no such run-in. Whatever must be done quickly will be done on minimum information, and decisions that draw on wide knowledge of the plant state will have to wait until there has been time to check and think.

Monitoring

It might seem that an operator who is only asked to monitor, and to call the supervisor if something is wrong, escapes these problems. The supervisor, of course, has been deskilled in the same way. And the monitoring task has troubles of its own.

The first is that people cannot do it. From the vigilance studies that began with Mackworth in 1950, Bainbridge reports that even a highly motivated person cannot keep effective attention on a source of information on which very little happens for more than about half an hour. Monitoring for unlikely abnormalities is therefore not a human job and has to be given to an alarm system. That raises the question of who notices when the alarm system is not working. The classic device for forcing attention, making the operator keep a log, fails too: people can write numbers down without noticing what they are.

Figure 1.3 — Watching a process on which very little happens. The job assumes the dashed line; people deliver the solid one. Schematic, after the vigilance literature Bainbridge cites.

The second trouble she calls a more serious irony. The automatic system was put in because it does the job better than the operator, and the operator is now asked to monitor that it is doing so. In complex modes of operation the monitor would need to know what correct behaviour looks like, which takes special training or special displays. And if the decisions can be fully specified, the computer makes them faster, over more dimensions and by more precise criteria than a person can, so there is no way for the operator to check in real time that the rules are being followed. The operator can only judge, at some remove, whether the decisions look acceptable. But if the computer is deciding because human judgement was not adequate here, whose verdict is to be accepted when they disagree? The human monitor, she writes, has been given an impossible task.

Operator attitudes

A short section that later readers tend to skip. Bainbridge knew of one automated plant where management had to be present on the night shift, or the operators switched the process to manual. Skill matters to the person who has it. An operator who is skilled knows that they can take over if required; without that, the job is among the worst there are, very boring and very responsible, with no chance to acquire or keep the qualities the responsibility demands. Skill is also status, inside the plant and outside it, and a job reduced to monitoring is hard for the people in it to come to terms with. She cites a survey of control-room work in which coherent process information, high process complexity and high controllability went with low stress and good health, while fast process dynamics and actions that could not be made directly on the interface went with high stress and poor health. Stress produces errors. The human costs of the deskilled job come back as operating costs.

Approaches to solutions

Bainbridge restates the problem as a paradox: by automating the process, the operator is given a task that is only possible for someone in on-line control. Her proposals are offered for discussion and she stresses that the right answer depends on the plant, the speed of failure, the cost of shutdown and the operator.

Problem	What she proposes	The catch she names
A person cannot watch for rare events	Alarms, alarm analysis, if necessary alarms on alarms; displays that show target values	Too many flashing lights confuse; if the computer can generate the target values it could do the monitoring itself
Automatic control hides failure by controlling against it	The automatics should monitor unusual variable movement, and should fail obviously	Graceful degradation, prized in humans, is the wrong goal for a machine that is being watched
The operator cannot follow the computer's decisions	Have the computer decide by methods, and at a rate, the operator can follow	Less efficient technically; and adding a computer aid can make performance worse when the operator decides anyway and checking adds load
No picture of the current state at takeover	Shut down automatically when that is cheap; where the process must be stabilised, buy time with overlearned manual responses	Very fast failures need reliable automatic response whatever it costs, and if that cannot be had the process should not be built
Manual skill decays	Hands-on control for a short period in each shift	If that is laughable, then simulator practice, at high fidelity where fast reactions matter
Knowledge decays and simulators only rehearse known faults	Train general strategies, and practise solving problems within known information	Unknown faults cannot be simulated; it is ironic to train operators to follow instructions and then put them in the system to provide intelligence

The section ends with what she calls perhaps the final irony: it is the most successful automated systems, those with the rarest need for manual intervention, that may need the greatest investment in operator training.

Human-computer collaboration

The third section opens with the paper's most quoted sentence.

By taking away the easy parts of his task, automation can make the difficult parts of the human operator's task more difficult.

The Fitts-list approach, assigning to person and machine the tasks each does best, is no longer enough, because it says nothing about how the two are to be integrated or how the person's skill and motivation are to be kept up. And people will stay in these systems regardless, because automating every mode is not always worth the product, and because the public will not accept high-risk systems with no human in them. So the question becomes how the computer can intervene in human decision-making rather than the reverse.

She reviews four forms. Instructions and advice: giving the operator instructions is pointless if the operator is only a transducer, since the computer could drive a more reliable one; and an operator who follows advice reacts more slowly and less coherently than one who generates the sequence, and gets no practice in being intelligent. Mitigating error: interlocks and checks are best placed on the effects of actions rather than on the actions, because that makes no assumption about strategy, and people who get feedback within seconds correct their own errors, while those setting up automatics without feedback make the same errors and never see them. Software displays: a display tuned to normal operation can camouflage the development of abnormal conditions; a display so compatible with the task that the operator need not think to take it in may leave the operator knowing less about the process, since memory follows processing for meaning; and switching displays under time pressure confuses. Relieving work-load: aiding helps most at high loads, and a person must always know which tasks the computer is handling and how, or the system has the problems of a team with no clear allocation of responsibility.

Conclusion

The paper's own conclusion is two sentences long. People working without time pressure are impressive problem solvers; the difficulty is that they are less effective under it. Automating does not necessarily remove the difficulties, and resolving them may require more technological ingenuity than classic automation did.

Since 1983

The paper is a review. Its evidence is a sentence each from some thirty studies, and its authority now rests on how often its predictions have been confirmed since, in aviation, nuclear operations and medicine, rather than on anything measured inside it. A reader who wants the numbers has to go to Mackworth and to the skill-retention literature directly.

The remedies are managerial and she does not pretend otherwise. Hands-on control each shift means running a plant worse on purpose so that people stay able, and she anticipates that the suggestion will be found laughable. In most industries it has been; aviation, with its mandated manual flying, is the exception that shows the difficulty was never one of understanding.

The setting is a single operator, a physical plant and real time. Where the automation produces an artefact that can be inspected at leisure rather than a state that must be caught as it happens, the takeover argument loosens: there is no moment of takeover, only a decision about how closely to look. The skill-decay and working-storage arguments survive that move intact. The vigilance argument changes shape into a question about attention across many artefacts rather than one process.

The half hour is a rule of thumb from laboratory watch tasks, not a constant, and she uses it as an order of magnitude.

Finally, the paper assumes that the skill the rare case demands is the one the routine work rehearsed. Often it is. Sometimes the rare case needs a different skill altogether, and no schedule of practice on the routine would have supplied it. Her own section on training general strategies rather than specific responses is the nearest she comes to saying so.

Claims drawn from this chapter · 9

opinionTasks after automation · Monitoring

A human monitor cannot check decisions made faster than she can follow

Bainbridge argues that when a decision can be fully specified, the computer makes it faster and by finer criteria than a person can. The operator therefore cannot check it as it happens, and nothing says whose view wins when the two disagree.

Becauseshe argues from the structure of the arrangement rather than from any measurement that the monitor was given an impossible task

LimitsIt is an argument, not a finding; the paper offers no study in which monitors were shown to be unable to adjudicate. Where the machine's decision can be re-derived offline, the real-time objection does not apply.

PracticeBefore appointing a human check on an automated decision, write down the criterion by which the check would overrule it. If you cannot, the sign-off is a formality.

After AIThe mismatch is now routine: a model produces a page of reasoning in a second and the reviewer is asked to say whether it is right. Bansal and Buçinca measured what Bainbridge predicted, and found the sign-off satisfied in form and empty in substance.

evidenceTasks after automation · Manual control skill

An operator who only watches loses the skill to take over

Bainbridge says the routine work is what keeps a manual skill alive. An operator who has only monitored an automated system for months performs like a beginner when they have to take control.

Becausestep-change studies separate the experienced operator from the inexperienced one, and the skill that separates them is the one monitoring stops exercising

LimitsThe paper is a review; the step-change studies are cited a sentence each and the numbers have to be fetched from the skill-retention literature directly. It also assumes the skill the rare case demands is the one the routine work rehearsed, which is often but not always true.

PracticeKeep a share of the delegated work in your own hands on a fixed schedule, not because the output needs you but because the skill does.

After AIThe delegate is now a model that takes the routine work in every domain at once, so the decay is no longer confined to one plant and one operator. What is new is that there is usually no moment of takeover to expose it, only a decision about how closely to read.

opinionThe two ironies

Automation leaves people the jobs nobody could write down

Bainbridge argues that the tasks left to a human after automation are not a random leftover. They are exactly the ones nobody could write a specification for: the rare, the abnormal and the poorly understood.

Becausethe second irony is argued from what designers choose to automate rather than from a survey of what operators were left with

LimitsIt is a claim about designers' intentions, not a measured remainder, and it assumes the automation boundary is drawn by specifiability rather than by cost or convenience.

PracticeKeep an explicit list of what stays human, derived from what you could not specify, and treat every item on it as needing more support rather than less.

After AIA model does not need a specification to take a task, so the boundary is now drawn by what the model happens to do well, and the remainder is no longer reliably the hard part. What is left may be the part nobody thought to check.

contestedOperator attitudes

Bainbridge says watching a machine is stressful and the stress causes errors

Bainbridge argues that cutting an operator's job down to watching an automated system removes the control and the standing that made the responsibility bearable. The stress that follows is itself a cause of operating errors.

Becauseit rests on one plant where management had to be present on the night shift and one control-room survey linking controllability to low stress and good health

LimitsTwo observations, one of them anecdotal, carrying a causal claim about stress and error. The survey is cross-sectional and cannot separate controllability from everything else that differs between control rooms.

PracticeCount the human cost of a monitoring role as an operating cost. If the job has responsibility without control, expect errors from that alone and change the job, not the person.

After AIThe review seat is the deskilled job: full accountability for the output, no hand on the work that produced it. Self-determination theory and the job-crafting literature both predict the resentment Bainbridge only observed.

evidenceTasks after automation · Monitoring

Nobody can watch a quiet display for more than half an hour

Bainbridge argues that a person cannot hold useful attention on a display where almost nothing happens for longer than about half an hour. Watching for rare faults is therefore a job for an alarm, not for a human.

Becausethe vigilance literature from Mackworth 1950 puts a measured ceiling on how long attention holds on a quiet source

LimitsThe half hour is a rule of thumb from laboratory watch tasks and Bainbridge uses it as an order of magnitude rather than a constant. Where the output is an artefact that can be inspected at leisure the argument changes shape into a question about attention spread across many artefacts.

PracticeStop writing review duties that consist of watching for something that almost never happens; build the alarm instead, and then ask who watches the alarm.

After AIReviewing model output is the vigilance task at scale: long stretches in which the suggestion is fine, and the rare wrong one arriving with the same surface as the rest. The log that was supposed to force attention is now the diff nobody reads.

contestedThe two ironies

Removing the operator leaves the designer's mistakes with nobody to catch them

Bainbridge says designers' mistakes are a major cause of problems in running a plant. Automating a job to get rid of the unreliable human leaves those mistakes in place, with nobody left on duty to catch them.

Becauseshe asserts it from studies whose collectors she says are reluctant to publish, so the reader is given the conclusion without the data

LimitsThe evidence is second-hand and unpublished by her own account, which is why the claim stays contested; Norman reaches the same place from accident statistics and qualifies it himself for drunk and sleep-deprived operators.

PracticeWhen you remove a person from a loop, name what they were catching and say who catches it now. Treat the automation's own defects as a failure mode with an owner.

After AIThe designer's errors are now the training data's, the objective's and the scorer's, and they are distributed across every use at once rather than sitting in one plant. Nobody is on duty to catch them because the duty was never assigned.

evidenceTasks after automation · Cognitive skill

Skills fade when unused and grow only through practice with feedback

Bainbridge argues that what you know fades when you stop using it, because recall depends on how often you use it, and it grows only through use plus feedback. Teaching an operator without letting them practise leaves someone who neither understands nor remembers.

Becauseshe rests it on the memory literature on retrieval frequency, and offers the reader a check they can run on any school subject they have not thought about since

LimitsThe supporting studies are cited rather than reported, and the claim is about long-term memory in general rather than about process knowledge specifically.

PracticeDo not count instruction as preparation. If someone is expected to handle a case, they have to have handled cases and been told how it went.

After AIHer warning that the automated plants of 1983 were riding on the skills of former manual operators reads now as a description of every team whose seniors learned the work before the model arrived. The next generation is the one she said could not be expected to have it.

evidenceTasks after automation · Cognitive skill

Someone taking over from automatic control starts without a mental picture

Bainbridge argues that an operator's running picture of what a plant is doing is built up by watching it over time. Manual operators arrive a quarter to half an hour early to build it; someone taking over from automatic control has had none of that.

Becausemanual operators are observed arriving a quarter to half an hour before a shift to get the feel of the process, and an operator taking over has had no such run-in

LimitsThe observation is of process-plant practice rather than a controlled study of working storage, and the quarter to half an hour is her report of what operators do, not a measured build time.

PracticeBudget the run-in. Anything you intend to take over you have to have been following, and if you have not been following it, do the cheap thing first and think before acting.

After AIReading a model's output cold is exactly this position: the decisions that built it were made without you, and the state you would need to judge them was never assembled. The half hour of run-in has no equivalent in a chat window.

opinionApproaches to solutions

The more reliable an automatic system the more operator training it needs

Bainbridge argues that the automated systems which almost never need a person to step in are the ones whose operators need the most training. The rarer the manual case, the more practice it takes to stay ready for it.

Becauseshe calls it the final irony and derives it from the argument rather than from any training study

LimitsIt is a deduction, not a measurement, and it runs against the cost logic of every organisation that buys automation to reduce headcount and training. Aviation is the one industry that has acted on it.

PracticeSet the training budget from how rarely the manual case arises, not from how often. The better the tool, the more you have to spend keeping people able to work without it.

After AIThe more reliable a model becomes, the more expensive it gets to keep anyone able to catch the case it misses — and the harder it is to justify that expense to anyone looking at the success rate.