46Trust in Automation: Designing for Appropriate Reliance

John D. Lee and Katrina A. See, Human Factors 46(1), pp. 50–80, 2004

Linked to 32 chapters: supports 14 · contradicts 1 · mechanism 26 · same claim 5 · pairing 7

Lee and See open with two accidents and a refusal. A flight crew, trusting the autopilot, did not take manual control while it flew their Airbus A320 into the ground. The crew of the cruise ship Royal Majesty let a malfunctioning navigation system carry the vessel off course for twenty-four hours before it ran aground. In the paper mills Shoshana Zuboff studied, operators rejected automated controllers that worked. Parasuraman and Riley had named the two failures: misuse, relying on automation where its assumptions do not hold, and disuse, refusing what it can do. Lee and See argue that both are the same failure measured from opposite sides. People respond to machines socially, and once automation is complex enough that no operator can hold a complete account of it in mind, reliance is guided not by analysis but by an attitude. That attitude is trust. Their review gathers what the organisational, sociological, interpersonal, psychological and neurological literatures have found about trust between people, and asks how much survives the move to a partner with no intentions of its own. What comes out is a definition, three bases on which trust can rest, three processes by which information about the machine is taken in, and a closed loop in which trust and reliance feed each other. The design conclusion is the first item on their list and the least intuitive: design for appropriate trust, not greater trust.

Trust and affect

Beliefs, attitudes, intentions, and behaviour

The definitions in circulation disagreed about the kind of thing trust is. Rotter called it an expectancy that another's word can be relied upon; Barber, an expectation of technically competent role performance; Mayer, Davis and Schoorman, a willingness to be vulnerable to another's actions irrespective of any ability to monitor them, which Lee and See note is the field's standard, with 203 citations in the Institute for Scientific Information database as of April 2003; Kramer, a state of perceived vulnerability. Belief, attitude, intention, behaviour: four different answers.

Ajzen and Fishbein's sequence sorts them. Beliefs are the information base, an attitude is an affective evaluation of beliefs, attitudes produce intentions, and intentions produce behaviour subject to what the environment permits. Trust is the attitude. Reliance is the behaviour. Keeping the two apart explains why trust guides reliance without determining it: an operator may trust the automation, intend to use it, and still lack the time to engage it, or find configuring it more work than it saves. Workload, perceived risk and self-confidence stand alongside trust at the same point in the chain. The definition follows: trust is the attitude that an agent will help achieve an individual's goals in a situation characterised by uncertainty and vulnerability. It is goal-oriented by construction, so trust is not a consideration at all where the trustor depends on the trustee for nothing.

Calibration, resolution, and specificity

Three properties describe how well trust matches the machine. Calibration is the correspondence between a person's trust and the automation's actual capabilities; overtrust leads to misuse, distrust to disuse. Resolution is how finely a judgment of trust discriminates between levels of capability: with poor resolution a wide range of capability maps onto a narrow range of trust, so large changes in what the machine can do register as small changes in what the operator feels. Specificity is what the trust attaches to — functional specificity, whether it is aimed at one mode or at the whole system, and temporal specificity, whether it tracks moment-to-moment fluctuation or only the long-run average.

The individual, organisational, and cultural context

Trust does not start from zero. Rotter's Interpersonal Trust Scale measures a stable disposition, with an internal consistency of .76 and a test-retest reliability of .56 across seven months. High-trust people are not more gullible and are no more intelligent; they are seen as more trustworthy, behave more truthfully, and predict other people's trustworthiness more accurately. Lee and See use this to explain an awkward result in their own field: on a complacency scale, high-complacency individuals detected more automation failures than low-complacency individuals under constant reliability, 53.4% against 18.7%. Disposition may govern not how much a person trusts but how well they adjust.

The organisational context supplies trust indirectly, through reputation, gossip and role: an engineer is trusted not for any demonstrated ability but for the education and regulation standing behind the title. Culture supplies norms. A questionnaire of 1136 Japanese and 501 American respondents found the Americans more trusting in general and more attentive to reputation, the Japanese placing more weight on committed personal relationships, which the authors read as social stability making trust less necessary; across countries, the level of social trust accounts for over 64% of the variance in Internet adoption. Within a country, groups differ by exposure: pilots, accustomed to automation, trusted and relied on it more than students did.

The basis of trust

Performance, process, and purpose

The information trust rests on can be sorted by how abstract the attribution is. Lee and Moray had named three levels, and the paper's Table 1 shows fourteen published lists of trust dimensions collapsing onto them. Performance is what the automation does: reliability, predictability, competence demonstrated at the operator's goals. Process is how it works: whether its algorithms suit the situation and can be understood, with openness part of the same dimension. Purpose is why it was built: the designer's intent, and whether the machine is being used inside the realm of that intent. In the interpersonal literature these are ability, integrity and benevolence; in Zuboff's operators meeting new technology they were trial-and-error experience, understanding, and a leap of faith.

The three are not independent. Observing performance supports inferences about process, and understanding the process supports inferences about intent; the reverse inferences run too. So trust can fail in two ways: poor correspondence, where the inferences do not match what is observed, and poor coherence, where the inferences at different levels contradict each other. A manager whose stated intentions disagree with his actions damages trust particularly badly. Trust resting on a single basis is volatile; trust resting on several is stable, which is why trust in automation sometimes looks fragile and sometimes immovable.

Analytic, analogical, and affective

Information becomes trust by three routes. The analytic route is the rational-choice account, an expected-value calculation accumulating over interactions, corresponding to Rasmussen's knowledge-based behaviour; Lee and See think it describes a normative level of trust rather than how trust develops, since expert decision makers seldom compare alternatives exhaustively. The analogical route is cheaper: trust assigned by category, rule, role and reputation, corresponding to rule-based behaviour. It is efficient and brittle. Where roles had been defined by a binding contract, trust fell sharply once the contracts were removed; and responses to warnings, which slow after false alarms, sped up again under gossip that understated the false-alarm rate.

The third route is affective, and the paper argues that is where trust finally lives. The neurological evidence is the strongest part of the case. Patients with lesions in the ventromedial prefrontal cortex keep their reasoning, working memory and declarative knowledge and lose the ability to decide; in a gambling task they responded to immediate prospects and ignored long-term consequences. Healthy participants produced a skin-conductance response to large losses and, more tellingly, an anticipatory response that steered them off the risky decks before they could say the decks were risky. Affect also travels through channels no display carries. Zuboff's process operator, describing what computerisation removed, said he used to "listen to sounds the boiler makes" and knew from the colour of the fire what adjustment it needed.

Generalising trust in people to trust in automation

Three differences limit the transfer. Automation lacks intentionality, which is what the purpose dimension is built from; the answer is that automation embodies the intentionality of its designers, and that people impute motives to it anyway as it acquires speech and other human characteristics. There is also no symmetry, because the machine is not aware of being trusted and does not adjust. In a semi-automatic pasteurisation plant where control of a pump could be delegated, delegation to what participants believed was another operator depended on how trustworthy they thought they themselves appeared, while delegation to the automation did not. Responsibility feels shared with a person and undivided with a machine.

Third, the order of development is not fixed. Muir predicted that trust in automation would follow the progression found in romantic relationships, from predictability through dependability to faith. Muir and Moray found the reverse: given a written description of the automation's purpose, operators started with high purpose-level trust and acquired a feel for dependability afterwards. Lee and See dissolve the contradiction by treating performance, process and purpose as levels of abstraction rather than stages of a relationship. Which basis comes first is decided by what the interface, the documentation and the training supply.

A dynamic model of trust and reliance

The paper's central diagram closes the loop. Trust guides reliance, and reliance generates the observations on which trust is revised. A system that is not trusted is not used, and a system that is not used yields little information, so trust cannot grow. That makes observability a design variable. Of the four types of automation Parasuraman, Sheridan and Wickens distinguish, information acquisition can be watched while it is being ignored, since an operator can compare a target cue against the raw data; action implementation cannot. A pump controller reveals nothing to an operator who has taken manual control, which may be why trust fails to recover fully when a system's reliability improves.

Because the loop has gain, small differences produce large ones. A minor difference in predisposition can tip one operator into engaging the automation and another into manual control, after which the first operator's trust climbs and the second's declines. Poor temporal specificity acts as a time delay in the loop, making reliance unstable and trust volatile. Faults have a shape in time: an acute fault, one that is repaired, drives trust down and lets it recover; a chronic fault drives it down until operators understand the deficiency and work around it. Neither effect is instantaneous. A time-series analysis showed a first-order differential equation accounting for the change, and the fitted equation explained between 60% and 86% of the variance in operators' reliance.

Trust is conditioned by an automated system's worst behaviour, and initial experience leaves a mark: trust is more resilient when reliability starts high and declines than when it starts low and improves. There is no reliability figure below which trust collapses — estimates of the drop-off range from 90% to 70% to 60% across studies — and the predictability of a fault matters as much as its size, a small fault with unpredictable results damaging trust more than a large fault of constant error. Operators will rely on automation they know to be faulty, if they know how it is faulty. Self-confidence supplies the other half of the comparison: high self-confidence with low trust produces manual control, and low self-confidence produces delegation. Errors in a vehicle navigation aid cost more trust among drivers in a familiar city than an unfamiliar one. And monitoring collapses under multiple tasks: operators responsible only for watching the automation detected almost all its failures and operators with other work did not, a result that survived repetition with experienced pilots and with rare faults, while ten minutes of manual control handed back by adaptive automation substantially improved subsequent detection.

Since the automation is usually perceived through a display, the content and format of that display decide what trust can be calibrated against, and the evidence there is uncomfortable. In an online survey of more than 1400 people, the credibility of a website depended heavily on what the authors called real-world feel: speed of response, a listed physical address, photographs of the organisation. A formal photograph of an author raised the trustworthiness of a research article and an informal one lowered it. Increasing image realism increased reliance on target cueing. Concrete, vivid, specific presentations move affect more than abstract data do, which means the devices that can calibrate trust can also manufacture it against the evidence.

Implications for creating trustable automation

Design errors, maintenance problems and unanticipated variability make completely reliable automation unachievable, so Lee and See aim at a different target. Trustworthy automation performs well. Trustable automation conveys what it can do, so that reliance adjusts to it. The two can conflict: the algorithms that make a system perform best may be the ones no operator can follow, and where joint performance depends on appropriate reliance, a simpler and less capable system can be worth more than a complex and better one.

One gap closes the section. As soon as several people share responsibility for a machine, the governing quantity becomes what the authors call meta trust: the trust one operator has that another operator's trust in the automation is appropriate. Nobody had studied it.

Since 2004

The paper became the standard reference for its subject and supplied the vocabulary the field still uses: calibration, resolution, specificity, and the triad of performance, process and purpose. Its central asymmetry — that the goal is matching, not maximising — has been absorbed well enough that "building trust" now sounds naive in human factors and remains the default phrase everywhere else.

Three lines of work have tested it. A 2011 meta-analysis of trust in human-robot interaction found the characteristics of the machine, and its performance above all, to be much the largest determinant of trust, with operator traits and environmental factors far behind; that vindicates the performance basis and leaves the long sections on individual and cultural context resting on the thin evidence the authors admitted they had. Hoff and Bashir's 2015 review reorganised the same literature into dispositional, situational and learned trust, a cleaner cut than this paper's. And the recommendation to reveal process became the transparency literature, where results are mixed: more explanation raises workload without reliably improving calibration, and operators given an algorithm's intermediate results still fail to pick out the cases where it is wrong.

Measurement remains the weak joint. The trust scales in common use, including the one Jian, Bisantz and Drury published in 2000, are self-report instruments administered after the fact, which sits badly with a construct the paper says is affective, fast and continuously updated. Most of the evidence in the review comes from microworlds run for an hour or two, and the loop the paper draws has its longest time constants in exactly the settings hardest to study.

Two predictions have aged conspicuously well. The first is that trust would become a consumer problem, in home automation, personal robots and cars; the partial-driving-automation incidents of the following two decades are calibration failures of precisely the kind the paper diagrams, with observability of the sort it identifies as the hardest. The second is the warning about synthetic conversational partners. Systems that speak fluently recruit the analogical and affective routes directly, borrowing a category membership they have not earned, so the trust they collect is set by surface features rather than by the record — the mechanism Lee and See described when the state of the art was a text-to-speech engine and a photograph on a web page.