Automation Reliance

General Description

Most automated systems involve human operators in a supervisory or monitoring role. In this role, operators are often in the position to decide when and how to use the automation. The decision to use or not use automation is one of the most important an operator can make, particularly in time-critical situations. An operator’s decision to rely on automation depends on trust and self-confidence (Lee & Moray, 1992; Muir, 1988). For instance, with higher self-confidence in one’s ability to perform a task than trust in automation, an operator is likely to perform the task manually, whereas for a higher trust in automation than in manual performance, an operator is likely to rely on the automation. This relationship is mediated by other factors, including automation reliability, operator’s level of workload and the level of risk associated with the particular situation (see Riley, 1989, 1996 for a description of how these factor interact). Automation use decisions also can depend upon different system management strategies (see Lee, 1992), and operator attitudes (Singh, Molloy, & Parasuraman, 1993a, 1993b). With the decision to use automation, comes the ability to use the automation inappropriately.

Factors that lead to inappropriate reliance

Inappropriate reliance occurs when people rely on automation that performs poorly or when they do not use automation that is superior to manual control. Specifically, misuse refers to those instances when operators overrely on automation, using it when they should not or failing to monitor it effectively, and disuse to those where an operator neglects or underutilizes the automation, turning off or ignoring automated alarms or safety systems (Parasuraman & Riley, 1997). Another distinction for how operators use automation is reliance and compliance (Meyer, 2001). Reliance refers to when an operator assumes the system is in a safe state because the automated system has not delivered a warning and/or it appears to be operating within a normal range. Compliance refers to when an operator responds as if there was a system failure or abnormality when the automated system indicates through a warning or command of the possibility of such non-routine behavior.

Many studies have assessed the factors that contribute to over- and under- reliance (Dzindolet, Pierce, Beck, & Dawe, 2000; Dzindolet, Pierce, Beck, & Dawe, 2002; Dzindolet, Pierce, Beck, Dawe, & Anderson, 2001; Lee, See, Crittendon, Marquard, & Folmer, in review; Molloy & Parasuraman, 1996; Parasuraman, Molloy, & Singh, 1993; Singh, Molloy, & Parasuraman, 1992a, 1992b). Over-reliance is attributed to factors of workload, automation reliability and consistency, and the saliency of automation state indicators (Parasuraman & Riley, 1997). Under-reliance is attributed to false alarms (Parasuraman & Riley, 1997), inadequate feedback about the automation’s actions and intentions (Norman, 1990), automation failure (Muir, 1989), and a perceived disadvantage of automation due to cognitive overhead (Kirlik, 1993).

Over-reliance
Over-reliance on automation or automation bias (Mosier, Skitka, Heers, Burdick, 1998) represents an aspect of misuse that can result from forms of human error such as decision biases and failures of monitoring. Biases that involve reaching decisions under uncertainty stem from use of decision heuristics (Tversky & Kahneman, 1984), which are used to reduce the cognitive effort that is required to solve a problem (Wickens, 1992). Although useful, these heuristics can lead to poor decision performance. Decision support aids are an example of an automated system that may reinforce the human tendency to use heuristics (Mosier & Skitka, 1996). Reliance on automation as a heuristic is at times an effective strategy but overreliance, as with decision heuristics, can lead to errors. For example, automation bias may result in errors of omission in which an operator fails to detect events not detected by the automation. Errors of commission occur when an operator incorrectly agrees with erroneous detection of events by the automation (Skitka, Mosier, & Burdick, 2000). Reliance on automation decision can also lead operators to be less attentive to contradictory sources of evidence. Mosier, Heers, Skitka, & Burdick (1996) found that pilots often did not use disconfirming evidence available from cockpit displays when a conflict arose between expected and actual automation performance. In addition to decision biases, operators may not adequately monitor the inputs to automated systems to fully understand its behavior should the automation malfunction or fail. Overreliance is an important contributor to poor monitoring. Mosier et al. (1994) reported that 77% of ASRS incidents that were suspected to result from overreliance likely involved a failure in monitoring. The presence of other manual tasks also influences monitoring performance, being more likely to degrade monitoring of automation and detection of failures when operators are required to simultaneously perform other manual tasks (Parasuraman, Molloy, & Singh, 1993).

Over-reliance increases with increased automation reliability, consistency of reliability, and length of interaction with reliable automation (Lee et al., in review). In Dzindolet et al. (2001), participants over-relied on an automated detection aid to perform a visual detection task, as measured by a low detection rate of automation failures, when it had a high reliability level. Studies that varied the consistency of automation reliability (Parasuraman et al., 1993; Singh, Molloy, & Parasuraman, 1997), reported that participants relied more appropriately on the automation, detecting more automation failures, when the reliability varied compared to when the reliability remained constant across a trial. Finally, Molloy & Parasuraman (1996) reported that a 20-minute period of reliable automation prior to an automation failure resulted in a reduced detection rate than when the period of reliable automation was only 10 minutes.

Under-reliance
Under-reliance on automation is disuse of automation. This disuse can stem from mistrust and even dislike of a newly introduced automation system. However, once experience is gained with the automation and it is both reliable and accurate, operators will tend to trust the system. Those automated systems that have a propensity for false alarms however, such as the early design of the Ground Proximity Warning System (GPWS), are not trusted and therefore not used (Parasuraman & Riley, 1997). Two factors which impact the system false alarm rate, and consequently an operator’s trust in the automated system, are the values of the decision criterion and the base rate of the hazardous condition (Parasuraman & Riley, 1997). The decision threshold is set based on the cost of a missed signal versus that of a false alarm; in general, a low false alarm rate is necessary for acceptance of a warning system by human operators. Certain false alarms can be informative and beneficial to operators however such as “likelihood alarms”, that are used to indicate several possible levels of a dangerous situation ranging from very likely to very certain (Sorkin, Kantowitz, & Kantowitz, 1988), and non-useful alarms, which are delivered in situations judged hazardous by the warning algorithm but not by the operator, though these alarms are associated with a meaningful context that may help the operators to understand the reason for the alarm (Lees & Lee, in review). The base rate of the hazardous condition refers to the frequency with which hazardous events occur and the need to consider this rate in setting the decision threshold (Parasuraman, Hancock, & Olofinboba, 1997).

Under-reliance increases when the behavior of the automation is unclear and operator attitudes are negatively biased towards automation. Inadequate feedback about the automation’s actions and intentions causes humans to ignore warnings and mis-assign behavior of the automation (Norman, 1990), degrading trust and use in the automated system. Failures of automation also contribute to under-use (Muir, 1989), particularly when the failure is of a high magnitude (Muir & Moray, 1996; Moray, Inagaki, & Itoh, 2000). Importantly though, allocation of function is determined both by trust and level of self-confidence in performing a particular task manually (Lee & Moray, 1992). Operators are prone to disuse automation and do the task manually if they perceive that the advantage offered by the automation is not enough to overcome the cognitive overhead involved (Kirlik, 1993) or if they have a bias in expecting the automated aids to perform at nearly perfect rates where the disuse stems from an increased attention to failure events or errors and an underestimation of the true reliability of the system (Dzindolet et al., 2001b).

Eutactic monitoring behavior

Inappropriate reliance, as defined by over- and under- reliance on automation relative to the automation’s capabilities, while it may reflect poorly calibrated trust, automation bias, and complacency may also reflect eutactic behavior (Moray & Inagaki, 1999; Moray, 2003). If so, the particular use of automation may be appropriate given the consequent costs and benefits to operator state and system performance. For example, an operator that infrequently monitors an automated system that only infrequently fails with modest cost to system performance (arguably) samples at a eutactic rate. To define appropriate reliance, the sampling interval must be defined. Monitoring behavior would thus be classified as either complacent (i.e., operators monitored a source less often than optimal), skeptical (e.g., operators oversampled and thus expended unnecessary energy observing a particular system), or eutactic (e.g., operators sampled at the optimal rate respective to the failure rate). Failure detections and response times should be considered in the context of the specified sampling rate as appropriately, if a fault probability is low then the probability of a prompt detection is reduced.

Key Terms

Reliance
Automation
Trust in automation

References

Dzindolet, M.T., Pierce, L.G., Beck, H.P. and Dawe, L.A. (2002). The perceived utility
of human and automated aids in a visual detection task. Human Factors, 44 (1), 79-94.

Dzindolet, M.T., Pierce, L.G., Beck, H.P., Dawe, L.A. and Anderson, B.W. (2001).
Predicting misuse and disuse of combat identification systems. Military Psychology, 13
(3), 147-164.

Lee, J.D. and Moray, N. (1992). Trust, control strategies and allocation of function in
human machine systems. Ergonomics, 22(6), 671-691.

Lee, J.D., & See, K.A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50-80.

Muir, B.M. and Moray, N. (1996). Trust in automation: Part 2, Experimental studies of
trust and human intervention in a process control simulation. Ergonomics, 39, 429-460.

Parasuraman, R. and Riley, V. (1997). Human and automation: Use, misuse, disuse,
abuse. Human Factors, 39(2), 230-253.

Cross References

CSL lab: Automation reliance research focus

Georgia Institute of Technology Human Factors & Aging Lab: Perceived consequence on automation reliance