This article makes three points. First, errors happen. This may seem trivial on the surface, but it is important to recognize that no matter how good a system or design is, people will always make mistakes. Second, systems can encourage errors. Although errors are often attributed to the action of an individual, there is often a set of external forces and preceding events that leads up to the error. Often these forces and events are difficult to anticipate; yet the reaction to errors is often to punish the individual, rather than examine the error-prone system as a whole.

This system includes any group of people (which in medical applications, for example, might include physicians, nurses, and programmers), objects (such as signs, operating-room equipment, computer systems), and knowledge (such as medical training, hospital procedures) that comes to play in a particular process. Third, a basic understanding of the kinds of errors humans make can help us design better systems. Since we can expect users to make errors, we can focus on error tolerance and error recovery as well as error prevention when we design computer systems. In fact, many of the guidelines and design heuristics employed in user interface design have their roots in cognitive science and the analysis of human error.

The Therac-25 Incident (1986)
The Therac-25 was a million-dollar radiation machine designed to precisely aim a beam of radiation at a patient in order to treat tumors or cancerous growths. Patients were often recovering from operations that had removed the bulk of a tumor, and underwent these radiation treatments to remove what was left.

The Therac-25 was a high-energy radiation machine, but its radiation treatments usually involved many low-energy dosages across successive treatment sessions. The machine was controlled through a computer located in another room, as are most radiation therapy controls, in order to protect the technicians from unnecessary exposure.

There were two basic modes in which the Therac-25 could function. The first was the low-energy mode mentioned above, in which an electron beam of about 200 rads was aimed at the patient and sent off in a short burst. The second mode was an x-ray mode, which used the full 25 million electron volt capacity of the machine. When the machine was switched into this mode, a “beam flattener” would get inserted between the beam source and the patient; as the beam passed through the flattener, it was transformed into an x-ray which would irradiate tumors and the like (see Figure 1).

To switch to “electron mode,” the technician typed “e” at the computer terminal. To switch to “x-ray mode,” the technician typed “x” at the computer terminal. Sounds simple, right?

In 1986, Ray Cox, a Texas oil worker, went in for his usual radiation treatment for a tumor that had been removed from his left shoulder. He had been here eight times before, so this was business as usual. They got him set up on the table, and the technician went down the hall to start the treatment. The technician sat down at the terminal, and hit “x” to start the process. She immediately realized she had made a mistake, since she needed to treat Ray with the electron beam, and not the x-ray beam. She hit the “Up” arrow, hit “e” for electron beam, and hit “Enter,” then finished by hitting “Enter” several more times.

The total time for this interaction was under eight seconds.

The system presented the technician with a “beam ready” prompt, indicating it was ready to proceed; she hit “b” to turn the beam therapy on. She was surprised when the system gave her the following error message:

Malfunction 54

She was not familiar with this particular message, but these kinds of errors usually meant the treatment had not proceeded. She hit “p” to proceed with the treatment, but received the same result: the system printed an error message and seemed to shut down.

Meanwhile, back in the treatment room, Ray was feeling burning, stabbing pains on his back. None of the previous treatments had been like this. Although he cried out several times, asking (first jokingly) whether the system was configured correctly, no one came to check on him. Finally, after the second painful burst, he pulled himself off the table, and walked to the nurses’ station...

The problem was this: when the particular sequence of commands was executed fast enough, the machine withdrew the beam flattener from the beam path, but left the power setting on maximum! Although the user interface told the operator the system was in electron beam mode, it was actually in a hybrid mode. As a result, the system was delivering a radiation blast of 25,000 rads with 25 million electron volts, more than 125 times the normal dose.

Ray Cox’s health deteriorated rapidly from radiation burns and other complications from the treatment overdose. He kept in good humor about his condition, joking that “Captain Kirk forgot to put his machine on stun.” He died five months later.

Latent Errors: Accidents Waiting to Happen
There were a number of checks in the Therac-25 case that might have prevented the radiation overdose. There was an audio intercom system between the treatment room and the control room, but it was broken. There was a video hookup between the two, but the monitor was disconnected on that particular day. Obviously, the system itself noticed the error, but the message to the user was not evocative enough to accurately reflect the condition or gravity of the error. Thus, although the technician noticed she had made a mistake, she assumed the treatment had not been given—because that is what these and similar error messages usually meant.

This kind of progression of multiple errors in procedures, performance, and equipment is seen in many accidents in complex systems. In the 1986 Chernobyl reactor accident, for example, most of the safety systems had been turned off because the staff was running some test experiments. Similar situations occurred at Three Mile Island (1979) and Bhopal (1984). In all of these cases a particular context and sequence of events, sometimes spanning months (the intercom in the Therac-25 case had been broken for awhile), led up to the accident. The net effect is that the system virtually elicits the error from the user. Such errors are called latent errors.

Latent errors demonstrate the difficulty in making uniform attributions of error causality. In the Therac-25 example, the machine malfunctioned (software error) and printed an uninformative error message (design error)—and the operator proceeded incorrectly (user error). All of these events led up to the “accident.”

Since user errors are inevitable, system designers should organize their work around them. They should think not just about users and their tasks and goals, but also about what errors users might make. To minimize future accidents, we can synthesize system design principles based on the study of Human Error, principles that help us design systems that are easy to use and make it difficult for users to make mistakes. Furthermore, if we admit that we cannot predict and prevent all user errors, we must design error-tolerant systems that behave robustly when errors do occur, rather than if they occur. As it turns out, there are certain broad themes in the study of human error that can be generalized and used as design guidelines. This awareness, coupled with a basic understanding of specific types of errors that can occur, can help designers build better systems.

In general, adopting this design philosophy means:

• Learning about types of user errors before designing
• Using error knowledge and design rules of thumb during design
• Evaluating your design by observing errors when they occur

Slips: Errors of Action
A slip is an action not in accord with your intentions: a good plan but poor execution, sort of like fumbling the ball on a good play. Since they are part of automatic, unconscious actions, slips are typically unintended acts due to a break in the routine. Examples of slip error mechanisms include capture, description, associative action, and loss of action.

Capture slips occur when you automatically do something that you did not mean to, usually because you fell into a pattern you perform frequently. For example, if you dial a particular telephone number often, your fingers get used to hitting that particular sequence of telephone buttons. This is common in data entry tasks, especially if users are entering codes: if certain codes are repeated frequently, the pattern for a given code may “take over” when trying to type a similar one in a hurry.

Description slips occur when you have not correctly told yourself what you want to do, and therefore your intention is ambiguous or incompletely specified. Because your intended action is similar to other actions you perform frequently, you perform the right action but on the wrong object. Examples include sticking the salad in the oven and the cake in the refrigerator, or accidentally sending a love letter to your manager and your project progress report to your main squeeze because you accidentally switched email addresses. In a certain sense, the actions are correct, but their parameters are wrong; contrast this with capture slips, where the actual action was wrong due to force of habit.

Associative-activation slips occur when your brain makes a faulty connection or mental association between two ideas, often when one is an external stimulus that typically provokes a certain action. A simple example is answering the phone when you hear the doorbell, which might happen because a person is distracted or preoccupied. Associative activation slips are more likely to occur in workplaces where there are many simultaneous sounds and signals that users must attend to. For example, a hospital Intensive Care Unit may have several machines with audible alarms: an associative activation error could occur if a nurse reacts to Alarm A as if it is Alarm B going off.

Loss-of-activation slips occur when you lose track of why you are engaged in a particular activity (you lose the “activation” of the activity). This is essentially a temporary memory loss, often due to interruption such as someone handing you something or asking a question. Certain readers may recognize this as the “Now why did I get up to come into this room in the first place?” phenomenon.

Mistakes: Errors of Intention
A mistake is a planning failure: actions go as planned, but the original plan was bad. These are errors of judgment, inference, and/or some other mental process that results in an incorrect intention, incorrect choice of criterion, or incorrect value judgment. Mistakes are the real challenges in the analysis of human error. Whereas slips can often be prevented through checks built into equipment and tools (a software check can prevent a bank teller from storing an incorrectly-formatted account number; an N₂O ratio-limiter can prevent an anesthesiologist from accidentally administering a dangerous combination of gases), mistakes stem from cognitive breakdowns—which are often influenced by a number of external system factors. Mistakes, therefore, are even harder to predict and prevent. Mistakes include rule-based errors, such as coning of attention and reversion under stress, and knowledge-based errors.

Rule-based errors occur when the wrong rule is chosen due to the misperception of the situation, or the misapplication of a rule. One example would be selecting the wrong medication for a patient: the medication may be correctly ordered and administered, but it is the wrong medication for that particular patient. Misperceptions that lead to rule-based errors can stem from a number of sources, including external factors such as an unclear or partially hidden readout on displays, confusing patient charts or lab result displays, and so forth.

Stress can often produce rule-based errors. One stress-induced phenomenon is called coning of attention, which refers to the way people concentrate on a single source of information in stressful situations, often to the point of ignoring other potentially useful sources of information. Imagine, for example, concentrating on opening an airplane door for an emergency exit in a crash situation (the correct official procedure), when there is a huge hole in the bulkhead nearby that people could use as an exit. Another phenomenon is called reversion under stress, where people switch to using older, more comfortable rules in stressful situations, even if they have learned new, more “correct” ones recently.

Knowledge-based errors are the most complex of the errors to analyze. They typically occur, as one might expect, from a lack of or misapplication of knowledge, resulting in an incorrect intention on the part of the user. Common examples of knowledge-based errors stem from various cognitive biases that come to play when people make decisions. For example, users may show an availability bias when choosing a particular course of action because it is the one that comes most readily to mind. People tend to judge the likelihood of things happening based on whatever specific examples they can call to mind. For example, we tend to overestimate the loss of life from airplane disasters than from illness because the disasters make a stronger impact on our memories. Similarly, programmers may be more likely to attribute bugs to memory leaks simply because the last bad bug they worked on was diagnosed as such. Users who fixate on a particular course of action, actively pursuing supporting evidence or ignoring contradictory evidence, suffer from confirmation and overconfidence biases. This leads to faulty conclusions about a situation, and therefore faulty plans to accomplish the given task(s).

This taxonomy is by no means complete. Other taxonomies present different classifications: phenomenological errors (errors of omission or substitution), hypothetical internal processes (capture, overload, or bad decisions), neuro-psychological mechanisms (forgetfulness, stress, attention), and external process influence (poor equipment design).

Better Design Through Human Error
In addition to learning about errors in general, we can learn from errors in other fields. Some fields have embraced the study of human error as a fundamental way of doing business.

Anesthesiology is the one area in medicine that has shown a concerted effort to address issues of human error and performance. Part of this focus is probably due to the clear potential for death or brain death during surgery. These deaths are at about 1 in 200,000 now, down from 1 in 10,000 a decade ago; many believe this is partially due to improvements in ergonomics, operating room layout, monitoring interfaces, and other system-wide refinements that were suggested by the study of error and performance in anesthesia.

Aviation, too, has a long history of studying and tracking errors and accidents to learn from them. Because of the assumption that errors occur, crosscheck and verification systems are put in place to attempt to catch errors before they cause harm. There are several institutions that monitor the industry and concern themselves with minimizing the occurrence of errors in flying. The Federal Aviation Administration (FAA) regulates all aspects of flying, including the educational requirements for pilot training and the correct procedures for flying aircraft. The National Transportation Safety Board (NTSB) investigates all accidents related to commercial aviation, and attempts to determine the specific cause for the crash and whether anything can be done to avoid a similar accident in the future. These reports are included in a number of commonly available publications, and some accident reports are even available on the Internet.

The Design Process
Design is not a process that lends itself to template solutions: there is no fixed set of rules that will guarantee success. But they can provide some steering aids to help us get the job done. In this section we’ll look at some specific suggestions for how to integrate an appreciation of human error into the design of interactive systems.

Increase your error awareness. Simply being aware of the inevitability of errors can affect your design. When you talk with users about their tasks, tools, and workplace, discuss errors that occur in that context. Ideally, you want to keep a system-wide “lookout” for errors, trying to find and identify errors on a routine basis. However, do this without violating any rules (spoken or unspoken) about accepted behavior. Physicians, for example, are typically resistant to attempts to monitor or document their performance and skills, for fear of malpractice claims or that their actions will be misinterpreted out of context. In such situations you may have to rely on self-reported user feedback in order to gather data about errors.

Look for “big picture” errors. When discussing errors, keep an eye out for larger system problems, since they may be the ones that really need fixing. Remember that user errors are rarely single, independent events, but are instead part of a larger sequence. When you do find a problem, try to reason back from the immediate cause to see if you can redesign and reorganize the process to avoid such problems in the future. In particular, your analysis of users’ tasks, workflow, and information needs may point out problems in the way work is done that exist before the computer enters the picture. You should consider whether the computer systems can help solve these problems, or whether they need to be solved at a different level (e.g., administrative, regional, policy, etc.). In other words: “Don’t swat at mosquitoes—drain the swamp!”

Capture errors imply need for better feedback. In the Therac-25 incident, the technician dismissed the error dialog under the assumption that it was essentially analogous to the other error dialogs she had seen (and so the more familiar of two sequences took over). If the system had given better feedback as to its internal state (i.e., how much radiation had just been administered), the technician (we hope) would not have dismissed the error so readily.

Description errors suggest the need for better system configuration. These errors occur because an action was not “specified” accurately enough, and a different but similar action took place (as in the cake in the refrigerator and salad in the oven example). These errors are common when operations involve throwing switches, pushing buttons, etc., and when these operations are similar. For example, knobs for adjusting different display scales on an EKG machine might be confused if they look the same and are placed next to each other. One solution would be to differentiate between the control functions, using symbols or more diverse control placement.

The inevitability of errors implies a need for reversible actions. If you take it for granted that even expert users will make mistakes, your system should be forgiving when they are made. In particular, user actions should be reversible. On familiar desktop systems, this is usually supported via the “Undo” command. However, these features are largely confined to personal computer applications; many vastly larger and more expensive systems have no equivalent.

Loss-of-activation problems can be avoided through visible affordances and feedback. Affordances are objects like menus, buttons, and the mouse that offer users the opportunity to interact with the system. Feedback informs the user of the results of that interaction. For example, you suffer a loss-of-activation slip if you are interrupted while going to fetch something, and then cannot remember what you wanted to get. If you had a checklist in your hand that listed things you needed to retrieve, you would be more likely to remember what you came to get. An online system could provide similar support by integrating a visible model of the workflow to help the user accomplish all steps of the task.

Improving the Design Process
As a final note, here are four specific recommendations on how knowledge of human errors may affect your design process.

1. Be willing to redesign. This is probably the biggest challenge to software designers, but it is central to good design. You must be willing to throw away some of your work in order to improve the overall system. In some cases, you may need to redesign the whole system; be willing to look beyond the immediate causes of the error and consider the larger influences at work.

2. Use errors analysis as part of your design analysis. You may already be doing this through User-Centered Design: usability studies give you feedback on your system design by demonstrating errors that users can make. But remember that although usability testing can point out some errors, it will not find them all. Try to prevent errors, but design your system to handle them when they do occur (e.g., forgiveness of user actions, redundancy in data storage, etc.).

3. Use simulators when possible. Aviation has benefited from the use of flight simulators to test new cockpit designs. Usability studies can provide a limited form of simulation, as they attempt to model the user’s work environment using a new tool (say, the prototype for a new user-interface). And automated testing tools can provide structured, rigorous testing of user interfaces and software components.

4. Build tracking features to facilitate retrospective error analysis. Software developers have a unique opportunity to automate data collection for subsequent error analysis. One of the great challenges in the analysis of human error is obtaining enough data to understand how an accident occurred. Computers have the ability of recording user actions automatically, much the way cockpit “black boxes” record all vital flight information. Consider building such tracking features into your software in order to analyze problems when they do occur. Note that if your software provides an “undo” functionality, you likely already have the basis for such user-action tracking in place.

Author’s Note: Netscape has adopted this approach in recent releases of their Navigator Web browser, which includes a “Quality Feedback Agent” that sends a bug report to Netscape when Navigator crashes. This is noteworthy in that (a) Netscape assumes that there will always be some bugs in shipped software and (b) they leverage Internet connectivity to send this information back to their Web site with a minimum of user intervention.

A bibliography for this article is located in the Webinfolink section of the magazine Web site. See below.

Figure 1
Back

Figure 2
Back