Outcome bias
Claim assessed: We judge a decision by how it turned out, not by how well it was made.
Published · how we work
In short
Outcome bias means judging a decision by how it turned out, not by how well it was made with the information available at the time. Every real decision is made under uncertainty, so it is a kind of bet. A good bet can be lost, and a bad one can be won. When the outcome tells us something new about the decision, it is natural for it to count. The problem is when it counts even though the decision maker couldn’t have known more.
What it says
The original experiment. Baron and Hershey (1988) gave students descriptions of medical decisions. One example: a 55-year-old man has chest pain that stops him from working. A bypass operation would relieve the pain and raise his life expectancy from 65 to 70, but 8% of patients die during the operation. The physician decides to operate. In one version, the operation succeeds. In another, the patient dies. The students rated the quality of the decision on a scale from −3 to +3, and were explicitly asked to judge the decision and the thinking behind it, not the outcome. The same decision got, on average, 0.85 when the operation succeeded and −0.05 when the patient died.
Even when you know it shouldn’t. Of the last 8 students asked, 7 said the outcome shouldn’t count in the evaluation, and the effect appeared in them too. In another experiment, with 29 participants, the outcome also changed how competent the physician seemed for future decisions. The instructions said clearly that the physician hadn’t performed the operation and hadn’t known more than the participants. The effect also appeared with money gambles: a choice was rated better when the rejected option would have turned out badly.
The replication. Aiyer et al. (2023) repeated the first experiment, with the analysis registered in advance, on 692 online participants. Unlike the original, each participant saw only one version, not both. The effect appeared clearly and was even stronger. The physician’s decision got 1.81 when the operation succeeded and 0.45 when the patient died. The effect also appeared in the 44 participants who had said the outcome shouldn’t count.
Outside the lab. Lefgren, Platt and Price (2015) followed NBA coaches. After a loss, they change strategy more often than after a win, even when the game was lost by a very small margin, which says very little about how good the team is. The change appears even when the loss was expected, and when it was caused by factors outside the team’s control. Meier, Flepp and Franck (2023) replicated the result: NBA coaches change their starting lineup more often after narrow losses than after narrow wins. The same pattern appeared in the top women’s professional league, in college basketball and in American football (NFL), although in some cases it weakened over time.
The outcome also changes what seemed predictable. Kneer and Skoczeń (2023), in ten preregistered experiments with 2,043 participants, showed that a harmful outcome makes people believe, afterwards, that harm was more likely. As a result, the decision maker seems more negligent and more to blame. Some ways of correcting this distortion worked, others didn’t.
Example
A team launches a product after a good test with a small group of customers. The launch fails because of a supply crisis nobody anticipated. In the meeting afterwards, the decision to launch is called “rushed”. Had the launch succeeded, the same decision, with the same data, would have been called “bold”. The example is an illustration, not a study cited here.
How to apply it
The steps below are a practical way to apply it that we propose, based on the studies cited.
- Write down your reasons before you learn the outcome. What you knew, what options you had and what risks you weighed. Otherwise, after the outcome, you can no longer reconstruct accurately what could have been known.
- When you evaluate a decision, first ask what was known at the time. Then ask whether the outcome brings new information about the decision or was just luck, good or bad.
- Don’t change a strategy after a single narrow failure. A game lost by one point, or a project that failed from an outside cause, says little about how good the plan was.
- Don’t rely on knowing about the effect. In both lab studies, the effect appeared even in those who said the outcome shouldn’t count. A written process, set in advance, helps more than the intention to be fair.
Limits and nuances
- The outcome isn’t always irrelevant. Baron and Hershey note that judging by outcomes is often a useful shortcut. When you don’t know what information the decision maker had, or when they might hide what they knew, the outcome is sometimes the only clue. The problem arises when the information is complete and the outcome still changes the evaluation.
- The lab scenarios are hypothetical. Participants judged written cases, not real decisions with consequences for them. The coaching studies, though, show the same pattern in real decisions.
- The coaching studies are observational. They compare narrow losses and narrow wins, to get as close as possible to an experiment, but they aren’t experiments. The result weakened over time in some leagues.
- The size of the effect depends on how it is tested. The original study had only 20 participants, and each saw both versions; some said they remembered their earlier rating and repeated it. The replication, in which each saw only one version, found a larger effect.
- What we didn’t use. A well-known study on outcome bias in evaluating ethical decisions is co-authored by Francesca Gino, who has had papers retracted for fabricated data. We didn’t use it.
- What we read. Baron and Hershey (1988), from Jonathan Baron’s site at the University of Pennsylvania, and Aiyer et al. (2023), from the International Review of Social Psychology, we read in full. For Lefgren et al. (2015), Meier et al. (2023) and Kneer and Skoczeń (2023) we read the abstracts. The full copies of the last two, in university repositories, were blocked by anti-bot checks or access limits.
Sources
- Jonathan Baron, John C. Hershey (1988). Outcome bias in decision evaluation
- Lars Lefgren, Brennan Platt, Joseph Price (2015). Sticking with what (barely) worked: A test of outcome bias
- Sriraj Aiyer, Hoi Ching Kam, Ka Yuk Ng, Nathaniel A. Young, Jiaxin Shi, Gilad Feldman (2023). Outcomes affect evaluations of decision quality: Replication and extensions of Baron and Hershey's (1988) outcome bias experiment 1
- Pascal Flurin Meier, Raphael Flepp, Egon Franck (2023). Replication: Do coaches stick with what barely worked? Evidence of outcome bias in sports
- Markus Kneer, Izabela Skoczeń (2023). Outcome effects, moral luck and the hindsight bias
See also: Fundamental attribution error, Hindsight bias, Regression to the mean