by Kevin Silberberg
Some evidence may be weaker than you think
You’re in the passenger seat while your coworker is driving you home from work on a sunny Tuesday afternoon. Suddenly, traffic slows to a crawl. After five minutes of stop-and-go you spot the problem: a police checkpoint is making every driver pass a breathalyzer test. You finally make your way to the front of the line. Your friend blows into the breathalyzer… they fail. They swear they haven’t touched any alcohol, but the next thing you know they’re in handcuffs.
What’s the probability that the breathalyzer could be wrong? In this situation, it’s much higher than you think.
You don’t know your coworker that well but you’re skeptical that they would be drunk on a workday. Let’s assume that only one in a thousand drivers are drunk on a Tuesday afternoon.
However, as the police officer is arresting your coworker, he informs you that the breathalyzer test incorrectly classifies a sober person as drunk only 3% of the time and will always correctly classify a drunk person as drunk.
Instinctively, we tend to overvalue the impact of this new evidence. Due to the breathalyzer’s 3% false positive rate, many might give estimates as high as 97% for the probability that the coworker is drunk. In reality, the probability is much lower.
Given what you know, we can assume that if 1,000 random vehicles are stopped, on average:
- One driver would be drunk while 999 would be sober, since 1 in 1,000 drivers are drunk in general
- The one drunk driver would fail the breathalyzer
- 30 of the 999 sober drivers would falsely fail the breathalyzer due to the 3% false positive rate
Therefore, when randomly breathalyzing drivers, for every 31 drivers that fail, only one of them will actually be drunk. Thus, the actual probability that your coworker is drunk given that they failed the breathalyzer is still only 1/31 or 3.2%. In other words, there is a 96.8% chance that the breathalyzer was wrong.
In this situation, we can say that despite the breathalyzer evidence, your coworker was likely sober. We overvalued the breathalyzer test because we ignored how unlikely it was that a random driver was drunk in the first place on a Tuesday afternoon.
While virtually every company relies on statistics to make better decisions, this situation is just one example of how unintuitive statistics can be. At SATOV, we pride ourselves in our analytical processes by basing our insights at the intersection of deep statistical rigor and business strategy.
This is my second installment in my statistically focused article series. Stay tuned for my next article as I explore how to better leverage data and analytical tools for decision making so you don’t fall into the same trap as the breathalyzer example. If you missed my first installment you can find it here.