Chapter 1: The Concept of Regression to the Mean

The idea of regression to the mean can often be perplexing, particularly when it comes to our intuition. One of my personal favorite logical missteps is the 'post hoc ergo propter hoc' fallacy, which translates to 'after this, therefore because of this.' This fallacy suggests that just because event B follows event A, it does not imply that A caused B. This misconception underpins numerous superstitions worldwide, including various religious beliefs. For instance, in India, encountering a black cat crossing your path is often viewed as an ill omen.

Moreover, people might believe that starting a task immediately after sneezing will lead to negative consequences.

Superstitions operate on a flawed yet straightforward principle: if the outcome of an event is unfavorable, blame the preceding event and avoid it in the future. Even if you don't consider yourself superstitious, the 'post hoc ergo propter hoc' fallacy might influence your decision-making in ways you don't realize—similar to how the zombie-ant fungus Ophiocordyceps unilateralis manipulates its host ants.

Let's delve deeper into this topic before we get sidetracked by other thoughts.

Nobel Prize-winning psychologist Daniel Kahneman once shared insights with flight instructors in the Israeli Air Force regarding the effectiveness of training techniques. He enthusiastically asserted that positive reinforcement leads to better performance than negative feedback. The instructors, however, were initially silent. Once he finished, one seasoned trainer stood up to share his experience:

“I've often praised cadets for executing maneuvers well, only to see them perform worse the next time. Conversely, when I scolded them for poor execution, they tended to improve.”

This moment sparked a revelation for Kahneman, who recognized the instructor's observation as insightful but his conclusion as flawed. The instructor failed to see that the praised cadets were initially performing above average due to luck, while the reprimanded cadets were likely to improve due to a prior poor performance.

This common misconception is known as regression to the mean. In reality, random fluctuations in performance explain the variations better than a cause-and-effect relationship.

Kahneman elaborated on this by explaining that the instructor tended to praise only those whose performance exceeded the average, which could lead to a decline in their subsequent attempts. Conversely, he only criticized those who performed poorly, who would naturally tend to perform better on their next try.

To demonstrate this point, Kahneman conducted an impromptu experiment in the classroom.

He drew a target on the floor and instructed each officer to turn their backs and throw two coins at it without looking. The distances from the target were recorded, and the results were arranged from best to worst. It became evident that those who excelled in the first attempt often performed worse in the second, while those who struggled initially showed improvement.

This practical demonstration helped Kahneman gain the trust of the instructors by unraveling a fundamental aspect of human psychology.

The concept of regression to the mean is evident in our everyday experiences. If you feel that people tend to treat you poorly when you are kind to them, you might be falling for this cognitive pitfall.

“The feedback to which life exposes us is perverse. Because we tend to be nice to other people when they please us and nasty when they do not, we are statistically punished for being nice and rewarded for being nasty.”

— Daniel Kahneman

TheUnknownDoktor

References:

• ‘Thinking, Fast and Slow’ by Daniel Kahneman
• Regression toward the mean
• Regression Toward the Mean: An Introduction with Examples

Chapter 2: Misunderstandings of Regression to the Mean

To further explore this concept, check out the video titled "Misunderstanding Regression to the Mean" on YouTube, which delves into common misconceptions surrounding this statistical principle.

Additionally, the video "Regression to the Mean: Misinterpreting Statistical Evidence" provides valuable insights into how this fallacy can mislead our understanding of statistical evidence.