Are We There Yet? vol. 63

Do you ever make a decision or judgment that is just way off?  I do. The new book Noise: A Flaw in Human Judgment by Daniel Kahneman says that not only does this happen to everyone, but it happens to a much larger degree than we all think it does. Mr. Kahneman, author of Thinking Fast and Slow, was interviewed by Shankar Vedantam on the Hidden Brain podcast. 

Kahneman defines noise as unwanted variability in judgments or decisions and the book provides many examples about the level of noise in areas where we expect more accuracy and consistency. In the insurance industry where one might expect that the underwriting process to consistent, a study found the variability to be 55%. Industry executives estimated it would be 10% before the study. In the criminal justice system, two cases with very similar facts result in dramatically different sentences.  In college admissions, studies show that on cloudy days, admissions officers pay more attention to academic achievements of candidates than other attributes, and in the medical field studies show that there is not only significant variability among medical professionals examining the same data, there is even significant variability for the same medical professional who examines the exact same date but at different times.  

There is an ongoing debate on whether using algorithms is a path to getting better results.  In one instance, scientists devised an algorithm to advise judges on whether to grant bail to suspects.  Keeping suspects in jail has all kinds of bad effects but it is done to protect the public. According to Kahneman “They found that if judges incorporated the recommendations, this could reduce the number of people in jail by 42% without increasing the risk of crime.” The study also concluded that “allowing the algorithm to inform the judge is actually not the best way of doing it.  The research suggests quite strongly that when you have a judge and an algorithm that are looking at the same data, with some exceptions, it’s better to have the algorithm have the last word. And this is very non-intuitive.”  Noise is a source of inaccuracy and algorithms by their nature are noise free. However, many people do not want computers running their lives even in cases where there is overwhelming evidence that there would be better outcomes in many situations.  

So how do we fight noise in other ways where algorithms are not desirable or possible? Because noise is random, taking the average of independent judgments will reduce it. Vedantam and Kahneman discussed the story where 787 villagers at a county fair were asked to estimate the weight of a prized ox. There were a wide range of answers but when they took the average of those answers, it was within 2 pounds of the correct weight. From a practical standpoint, this informs me to do more due diligence when I’m making decisions.  Many of us want second opinions of medical diagnoses but maybe we should get even more.

But, as important, is to try to remember that there is always noise and that it is likely creating more variability and therefore inaccuracy than I might otherwise imagine. Recognizing that there is noise requires a certain humility and a firm belief that human judgment, ours included, may not be as good as we hope and believe it to be.

I hope you have a pleasant and noise-free holiday weekend. 

This week’s selection is:

BOOK:

Noise: A Flaw in Human Judgment by Daniel Kahneman, Olivier Sibony and Cass R. Sunstein

From the Nobel Prize-winning author of Thinking, Fast and Slow and the coauthor of Nudge, a revolutionary exploration of why people make bad judgments and how to make better ones.