We are publishing a new advice series here at the Pub called “Dear Fern,” where we will answer your questions about forecasting on INFER. Our goal is to address some of the pressing issues that our forecasters have with the process and demystify the world of forecasting. To do this series, we need questions from you! All questions will remain anonymous, so please ask your own questions by submitting them to [email protected].

Dear Fern,

Whenever I come across a forecast by an expert or someone else interviewed in the news, they will typically use terms like “unlikely” or “confident” when describing what they think will happen. These terms seem so much more common than the numerical probability format (0-100% chance) on INFER. Why doesn’t INFER use similar terminology?

Cheers,

Qualitative Quincy

Quincy,

Let’s play a quick game. I tell you that there is a “real possibility” that I have a crisp five-dollar bill in my pocket. You have the opportunity to trade me three dollars for whatever is in my pocket. Do you take it?

Whether you take it depends entirely on what percentage you correlate to the term “real possibility.” If you think a “real possibility” means upwards of 70%, you’re much more likely to take the deal. If you think a “real possibility” means below 40%, you’ll be less inclined to. 

Unfortunately, people perceive probabilistic language in different ways. Two people can have wildly differing opinions, due entirely to their interpretation of the words being used. Think about this in a national security context, where one leader’s perception could lead to a simple tweet of condemnation vs. an order to deploy the military. 

Let’s journey back to 1951, when the CIA Office of National Estimates told policymakers that there was a “serious possibility” that the Soviets would attack Yugoslavia. Sherman Kent, a CIA analyst curious about how individuals perceived such reports, asked others at the Board of National Estimates what they thought the percentage likelihood of an attack was based on that assessment. Some thought it meant a 20% chance of attack, others 80%. The policy implications of these extremes were starkly different.

Kent looked deeper into this topic in a previously confidential document, “Words of Estimative Probability,” where he noticed that in nearly every report put out by top intelligence analysts, they used subjective probabilistic language as opposed to concrete numerical predictions. He found that almost every probabilistic word proved to have a distinct range of probabilities associated with it. Kent asked nearly two dozen military officers from NATO to translate those words into numbers and came up with the distribution below. The Harvard Business Review did their version of this poll, and those results are also below:

The INFER team fell into a similar trap. I asked a few of my colleagues at INFER to assign probabilities to some of these words, and here are the results:

Not much better. 

If you want to do a similar test on yourself, check out this website, where you can see how different your perceptions are from the public. We can attribute these differences to context, gender, native language, and many other factors.

Numeric probabilities don’t have that level of ambiguity. To me, to you, to everyone, 70% is just that: 70%. Additionally, it is far easier to aggregate numeric values than linguistic predictions. What is the average of “maybe,” “unlikely,” and “serious possibility?” 

Now, let’s consider the same game we played earlier, but with more precise estimates. Let’s say I tell you that there is a 70% chance that I have a crisp five-dollar bill in my pocket. Then you have the chance to trade me three dollars for whatever is in my pocket. Your answer may depend on your level of risk aversion, if I seem trustworthy, and many other factors that would make the decision psychologist Daniel Kahneman perk up. However, given the probability, you can at least evaluate the expected value of your return and make a more informed decision than before. Do you take it? 

Given the strong, clear answer the numeric value is giving you, your next move is easier to figure out, isn’t it?

Happy forecasting,

Fern