Excerpts from our recent interview with Philip Tetlock on his new book, Superforecasting: The Art and Science of Prediction.
Financial Sense: Tell us about the fascinating research that led to this book?
Philip Tetlock: The origins of the project really date back to one of the greatest intelligence failures in US intelligence history and that was the mis-estimation of weapons of mass destruction in Iraq... One of the consequences of that was the creation of the Intelligence Advanced Research Projects Activity (IARPA), which is a research branch of the Office of the Director of National Intelligence, and they decided in 2010 to launch a series of forecasting tournaments to assess whether or not this is a skill that can be cultivated and is worth cultivating. And essentially our research between 2011 and 2015, the four years of the tournament itself, focused on trying to identify people who were better at making good probability estimates, at training them to make probability estimates, at organizing them into teams that were more than the sum of their individual parts, and developing some statistical techniques for aggregating crowd wisdom that could outperform simple averaging.
FS: Your book seems to make a pretty good case that forecasters on television—that is, those that are typically well-known for making forecasts—are actually the least accurate. It's as if there's this paradoxical inverse relationship between fame and forecasting ability. Is that true?
PT: There's quite a bit of evidence to support that. If you think about what draws the media towards pundits, it's really punchy soundbites that can grab an audience. The media is not really in an accuracy game by and large. They're in a market share game. Forecasting tournaments are entirely about accuracy and it turns out that the mental skills that go into forecasting accuracy in complex environments are not the same mental skills that go into an audience on television. The best forecasters in our tournaments have tended to be self-critical souls who say: 'on the one hand...on the other hand...'—they are more multivariate sorts of thinkers, whereas the thinkers who tend to create a more positive impression in the media, they're less likely to say 'however' and they're more likely to say 'moreover.' They build up a lot of momentum in their argument. They can give you four or five or six mutually reinforcing reasons why something is going to happen and that can be quite catchy and catchy-ness is quite important in the media world.
FS: What are some of the traits of superforecasters?
PT: They tend to be more granular... The really good forecasters in our tournaments have tended to be people who try to distinguish as many degrees of doubt as the problem permits. Now, obviously, there's a probability scale that runs from 0 to 1 and in principle that's infinitely divisible and no human being can make that many subtle distinctions, but our best forecasters can make more distinctions among levels of probability than most people can—upwards of 20—whereas typically people are only distinguishing three, four, or five.
FS: Tell us about the use of frequent updates and constant course correction in helping to be more accurate in your forecasts.
PT: Well, it helps a lot to be willing to change your mind. And most of the time, most of the news that we get doesn't call for a dramatic change in our mind. The evidence gradually accumulates that we're moving into another global recession or the evidence gradually accumulates that Hillary Clinton is not a very good presidential candidate. And gradually the probabilities adjust accordingly. It's not that that you're going to move from...60% probability...down to 0%, rather what the best forecasters tend to do is they tend to move incrementally and their adjustments are often 2-3% overtime and if the trends are consistent that accumulatively can produce a large change and it typically leads them in the direction of being closer to the truth than further from the truth and that's what causes you to win forecasting tournaments.
Listen to this full interview with Philip Tetlock on his new book, "Superforecasting" by clicking here. For a complete archive of our broadcasts and podcast interviews on finance, economics, and the market, visit our Newshour page here or iTunes page here. Subscribe to our weekly premium podcast by clicking here.
