Why So Many Predictions Fail, but Some Don't. The Art and Science of Prediction.
Animated Summary by the Swedish Investor
Key Takeaways
1. The Signal and the Noise
Ever since the creation of the printing press in 1439, the amount of information that can be stored and reused later has exploded and this has only been enhanced even further with the spread of the Internet.
Information created and shared seems to grow at an exponential pace. The problem is just that the amount of useful information is not increasing at the same rate.
Say for instance that you’re an investor and you would like to predict how the economy will do during the next five years to tell if you should put your money in the stock market or not. Well you have more than 45,000 economic indicators to choose from, that are produced by US government only.
But which ones are relevant and which ones are not? Which ones are signal and which ones are noise? It’s important to be able to tell this difference if we want to make our predictions reliable.
According to Nate Silver, the signal is the truth, the noise is what distracts us from the truth. The goal of a prediction model should be to capture as much as possible of the signal, while keeping the noise to a minimum.
Let’s pretend for instance that unemployment rate is the signal. It is what can help us predict how the economy will do for the next five years. Then all the other 44,999 economic indicators are noise. They are just distracting us from realizing that it’s only unemployment rate that we should be focusing on.
The problem is that we humans have a hyperactive pattern recognition. It may have helped us thousands of years ago to distinguish whether the rattle from that bush is just a bird or if it’s a lion, but it’s harming us when making forecasts.
We tend to identify more patterns among than noise — more on this in Daniel Kahneman’s book by the way Thinking Fast and Slow and also more on this in takeaway number three.
The Productivity paradox is a central part of this book. We face danger whenever information growth outpaces our understanding of how to process it.
2. We Have a Prediction Problem
Just before the outbreak of the the financial crisis, the rating agencies had given their top ratings the triple aid to thousands of mortgage-backed securities. Some of these were said to have only a 0.12 percent risk of defaulting. How many of them do you think defaulted in reality? A whooping 28%— that’s 200 times more than S&P had predicted.
In 1990 the IPCC, the International Panel on Climate Change, forecasted that the rise of temperature would most likely be at a rate of 3 degrees Celsius per century, but definitely no less than 2 degrees per century going forward. As of 2011 when Nate Silver released the signal and the noise, the increase has been just 1.5 degrees Celsius.
Here’s how economists have forecast that the GDP during the 18 years between 1993 and 2010. (See Chart) In the chart the boss represent the area where the economists have said that GDP growth has a 90% chance of ending up. As you can see, the economists have been correct 12 times out of 18 or only 66% of the time. That is quite far from their stated 90%.
All right so apparently at least in some fields we are terrible at making predictions and yes you could blame Black Swan events for these failed forecasts like the S&P did, but in reality, it’s much more likely that the fault is in the model of the forecaster than in the world itself. We should show some sympathy though because there are situations in which predictions are extra difficult to make.
Event is out-of-sample to think that housing prices could have such a major effect on the economy seemed quite unlikely looking at historical data. The problem was just that in 2006 historical data didn’t help much in making a prediction as the conditions were vastly different than ever before. Never had the economy being so highly leveraged and never had so many side bets been made on housing prices.
Dynamic systems when the behavior of the system at one point in time influences its behavior in the future, we have a dynamic system. This means that even if we are just slightly off in assessing the current state of the system, we will end up very wrong when predicting the future of it as mistakes multiply over time. This is the reason where we can focus the weather accurately a few days ahead but not more than that.
A lack of theory simply put we just don’t know enough about many of these systems. The economy, just like the climate, are complex beasts and although we have some heuristics or rules of thumb that we can conform to, it’s not always enough to make accurate predictions.
3. Correlation Does Not Equal Causation
Example 1: A mafia boss gives one of his minions three different locks with coats: a red, a blue, and a yellow one. He asks for a method to pick such locks. After a few days, the underling is back happily stating that if it’s a red lock just enter 1645, if it’s blue 3493, and if it’s yellow 0232.
The underling would have mistaken correlation for causation and would completely have failed his task. There’s no reasonable explanation for why the color red would have a person enter the exact digits 1645, it just happened to be so in this particular case.
Example 2: Is it true that consumption of ice cream causes shark attacks? From this chart you may come up with that conclusion as it’s quite clear that shark attacks increase whenever the ice cream consumption does, but obviously there’s no causal relationship here.
The ice cream consumption does not cause shark attacks. It’s just that both swimming in the ocean which makes us more likely to get attacked by a shark and eating ice cream is more enjoyable in the summer.
While these examples may seem a bit silly, mistaking causation for correlation is a very common forecasting problem. As stated earlier there are 45,000 economic indicators produced by the US government each year.
If you are trying to predict the stock market for example, and you look at historical data to do this, you are almost guaranteed to find that some of these variables seem to have strong predictive power. Yet they may not. You may just have been fooled by randomness.
Whenever you make a prediction, be sure that there’s a logical explanation for the mathematical relationship. Do not trust data unconditionally.
4. How Can We Become Better at Predicting?
Here are three examples on how we can become better forecasters:
1. Think Probabilistically Reality is not a yes/no situation, even though some people seem to think that.
- For example, if I ask you how the stock market will perform next year, what is the best answer?
- It’s not for example 5% and it’s not minus 5% minus 1% or 10% either.
- The best answer is a spectrum of outcomes with probabilities attached to them.
- Good predictors are great at seeing reality as such and have developed a skill for assigning probabilities to outcomes.
2. Change the Forecast with New Evidence say that you are dealt two aces in poker—the best starting hand possible. You decide to bet and you are called by one of your opponents. The first three cards on the table are the following: nine of spades, Ten of spades, and jack of spades. He checks, you bet and he calls. The fourth card is an 8 of spades, your opponent decides to bet. Should you call this bet or not?
Your initial forecast was that you were going to win this hand easy, but clearly you must factor in the new evidence. You are beaten by an awful many straights and flushes and should definitely fold this hand.
You should make the best forecast today regardless of what you said yesterday. (more on this in the final takeaway)
3. Look for Consensus good predictors are good at weighing multiple sources of information. They don’t get lost in narratives or stories and they are able to weigh both quantitative and qualitative information together before making their decisions.
- For example they understand that Netflix can be a great stock to buy because it has an enormous potential for scalability, but simultaneously they realize that it’s quite expensive to buy a stock no matter how great it may be for 130 times its last year’s earnings.
5. Bayes' Theorem
You’ve been holding on to that Tesla stock for quite a while now and this very evening they reported an update that the model 3 seems more difficult to mass-produced than first anticipated. Will Tesla go bankrupt?
This is an example which involves updating a previous forecast when new evidence has presented itself. Most humans are terrible at this. Luckily there is a solution—Bayes’ theorem.
Bayes’ theorem is a mathematical formula that can help you in calculating the probability of something occurring given that something else has happened.
Successful predictors in any field recognize this when new information presents itself. We must update our initial hypothesis. Perhaps we didn’t think that Tesla would go bankrupt the first time they presented a liquidity issue, but if new problems constantly arise, we must update our estimates.
People may see this as a sign of weakness. All right look at that guy, he’s constantly changing his mind, but it’s actually the most rational thing to do. Only when we are constantly refining our estimates can we come closer to the signal and further away from the noise.