Merchants of Risk - by Tomas Pueyo - Uncharted Territories Merchants of Risk
Some content could not be imported from the original document. View content ↗
15 Tactics to Improve How You Predict the Future
2021年8月1日
∙ Paid
I lived off of gambling for a decade.
I would wake up in the morning, go to the office, sit in front of my computer, and ask myself: “What type of experience can we create that people will love?” But I never knew what would work.
And we tried things. I built everything, from study documents, to investment services, applications to watch funny videos, videogame tournaments...
I never knew what would work. Once, we spent weeks building an easier way for users to sign up for our service. “It’s easier. It takes 5 clicks instead of 10. They have to sign up more!” They signed up less. Another time, the team would knock out a small change to an obscure way Facebook interacted with our product, and growth would explode.
The same thing happens to me as a writer. I could have never imagined that a COVID article I wrote would explode virally. Over time, as I gather more experience, I have developed an intuition for what people like most.
But intuition is not enough. We need to systematically get better at predicting outcomes.
My job growing online products does not really consist in coming up with new features. Rather, I need to guess with my teams the problems of our customers—you never know for sure—, guess the solutions that will solve these problems, guess the cost of these solutions, and then compare all these guesses across initiatives to guess which ones we should try first. Guess, guess, guess, guess, guess… I deal with uncertainty. Ambiguity is my business. I’m a merchant of risk.
You probably are too. Every investor is a merchant of risk. Every person who needs to pick initiatives to follow and isn’t sure of the outcome is a merchant of risk. Lawyers, doctors, nurses, journalists, writers, movie makers, politicians, engineers, marketers, recruiters, educators, soldiers, salesmen, CEOs, farmers, researchers… All of them need to make decisions under uncertainty, and the outcomes of their jobs—and their fortunes—depends on managing that risk well.
So how can you best pick what will work well? There are two strategies: taking more shots on goal, and improving the quality of every shot.
The first strategy, to take more shots on goals, has another, more famous name: diversify. When you don’t know the winner, bet a bit on all the candidates. It is so well proven that it’s backed by a Nobel prize [1] . Curiously, many people who deal with uncertainty don’t diversify. Some employees keep overinvested in the stock of their employers, some farmers only grow one crop, some writers spend decades on their books. For your trade, ask yourself: how can I try many more initiatives? How can I cut costs and produce much more output, so I can diversify my risk?
But diversifying risk is not predicting the future. That’s accepting that predictions are hard.
We also need to figure out tactics to improve our forecasting ability, so every shot is on goal, of higher quality. So here are 15 proven tactics to better predict the success of any initiative, extracted from Consulting, Product Management, Investment Theory, and Superforecasting [2] .
1. Be MECE
It stands for Mutually Exclusive, Collectively Exhaustive.
Most problems are too big to solve directly. The best way is to break them down. Smaller problems can be more easily tackled. But how do you properly break down a problem? You need two things:
You need to make sure that the subproblems account for everything in the bigger problem. Subunits that properly cover the entire main unit are called “collectively exhaustive”.
You need to make sure these subproblems don’t double-count, overlapping with each other. Subunits that don’t overlap are called “mutually exclusive”.
For example, if you want to forecast GDP, you can’t only forecast ten industries. That’s not collectively exhaustive. But you also can’t forecast the sales of every company, because you would be double counting what intermediaries are paying to their suppliers, that’s not mutually exclusive.
An example of MECE forecasting is what we covered earlier this week: forecast demographics. If you look at all the people coming in (births, immigration) minus those leaving (deaths, emigration), that’s a MECE breakdown: these four factors are all separate (mutually exclusive), and add up to the total population (collectively exhaustive). Now you can forecast each one of these components separately.
Another example: forecasting the revenue of your company. Revenue is customers * revenue per customer. That’s all there is.
You can further break this down. Your current customers are those you acquire plus those you retain. And so on...
Once you have smaller things to forecast, you want to be good at forecasting every one of these subproblems.
2. Distinguish Trends from Fads
A typical mistake in predictions is assuming something will be successful just because it’s hot. Sometimes it is, sometimes it isn’t. It happens in fashion, ideas, articles, online products...
Sometimes, the situation is even more complex: things are both trends and fads. They are trends that get overhyped, thus appearing as an initial fad. After the initial hype is followed by a trough of disillusionment, it grows into the trend it eventually becomes.
So how do you tell apart a trend from a fad?
Break down adoption and retention.
Remember how we broke down demographics in people going in vs. people going out? It’s the same: people going in is acquisition, people going out is retention.
There’s a new product in the market. Millions are joining. Is it here to stick? Look at people who are using it, and see how many of them stick around over time. If after a week nobody is using it, it’s a fad. If months later there’s still a huge audience, you know it’s going to be big.
This is how Venture Capital investors assess whether a promising startup really has potential.
3. Study the Best
Studying what you must forecast is a no brainer. Look at all the evidence you know, get informed about everything related to your forecast. But within that, there are some things more relevant than others.
For example, you should always learn from the best: independent thinkers who don’t try to lure you with emotional hooks. People who are intellectually humble, who know they might be wrong, who disagree respectfully, who explain their thought processes. Brains that are not scared to disagree with their peers when they think they’re wrong, and who will admit it when they were wrong.
Who in your prediction industry is a famous thinker? Ask her opinion.
4. Study Precedents
In many industries, people study precedents well: doctors learn from previous cases, lawyers look at jurisprudence, soldiers study the history of war and perform drills...
Other industries have learned to do that more recently. For example, now you have 724 Marvel Universe movies because they are proven to work, there are precedents that they will make money.
Other industries don’t pay as much attention. For example, policy-makers have plenty of precedents from policies having been tried in other places, but they don’t learn from that. It was made very clear with COVID, but it’s true for many other things, such as the valuable precedents of Portugal’s drug policy or New Zealand’s sex work policy.
Precedents are especially useful in complex, chaotic situations hard to understand. Look at COVID: one and a half year in, and there’s still debates about masks, vaccines, lockdowns, treatments… If you go into the detail of each one of these debates, the nuances are indeed quite complex. Studying it all is awfully hard. But you don’t even need to go into all this nuance. If a bundle of policies works over and over again, you can adopt the bundle wholesale as your starting point.
Similarly, when developing products, a core tenet is Don’t Reinvent the Wheel. Others say Steal Like an Artist. When something works, adopt it.
This is forecasting by analogy, as opposed to forecasting by first principles. With analogies you won’t reach as deep an understanding, but it’s a shortcut to good forecasting at a lower cost.
5. Study First Principles
The opposite of analogy is first principles.
So many product teams I meet get inspiration from the competition for their initiatives, but then they say “Let’s not do like them. Let’s do something different!” What happens next is they propose something different, but you’re not exactly sure why. When probing, it’s not clear what justifies the changes.
In other words, they want to depart from the analogy, but they don’t dive deep into first principles to understand what might make the competition’s solution work, and why it needs to be changed.
If you depart from analogy, you need to go to first principles: dive very deep in all the drivers of a trend and make sure you understand them. Tim Urban from Wait But Why calls this The Cook and the Chef.
The cook follows recipes without understanding them. The chef has gone deep into the physics and chemistry of food to understand what will work and not. She can innovate much better than the cook.
But being a cook is easy! Just take a recipe and follow it. You can forecast that the recipe will work because it has worked for others. And if you do it once well, you have an even better precedent, and can increase the confidence of your forecast for the next one. In other words, you can predict a good dish coming from a recipe that you’ve tried and enjoyed.
To upgrade from cook to chef takes years of studying and experience. Only then, once you know the fundamentals, can you innovate, can you come up with new concoctions that you can forecast with confidence will work.
For example, the fields of mathematics and physics are full of first principles. We have a good sense of the core laws. But in medicine, we know very little. Most of what we know is empirical: “We tried this, it worked, we tried it again, it worked again, so we assume it will keep working.” This might be somewhat informed by first principles, but it’s mostly reasoning by analogy.
6. Know When to Apply Analogies vs. First Principles
This 2x2 matrix tells you when to forecast based on analogies and when with first principles.
If something is very repeatable and so complex that you don’t understand it well, use analogies.
If a situation is novel but you understand its drivers really well, you can apply your 1st principles rules.
The more you've seen a situation, the more you can predict what will happen next without understanding it. The ancients might not have known how the solar system worked, but they knew the sun was going to come up tomorrow.
The best thinkers are not dogmatic. They do both. They usually start with analogies, go deep to understand where they can apply 1st principles instead, understand their limitations, and stick with analogies where they don’t full grasp all the drivers of a trend.
They will also compare both results. If predictions from analogies and first principles agree, their confidence in the predictions increases. If not, then they need to understand why. Time to study more. Something’s going on.
7. Update your predictions
If your predictions are based on evidence, then if more evidence appears, you need to update your predictions.
This is the principle behind Bayes’ Theorem. When you make a prediction, you start with what you know, your “prior”. As more information comes up, you update your prior.
This sounds very obvious, but most of us don’t do it. How many times have I seen teams who predict a certain result for their features but as they learn more about the customer or the costs of the project, they don’t change their forecast?
You should have a specific process to update your predictions or it won’t happen naturally. Because this under-adjustment has two strong causes: biases and lack of measurement.
8. Fight Your Biases
Three of the most potent biases are at play here: confirmation bias, anchoring, and the availability bias.
Confirmation bias is the worst bias of all. It should be taught everywhere, and we need more vocabulary to talk about it. It’s the fact that when you already think something, you easily take in new information that confirms your preconceived idea, while discount evidence that weakens it.
Once you’ve made a prediction, it’s your baby. You don’t want to be wrong. So you don’t correct it. Fight that tendency.
Anchoring is the most important bias in negotiation. Any number that you see will “anchor” your proposal. That’s why you have an advantage proposing the first price.
Similarly, in forecasting, any prediction you make or see will anchor you. If you see something has a 33% probability of succeeding, and you see news that makes the success much more likely, you might update its probability of success to, say, 40%, when the new evidence might deserve an update to 80%. You’re anchored by the previous 33% and underadjust.
Watch out for the availability heuristic too: it you see something a lot—or you’ve seen it recently—you will put more weight on it. For example, you know how you might have the feeling that the world is more dangerous now? How children are so unsafe? These perceptions are why kids don’t roam free these days as much as they used to. That’s the result of flawed predictions based on availability bias: people think their kids are at risk because they hear all these bad news. They predict a higher probability of danger than there actually is, and keep their kids close by. Sure, the weakening of social ties means the community might not look out for kids as much. But kids are safer than at any point in history. Our poor predictions due to availability bias restrict the freedom of our kids.
Knowing about biases like confirmation, anchoring, and availability helps to reduce them. But they’re so strong that even if you know about them, you can’t fully get rid of them. This is another reason why you should have a process that forces you to update your predictions as new information comes. But to update predictions, you must be able to quantify them.
9. Quantify Your Predictions
This is one of the most undervalued best practices when predicting the future. People prefer to say something is “likely”, “unlikely”, “probable”, and so on. This is problematic for three reasons.
First, different people understand these labels differently. What does “likely” even mean? For some, it might mean 80%. But if you’re talking about the fatality rate of an illness, 20% is quite a high likelihood. If you say “the likelihood is 20%”, people understand better.
Second, without quantified likelihoods, it’s impossible to discuss them. If two people think something is certain, but for one person that means 90% probability and somebody else it’s 99.9999%, using the label “certain” hides this massive difference. Putting numbers to it makes the conversation possible.
This has been useful to me in the past:
“I think we have a high chance of success with this initiative.”
“How much?”
“We might increase sales by 5%.”
“How confident are you?”
“Very confident.”
“Can you quantify that for me? What likelihood do you think there is that the initiative will drive 5% or more revenue? 95%? 90%?”
“Oh no! Maybe 50%?”
“Ok, then we need to discuss this, because the project is too expensive for a 50% chance of success.”
The last reason why quantification is so important is accountability. If you say something has 80% chances of success, it should happen 80% of the time. How can you know if you don’t keep tabs?
So you want to keep tabs on predictions. How do you do that? Use Prediction Markets like Brier Scores.
In my career, getting people to do this has been one of the hardest struggles. Even today, some important people in the industry defend to never quantify the initiatives they work on. Saying: “this will increase sales by 5%” makes them feel like:
It’s too hard. It takes so much time to have a precise prediction! Shouldn’t we use that time to actually build things?
It’s false precision. How are we supposed to know if something will grow by 5%, 10%, or 0%?
The answer is that it does feel weird, but it’s ok. We need to understand the point of the quantification: it’s not a sacred number, it’s a tool. It allows us to talk about the prediction and to improve it over time.
10. Be Precise
It’s not enough to say “there’s a 30% probability that Trump will be a presidential candidate in 2024”. You should strive to be precise. Is it 30% or 23%? How can you update your predictions otherwise?
Imagine you learn Trump was flown to a hospital. You don’t know anything else. If you knew he had an illness, maybe you’d reduce the odds to 20%. But you don’t know this. You just know he was flown to a hospital. The simple risk that he might have an illness grave enough to justify flying him to a hospital should reduce your confidence in his candidacy.
11. Challenge Yourself
Once you make a prediction, due to confirmation bias, you get married to it. Processes to update your predictions force you to revise these predictions, but you might still undercorrect. How do you avoid that?
One of the best ways to challenge yourself is to think of counterfactuals: imagine you were wrong. What could have caused the mistake? This will force you to think of things you might have missed in the process of making your prediction.
12. Fast Feedback Loops
Once you quantify your numbers, you can keep people accountable. And the accountability allows you to learn. The more predictions you make, the more you can see where you were wrong, and go deep to understand why. This type of thoughtful training is exactly what makes people better at anything, including predicting.
For example, you can have a prediction party in your team, where people make bets on the outcome of an event. Then, once the event has happened, you have a results party where the result is read, scores are updated, and the team conducts a retrospective not just on why the event unfurled the way it did, but also why the predictions were close or far off.
If on top of that you give a prize to the best predictors, you will have a sudden competition for best predictions, which is another thing you can do to improve the quality of your predictions.
This requires putting a deadline to the predictions. It’s not enough to say “there’s a 10% probability that China will invade Taiwan.” The key here is: “By when?”
The shorter the timeframe between the predictions and the result, the easier it will be to connect predictions to results, and the more opportunities you will have to learn.
Social Predictions
Nearly all the tactics so far work well individually. But adding others to your forecasts can improve their accuracy substantially.
13. Use the Wisdom of the Crowds
In 1904, the statistician Francis Galton was in a county fair when he stumbled upon a competition: who could guess the weight of an ox? After the competition, he was allowed to look at all the 800 submissions, and was shocked to notice that their average was just one pound away from the true weight of the ox—1,197 lbs instead of the true 1,198 lbs—and a better guess than most submissions individually. The Wisdom of the Crowds has been famous ever since.
Why did that work? Every guesser came at the problem with different knowledge and biases. Some might remember the weight of that ox from the previous year, others might have been able to see the ox from all angles, others could interpret the weight of its footprints in the mud, others might be butchers used to guessing weights, others might have seen the ox was slightly bigger than another one that weighted 1,100 lbs…
Separately, they only hold a bit of information, but together they have a lot more. Conversely, their biases go in all directions, so they cancel out when aggregating the answers. The result is an accumulation of all the good info and a cancellation of all the bad, resulting in an accurate guess.
Stock markets work the same way: most participants don’t know the true value of a company, but in most cases, the bets of millions of people results in a very accurate price, incorporating new information at the speed of light [3] .
You can use the same principles for yourself with something called “Prediction Markets”, which have been proven to work very well. Thankfully, you don’t need to set up a full prediction market to benefit from this: the more people you ask to predict something, the more accurate the prediction will be.
In my trade, for example, product managers, engineers, designers, marketers, and analysts can gather around a proposed roadmap and make guesses on the results (and costs) of different initiatives. The average of the guesses will probably be reasonably accurate. If an initiative becomes less compelling after going through this process, you can swap it for one that has now a higher return on investment. This has the additional benefit of gamifying an otherwise boring experience for most.
14. Get a Diversity of Perspectives
A requirement for that to work well is diversity. If you have 100 people sharing their predictions, but they all have the same experience and knowledge, how useful is that? Their individual predictions don't add information to the overall prediction.
This is a source of positive impact of DEI (Diversity, Equity, and Inclusion) in business: by hiring people that have a variety of experiences and perspectives, they contribute different information to the group.
Apply that to your forecasts. Don’t just ask different people their predictions. Make sure the people you’re asking have diverse sources of knowledge.
15. Avoid Groupthink
We were talking about biases before. A huge source of biases is social. That’s a problem when using the wisdom of the crowd.
For example, as Cialdini explains in Influence and Pre-suasion [4] , when you see others think or do something, you’re more likely to do it. It’s called Social Proof. You’re also more likely to pay attention to what is said by people you like, people with authority, or those inside your group.
So if in a group some people hear other people’s opinions, they will be swayed, discounting their own information (or giving it more weight than they should). That’s a loss. You want to engineer your social dynamics to avoid this. Here’s a set of rules that work to avoid groupthink in general, and apply well to predictions:
People must state their initial positions independently. For example, everybody writes down their initial predictions before listening to what others have to say, and include why they think that. Once it’s written, they’re anchored in their initial position and will be less socially swayed.
Share the rationale of the prediction. Explaining the data and reasoning of different people will add to the information of the group.
Avoid emotions and stick to data and arguments: stats they read, anecdotes they lived, experience they’ve had… Otherwise, it becomes personal. When it becomes personal, the identity is mixed with the idea, making it so much harder to unmarry them.
Conversely, stop ad hominem arguments. Who somebody is is irrelevant to the quality of their insights. The only factor relevant to people’s identity is their track record.
Have a process to challenge each other. Otherwise, people will want to keep the harmony of the group, and stop countering each other.
Praise those who challenge each other, so that challenging becomes safe and part of the culture.
Avoid the influence of people with authority. In a group, people will defer to the boss’ opinion, or pander to it for influence. Exclude those with authority from the process, make their contributions anonymous, make it a point to challenge it, or all three.
Takeaways
Most of us are merchants of risk. Predictions are a core part of our jobs. We need to get better at predicting. Here’s the summary of 15 ways to get better at predicting:
Be MECE: break down predictions into their drivers.
Distinguish trends from fads by breaking them down into acquisition vs. retention.
Study the best.
Study precedents, so you can apply analogies to your situation.
Study first principles, so you can deeply understand the mechanisms that drive the trends.
Know when to apply precedents vs. first principles.
Update your predictions, otherwise you won’t incorporate new information.
Fight your biases, such as confirmation bias, anchoring, or the availability heuristic.
Quantify your predictions. It’s tempting to leave them at “probably” or “unlikely”, but that doesn’t allow you to discuss them or to keep yourself accountable.
Be precise. The more precise you are, the easier it is to incorporate new information and nuances.
Challenge yourself: think of counterfactuals to avoid getting married to your predictions.
Use Fast Feedback Loops to measure your performance against reality, see where you failed, and improve over time.
Use the wisdom of the crowds to incorporate information held by others.
Get a diversity of perspectives so each person who adds information to the group does indeed add more information and not just strengthens the existing bias.
Avoid groupthink by making people feel safe and reducing the influence of ones over others.
What other factors are there to improve the quality of forecasts? Leave them in the comments!
Modern Portfolio Theory from Markovitz.
The term comes from Superforecasting by Philip Tetlock, which I just finished. If you’re very interested in this topic, you should read it. I extracted from that book and added in this article the insights that were most salient to me.
Wikipedia has a different process to gather information, and doesn’t deal with predictions, but it works under the same principle: people gather their insights and cancel out (a lot of) their biases, resulting in a lot of articles that are quite accurate, in a way that is overall much much better than any single encyclopedia.
The two best books to read on the topic of influence.