When a question resolves in a direction you weren’t expecting, it can be easy to feel demoralized. At INFER, we like to remind forecasters that each question presents a new opportunity for forecasting, each with their own leaderboard, and that scores reset at the end of every season. When you do drop in ranking due to a single inaccurate forecast (as happens to many of us sooner or later!), these moments also make for an excellent learning opportunity.

A forecasting best practice we encourage is to conduct a post-mortem: examine your rationale, understand its faults, and develop new practices to make better forecasts going forward.

First, understand in detail how you approached your forecast.

Start by re-reading your initial forecast, since this will likely represent the main thought processes you used in forecasting this question. Identify the assumptions that went into the rationale, both explicit and implicit. Look at the evidence you cited, and examine the logic you used to connect that evidence to the forecast you submitted. For now, avoid being overly self-critical, and just try to understand what you were thinking at the time. This can be difficult since you now have the benefit of hindsight, but it will help you later identify areas for improvement.

Second, after you fully understand your rationale, identify the faults in your forecast.

With hindsight, it might be easier to see where you strayed. Were you forecasting a foreign election, and assumed that the dynamics would be roughly similar to the dynamics in your own country’s politics? Did you over-update on a single piece of information you found instead of more strongly anchoring to a base rate?

When doing this, try to avoid thinking that the thing that derailed your forecast was just one specific factor that you couldn’t have predicted. Accounting for the unexpected (perhaps by forecasting closer to the crowd, or base rates) is a key element of making a forecast. If you aren’t conducting pre-mortems during your forecasting process, adding that can be an area for improvement in your practice.

Additionally, try to think of how you could have updated more frequently or accurately during the question forecasting period. On INFER you can set reminders to forecast at a repeated interval (weekly, monthly, etc.) or receive alerts when the crowd forecast changes by a certain amount.

Third, now that you understand your rationale and its faults, you can create a plan of action to improve for the future.

Developing a perfect 5-step plan to accurately forecast every time is impossible, so instead you can use heuristics and general guidelines to develop your forecasts. Using the faults you identified in your prior forecasting, you can add to or change the process you typically use. Maybe you made too many unstated assumptions in your forecast, so instead, try writing out a list of assumptions in your rationales going forward to help flag some inaccuracies that will hurt your forecasting if left unchecked. Or, if you just followed base rates without sufficiently updating on new information, in the future you might take a minute to examine how whatever base rate you are using might not apply to the question at hand. This process will be highly specific to your own forecasting style.

Always remember that, no matter how a particular question goes, there will be other questions to practice your skills and improve your ranking. Forecasting is a marathon, not a sprint, and hopefully with some of the steps outlined above, you’ll be able to improve your form if you stumble.

—---
If you liked this post, check out how one of our team members conducted a post-mortem on his embarrassingly failed forecast in his write up here.