First, let’s take a look at where we are in terms of our broader goals. INFER came to life because of a belief that every major policy decision within the U.S. Government should leverage the collective intelligence of its government workforce and supporters like you from around the world. Earlier this year, with a grant from Open Philanthropy, UMD’s ARLIS and Cultivate Labs, set out on a mission to build a crowdsourced forecasting capability that would support and add rigor to analysis conducted within the government and ultimately better inform policy decisions.
This first year, we launched and grew the site you’ve come to know as INFER-Public, which will be followed next year by an expansion of our forecasting ecosystem with an internal U.S. Government-wide site. INFER Public will continue to serve an important role for the program going forward, providing the government with unique outside forecasting perspective unlike any other.
2022 Season Highlights
33,000+ Forecasts 2,500+ Users 900+ Active Forecasters 25+ Countries Represented 86 Pros
One of this year's key achievements was developing and refining a new methodology for “strategic question decomposition”: our approach for working with U.S. Government stakeholders to identify and break down complex, high-stakes issues into the forecast questions that provide early signals around those issues. The approach allows us to break down 10+ year strategic questions into shorter-term guideposts that can be tracked and analyzed for events otherwise nearly impossible to forecast.
Central to what keeps INFER running are the partnerships we have formed with the U.S. Government. We are grateful for our relationships and advocates within elements of the Department of Defense, Intelligence Community, State Department, and other areas of government. Working with them, we identified INFER’s focus areas for the year and applied the decomposition approach to assess U.S. competitiveness in artificial intelligence, microelectronics, the global AI race, and synthetic biology.
To better support stakeholders in their decision making process, we launched a syndicated reporting and alert service for curated outputs from INFER. We published reports on our focus areas on a quarterly basis, which included probabilistic forecast summaries, changes to forecast trends, and supporting rationales – it’s this context around the forecasts that stakeholders find to be as valuable, if not moreso, than the forecasts themselves. As INFER tracked thousands of forecasts on dozens of questions, we also began releasing periodic alert reports on forecast questions where we saw significant shifts in the crowd consensus based on probabilistic updates and rationales. We reported on events including the rise of protests and riots in Russia (May), semiconductor fab plans in Europe (July), election results for the Brothers of Italy (September), and China’s predicted drop in Integrated Circuit production (November).
Our ability to generate these kinds of insights wouldn’t be possible without our forecasters. One of the most important aspects of the program has been our ability to attract and engage with a community of individuals who bring their own subject matter expertise, life experience, and diversity to our fold. We know that to get a high-caliber forecasting pool, we have to make investments in our forecasters, so that they’re also gaining benefits from being here. These are some of the ways we strived to do that this year. We:
- Secured sponsorship to run the Pro Forecaster Program – 86 participants qualified to serve as paid forecasters for their outstanding performance and contributions. Pro Forecasters amassed nearly 20,000 forecasts on 233 questions this season alone. Their team’s relative accuracy score was -0.779 (as of 12/22), representing significantly more accurate forecasting than the site average (see Figure 1 below). Two of our Pros, Ryan and Dawna, served as community mentors providing insight into a variety of questions.
- Established a new Team Lead role and filled 10 spots with representatives from 5 different countries. These individuals receive a small payment for recruiting and leading their own team of forecasters throughout the season and serve as ambassadors for the program within their communities. They’ve brought on a sizable number of forecasters on their teams in the three months the role has existed.
- Developed a framework for a forecasting training curriculum working with Pytho, which will be expanded upon to introduce it to our community in the coming year. As part of that curriculum development, Pytho taught a beta class on open source research methods to some of our Pros and a few other selected forecasters.
- Hosted quarterly Fireside Chats so that our forecasters could learn from experts in synthetic biology (Andy Kilianski, International AIDS Vaccine Initiative), microelectronics (Jesus A. del Alamo, MIT), and artificial intelligence (Gregory Dawson, Arizona State University), facilitated by Beth Sanner, former Deputy Director of National Intelligence. INFER also hosted exclusive events for Pros including a program kickoff with Neil Wiley, former Deputy Director of National Intelligence, and sessions with the State Department Bureau of Conflict and Stabilization Operations and INFER’s AI subject matter expert Mayank Kejriwal (University of Southern California).
- Ran the Effective Altruism Tournament, where university students in the EA community got the chance to develop their forecasting skills and compete for top honors in alignment with EA's philosophy of using better data for decision making and resource allocation. With more than 11 top universities in the world participating including Stanford, Cambridge, University of Delhi, and Yale, the success of this tournament led to the Future Bowl for our Team Leads program.
- Ran a number of reward challenges, allowing participants to earn badges, monetary compensation and recognition for their successes.
Figure 1. Pro Forecaster Accuracy Improvement During 2022 Season
Major new releases and enhancements to the INFER platform this year also centered on creating a better experience for forecasters, as well as improving the experience for data analysis of our consumers. Here are some of the release highlights (you can also see the full list):
- Overhauled the user profile to show personal performance statistics and visualizations, comparisons to the crowd, recent activity, and earned badges.
- Within question pages, built new visualizations, sorts, and filters that expose both contrarian thinking and the most “well-written” rationales to both buttress and challenge status quo thinking.
- Within the forecasting interface, launched a "pre-mortem" input to the forecast rationale to guide forecasters to challenge their own thinking (and give analysts valuable assumption-challenging fodder) to consider why they might be wrong.
- Introduced several new types of forecasting questions to better collect forecasting signals, including continuous questions (e.g. Will X happen in the next 6 months) and multi-time period questions (e.g. What will sales be in each of the next 4 quarters?)
- Created a new area to support analysts on individual forecast questions with forecast distributions, forecast update trends, and other insights.
- Produced a new heuristic for the crowd forecast’s “reliability” by benchmarking against all other forecast questions.
- Generated weak signal identification tools, including forecast cluster visualizations.
- Expanded API specification to accommodate more individual forecast analysis.
As we look towards the future, we’re excited about a partnership with a major player in the AI space for a new tournament that will be announced next month and kicks off in the spring. We are also continuing to collaborate with new partners within the U.S. Government and its allies, which will lead to new forecasting topics launched over the next few months. For those forecasters who qualify, we hope you will accept our invitation to join the 2023 Pro Forecaster Program (invites will be sent out through the end of January).
In January, you can also expect updated leaderboard rankings as many of our existing questions close at the end of December. We’ll start the 2023 season with a fresh new leaderboard, so anyone who was falling in the ranks can redeem themselves. And if you’re a top scorer this season, don’t worry, your performance will get saved in the 2022 leaderboard and you can always refer back to it!
We remain grateful for the support we’ve gathered from this community. Thank you to those who have been with us since Day One, and to our newer forecasters, we welcome you on this journey in the pursuit of more informed policy making. We are looking forward to what the next season brings and forecasting with you in 2023!
----
If you made it to the end of this post, we’d appreciate it if you could take 5-10 minutes to provide feedback on your forecasting experience to help us plan the year ahead.