The State of Forecasting: Dynamics, Challenges, Hopes

In short…

• Forecasting platforms and prediction markets are partially making the pie bigger together, and partially undercutting each other.

• The forecasting ecosystem adjusted after the loss of plentiful FTX money.

• Dustin Moskovitz’s foundation (Open Philanthropy) is increasing their presence in the forecasting space, but my sense is that chasing its funding can sometimes be a bad move.

• As AI systems improve, they become more relevant for judgmental forecasting practice.

• Betting with real money is still frowned upon by the US powers that be–but the US isn’t willing to institute the oversight regime that would keep people from making bets over the internet in practice.

• Forecasting hasn’t taken over the world yet, but I’m hoping that as people try out different iterations, someone will find a formula to produce lots of value in a way that scales.

The forecasting platforms and prediction markets

Here is a non-exhaustive list of how a few organizations make their money:

• Las Vegas is a profitable city. People go there to get married by Elvis, or to gamble money on things of no social utility, like cards or betting machines.

• DraftKings is a sports betting market. They make money from people betting on sports.

• Betfair is a massive betting market for sports in the EU and UK. They make money from charging a spread to bettors, and they occasionally host markets on politics.

• PredictIt supports itself with a 10% fee on profits and a 5% fee on withdrawals. This incentivizes successful bettors to attempt to compound their bets before withdrawing (to reduce the impact of the 5% withdrawal fee), but also makes markets less efficient—because moving the market to the correct price might not be worth it after fees.

• Cultivate Labs is an example of a more mature company in the field. Last time I checked, they were charging in the neighborhood of $100k/year for hosting a version of their platform, for e.g., INFER or for the UK civil service.

• I’m not very up to date on Polymarket’s finances. But initially they raised $4M from investors, and they have been spending money by providing liquidity and paying for their team. I would be impressed if they were making enough money from fees and market-making to break even or even be profitable, because this has historically been tricky.

• Manifold Markets’ finances are public, as are their user stats. They have raised $2.9M in venture capital funding, and an additional $1.1M in grants. As of @September 28, 2023, they had ~$2.1m cash in the bank and their most recent burn rate is $1M/year. Their mana sales are $10k/month, or an annualized $100k: this is small compared to their burn-rate, but better than the $1k it was about a year ago. Financially, the question for Manifold is whether they will be able to continue raising grants to retain their current size and ambitions, or whether they will be forced to rely on their users for funding, and if so, how.

  • Manifold also recently announced a major pivot into real money markets. It remains to be seen whether they will go through with it, and whether it will make them profitable.

• Metaculus turned itself into a Public Benefit Corporation, and they have been getting some very large grants from Open Philanthropy.

To me, these numbers could be telling a few stories. Here is the first: Las Vegas is an attractor, because enough people want to lose money in a casino, and pandering to them is profitable. Platforms start out with some idealism. Only through virtue and funding from philanthropists are platforms able to resist the pressure.

Here is another story we could tell: Efforts in the space are distorted by VC funding. There may have been the opportunity to build some sustainable business here. But instead, VCs funded platforms that would lose money in a bid to acquire a user base which could later be monetized. This was the world in 2022, and we are now seeing the consequences after that funding dried out.

Here is a third story: At least the VCs were trying to build businesses. The biggest distortion is the continued funding from Open Philanthropy to Metaculus. Anyone else trying to build something is going to have to compete against Metaculus, who is providing actually really good probabilities for free, as a public good.

Here is a fourth story: Funding flows are distorted not by Open Philanthropy, but by online hobbyists providing probabilities pretty much for free, at Metaculus, Manifold Markets, or INFER. These online hobbyists can be pretty intense, but also aren’t scalable.

Is this actually going on? Is it the case that providing a good for cheap prevents the creation of a more expensive but more scalable good? That’s a reasonably abstract question; for that to happen some entrepreneur would have to sacrifice years of his life to make it happen, and it’s not clear whether someone would.

Here is a fifth: Beyond the gambling houses, only Cultivate Labs and PredictIt are actual businesses, with sustainable revenue. Cultivate Labs in particular is the grownup in the room, where everyone else is chasing sexier but more unlikely dreams. I think in previous years, I didn’t pay enough respect to Cultivate Labs in particular, but as the years pass and they continue existing, I acquire greater respect for them.

Here is a sixth level of analysis: for each platform, consider who is the sucker: if you are winning money, who is losing it? In Las Vegas, the client is the sucker. In Polymarket, the VCs have historically been the “suckers”. They’ve provided the liquidity to incentivize early research for markets, in the hope of capturing market share. Later, unsophisticated users perhaps have been the suckers: there are the sharks that obsessively do research, and there is everybody else. On PredictIt, unsophisticated bettors were the suckers. On play money platforms, like Metaculus, Manifold or INFER, it’s a bit unclear: users come and provide probabilities for free, but they also don’t lose money, and may gain a reputation. Even if they don’t, perhaps writing down their probabilities and resolving questions makes them reach better beliefs. But in some sense, users are also putting in a large amount of time at below market rate, sometimes below minimum wage rate. And so users could become the product, which benefits the platforms, which they don’t control. Maybe something like that happened with the $8M grant to INFER.

Here is a seventh story: I should be more chill; not be focused so much on how things could be better, but more appreciative of how good things are right now. The previous stories are all too negative. Forecasting isn’t special here: there are many other types of businesses where you have freemium models, or where VCs attempt to capture market share. It’s pretty great that Metaculus is providing pretty great probabilities as a public good. VCs and philanthropists like Open Philanthropy do make the pie larger. The feel-good high level story is that forecasting and prediction market communities are exploring the space of how one can provide probabilities, driven by a mix of idealism and greed, and this exploration is a valuable public good.

In general, all of these stories point at something that feels true to me. They are a bit abstract, though. As I write more editions of this newsletter, hopefully I’ll get better intuitions about which ones apply to a greater extent.

The philanthropists

The philanthropic side of forecasting has also been in a wild ride. In 2022, there was an influx of money from the FTX Future Fund, which led a few months of bonanza and a longer period of stressful stewardship of funds as would-be institutions, like the Swift Center, the Quantified Uncertainty Research Institute, Sage Future and others wrangled with the FTX creditor restitution process and struggled to find alternative funding sources¹.

In the past year, the foundation of Dustin Moskovitz, a Facebook cofounder: Open Philanthropy–expanded their presence in the forecasting space by hiring two forecasting grant-makers, to give out $5M or more a year, per their job ad.

I personally was hoping for a dashing, Íñigo Montoya-esque figure, but instead, they have hired an academic (edit: Tereick writes to requests a pointer to his updated academic site) and transferred a previous grant-maker from their AI governance team.

Personally, I used to be pretty into the EA ecosystem. But I’ve become more disillusioned with EA in general and with Open Philanthropy, the main funder of important swathes of it.  My sense is that chasing its funding isn’t always the best game to be playing. I’m not saying don’t take the free billionaire money. But it seems good to not become dependent on it. And, more importantly, between doing what you otherwise would (and perhaps getting funding from Open Philanthropy sometimes), and changing one’s priorities and plans to appeal to it more, my sense is that doing more of the former is better.

Maybe a pithy way to put it is that EA is akin to a court. Being a courtier can be very rewarding. If a duke offers you money, you might want to take it. But being in that court might not be what you want to be doing.

That’s on an individual level, on the margin. When reviewing this draft, a bunch of friends pointed out that I’m not making suggestions about what Open Philanthropy itself could do better. This is because I’m demoralized. As a result, I think that the good move on my part is to speak to individuals who would interact with Open Philanthropy, because those minds I can maybe inform and change.

As is in character for the Effective Altruism community, the announcement that Open Philanthropy was expanding its forecasting grantmaking was met with some criticism, for example by my friend Eli.

Our future AI overlords

The first intersection between AI and forecasting involves using forecasting mores, tools and techniques to try to make sense of how the future of AI will go. My perspective is that forecasting is a bad fit: it’s far into the future, it’s hard to score, current concepts might not capture true dynamics, and in general forecasters have habits more trained on geopolitics. The thing is, though, that other approaches, like hedgehog-style thinking, might be even worse.

The other intersection is that, as AI becomes more capable, it’s starting to become relevant to forecasting practice. A specific example is Future Search, with which I haven’t yet played much. You also can get language models to summarize and triage large swathes of news, and this is what I’m doing at Alert, an emergency response team I’m running/setting up.

In general, AI now seems to still be more of an assistant. In the near future it could become more of a full replacement. Maybe the good forecasters will just orchestrate and double-check swarms of AI models, and offer the firm handshakes. At least for a time. None of these are firm predictions, though.

The forecasters and forecasting teams

The original forecasting team is and remains Good Judgment Incorporated. More recently, you also have Samotsvety and the Swift Centre. These exist in partial competition and partial symbiosis with forecasting platforms: they recruit and compete with them, but they also provide forecasts on them.

Historically, forecasters were underpaid relative to normal professions. Partly this is because, as I mentioned, they were competing with hobbyists who were working for free. This meant churn: as people progressed in their careers, the value of their time rose and people moved on. In recent times, this has been fixed a bit, but the underlying dynamic is similar.

I feel like there is more to say about forecasting teams. My own team, Samotsvety, has been doing well. But I don’t have that many summary dynamics to share at the moment.

The market regulator

Here is a sensationalized history of prediction markets in the United States:

In days of yonder there was ye olde PredictIt and ye elder Intrade, built on top of the US Dollar. Later arrived your good old Augur, which sprang from the calf of Ethereum. But PredictIt and Intrade limited betting amounts. Augur didn’t, but as the price of Ethereum’s currency rose, as Augur resolutions proved slow and cumbersome, and as Augur’s developers moved on to other things after offloading their hot tokens on expectant fans, the people grew restless.

Enter Polymarket, a site built on Matic (now Polygon), a chain that inter-operated with Ethereum, but was cheaper and more convenient. The people flocked to it, and were content, for a time.

But Polymarket’s success and PredictIt’s promise stirred envy in the dark heart of Kalshi, which conjured the black spell of plausibly-deniable revolving doors to set the powers of the CFTC on Polymarket and PredictIt. PredictIt and Polymarket were inconvenienced, for a time, and Kalshi gained bad karma, as their would-be clients noticed that Kalshi’s gain through regulatory capture was their loss.

But there were market forces protecting Polymarket, and they spoke words of power, which read “the Net interprets censorship as damage, and routes around it”, and “regulatory arbitrage”. And so the peasants living under the tyrannic reign of the CFTC got some respite, as the CFTC proved unable or unwilling to build the totalitarian surveillance regime that would be necessary to stop people from making bets over the internet.

There were also words of power protecting PredictIt, but their name was “yo, the CFTC can’t just create a monopoly on political betting in the US and give it to Kalshi, bro” and they were less powerful and so took longer to act. Crucially, though, Kalshi failed to capitalize on the year or two of disruption of its competitors, so now it just has the black karma.

More somberly, the CFTC has a trade-off to make. On the one hand, it could allow sophisticated, ambitious and beneficial financial innovation. In contrast with other instruments like stocks, commodities, currencies—which have recursive entanglements and always postponed investor expectations—prediction markets on binary events can create market incentives for the naked truth. On the other hand, the CFTC can protect vulnerable people from being parted from their money.

The two objectives trade off against each other. My sense is that the balance of utilities leans towards allowing and even incentivizing innovation, but the US isn't really set up for estimating that balance of utilities, but rather to do something akin to minimizing downsides.

It’s also the case that losing money is not a pure downside: it sometimes has a deeply healthy effect of disenchanting oneself with incorrect hypotheses. For example, many believers in QAnon prophecies lost money betting that Trump would be reinstated as president after he lost the 2020 elections. Hopefully those that lost money believed in QAnon prophecies less after that. Whereas more of a nanny regulatory regime allows for less learning like that. And there is the point that the money that is lost isn’t vaporized. Rather, it goes to people with better beliefs, who can use it to correct market probabilities more in the future.

Another dynamic I find really cute is that the stronger the censorship regime, the higher the market demand for censorship-resistant tools, like Tor, VPNs, Monero, or privacy-preserving features of Ethereum. In a sense, the optimal move might be to not censor stuff at all–so as to not allow defense mechanisms to spring up and spread–and then censor all at once. To do something akin to a Hundred Flowers Campaign: pretend to relax enforcement, then crack down suddenly.

Still, could the CFTC ultimately ban online betting, if it decided to? Maybe, but then it would require building something akin to the Great Firewall of China, and that seems repugnant to US values, which cherish liberty. At the same time, though Europe, which is usually more censorious than the US, does allow for political prediction markets, and houses like Betfair do see pretty large volumes.

Anyways, these issues and arguments were more alive when the CFTC was actively making these decisions, but now the situation seems more stable as they have settled on their regulatory rule making and the PredictIt case is making its way through the courts.

Hopes and challenges

I want to see more people using forecasting—quantification, incentive mechanisms to elicit better models of the world—to make better decisions and lead better lives. The forecasting and prediction markets communities are exploring different ways to do this, and I’m keen to continue seeing it happen. I’m particularly keen on seeing attempts to make forecasting eat the world; manifold.love seeed like a respectable try.

This post feels pretty abstract, seeing things from a bird eye’s view. I’m curious whether readers think that these dynamics are accurate, whether they are missing something, or whether there is more stuff that I’m missing. But also, as I start keeping more active tabs on the forecasting ecosystem again, I’ll start bringing up more specifics.

And that’s all I have for you for now.

---

I’m thankful for Thomas Bayes, everyone give it up for Thomas Bayes.

– Sam Trabucco, Nov 25, 2021

1. BTW, in late 2022, it was still tenable for me to believe that FTX had only been fraudulent at the end, or even that Sam Bankman Fried was some misunderstood iconoclast. But with the information that has since been unearthed during the trial, this was not the case, and FTX seems to have been rotten pretty much from the beginning.