I agree with you that "the latter" are worthy causes.
But I wouldn't discount Good Judgement / Tetlock and Hypermind just because they (we? I've been contracting with GJ since 2015 and GJP has made my reputation, so I am very much part of that; trust me at your peril) are corporate forecasting mercenaries.
All platforms struggle with weak policymaker interest because everyone thinks they're the best forecaster ever. The world seems to be in agreement that forecasting (all of research, actually) should be a "free" goody to come with marketing your main product. All platforms struggle retaining good talent: forecasting doesn't pay. And all forecasting markets struggle with regulatory problems, chains that need to be broken, but no one knows how for now. Crowd forecasts are public goods at the end of the day, and as such they will all need subsidizing forever I think. Unless someone figures out how to make a big news site with tons of good content and content related marketing on it, targeted to consumers of geopolitical forecasts and (maybe, partly) paywalled.
It's a chicken and egg problem. Clients say "give me interesting targeted forecasts now, then I'll see if I pay you later". Platforms say "give me money now and I'll build your targeted forecasts".
Clients never come around to pay though, because their ego, politics, and prestige are hurt when they discover they're actually not the best forecasters. Forecasting in companies is the CEOs job. In a round with some sweets industry CEOs I was in a few years back, the group forecast that cocoa prices would fall soon. That was right after one of the CEOs boasted he had secured his cocoa supply and price for two years just before the meeting with a long term contract.
What's the endgame? Considering the lack of take-off in the last seven years, geopolitical forecasting will always remain a (fun, for forecasters like me) small niche within foreign affairs reporting, which is itself a small niche in media reporting. In acute "crisis times" or "election times" there will be acute demand, but no willingness to pay and no systematic advantage of good forecasters over the general crowd. Superforecasters are indeed useful mostly for the 6m-2y advance frame, because that's when nobody else is interested in the question. And they're not very useful when deep liquid markets already exist.
Thanks for the counterbalancing opinion! I think I agree with you on the general point that GJP shouldn't be discounted, but disagree on some of the specifics.
> All platforms struggle retaining good talent: forecasting doesn't pay
Then they could pay more. Hindenburg Research or 538, or don't seem like it has problems making its research public.
If Good Judgment had been willing to bet large amounts on prediction markets (such as those on the Ukraine invasion or on the Olympics), it could have both made a lot of money and revealed its forecasts (after correcting the prices to what they think is correct). One could imagine other models as well. They would require a different ecosystem, though.
> And all forecasting markets struggle with regulatory problems, chains that need to be broken, but no one knows how for now
People do know how. One can either do it the hard way, like Kalshi, or one can go outside the US jurisdiction, or into crypto.
> Unless someone figures out how to make a big news site with tons of good content and content related marketing on it, targeted to consumers of geopolitical forecasts and (maybe, partly) paywalled.
I notice that 538 could afford to make really good political forecasts and making them public. Likewise with Hindenburg Research, or Wikileaks. This doesn't seem like it would require new physics, just trying a bunch of new stuff.
> It's a chicken and egg problem. Clients say "give me interesting targeted forecasts now, then I'll see if I pay you later". Platforms say "give me money now and I'll build your targeted forecasts".
The solution to this is to pay based on performance. "Give me money later in proportion to how much our forecasters beat your current best guess numerical forecast" (and perhaps we pay you money, or don't accept the contract, if we don't think we can improve much on your initial probabilities).
> the group forecast that cocoa prices would fall soon. That was right after one of the CEOs boasted he had secured his cocoa supply and price for two years just before the meeting with a long term contract.
Uuf.
> Considering the lack of take-off in the last seven years, geopolitical forecasting will always remain a ... small niche within foreign affairs reporting, which is itself a small niche in media reporting.
One could also have said "Considering the lack of take-off in the last seven years, geopolitical forecasting should do something different". Forecasting was geopolitical forecasting because it was seeded by IARPA, but then the intelligence community didn't want to pay for more of it. So it could stop being geopolitical forecasting and instead try to forecast other fields of importance that people are interested in (sociopolitical forecasting? philanthropic forecasting? business forecasting? etc.)
True that, we forecasters never seem to be able to think big enough.
It's in our genes, as they say.
Forecasters would have told Trump after the first primaries "it makes no sense for you to run for president. You're a longshot, and even if you make it, you will find it is a terrible experience both for you and the nation." Yet he ran, he won, and it was an all around terrible experience.
Forecasters would have told Putin "Don't invade Ukraine. Your chances of success are a long shot, and you will find it a terrible experience for you and Russia." Yet he invaded.
Forecasting doesn't have much policy sway these days. You advocate expanding it to other fields beyond geopolitics. But if we don't even manage to make a difference in geopolitics, can we be sure about our success chances in other fields?
I was thinking of 538 too. Nate Silver does marry excellent forecasting ability and excellent marketing ability. The combination of those skills is possible and powerful, but rare apparently. Maybe forecasting markets take off. If they do, it will be due to genius in marketing crowd forecasting insight to the millions of general public out here, as superior to (or at least as important as) the popular journalistic/expert interview formats. The forecasters themselves may or may not have much insight into the marketing angle, since sometimes they will be at odds with it. For instance when good marketing would require unwarranted contrarianism or pandering to the crowd.
Thank you for maintaining this newsletter.
Forgot to mention that Eli Lifland's $4k impactful forecasting price is still alive until the 11th of March: https://forum.effectivealtruism.org/posts/HDoMrQFG76QtkdrZJ/impactful-forecasting-prize-for-forecast-writeups-on-curated
I agree with you that "the latter" are worthy causes.
But I wouldn't discount Good Judgement / Tetlock and Hypermind just because they (we? I've been contracting with GJ since 2015 and GJP has made my reputation, so I am very much part of that; trust me at your peril) are corporate forecasting mercenaries.
All platforms struggle with weak policymaker interest because everyone thinks they're the best forecaster ever. The world seems to be in agreement that forecasting (all of research, actually) should be a "free" goody to come with marketing your main product. All platforms struggle retaining good talent: forecasting doesn't pay. And all forecasting markets struggle with regulatory problems, chains that need to be broken, but no one knows how for now. Crowd forecasts are public goods at the end of the day, and as such they will all need subsidizing forever I think. Unless someone figures out how to make a big news site with tons of good content and content related marketing on it, targeted to consumers of geopolitical forecasts and (maybe, partly) paywalled.
It's a chicken and egg problem. Clients say "give me interesting targeted forecasts now, then I'll see if I pay you later". Platforms say "give me money now and I'll build your targeted forecasts".
Clients never come around to pay though, because their ego, politics, and prestige are hurt when they discover they're actually not the best forecasters. Forecasting in companies is the CEOs job. In a round with some sweets industry CEOs I was in a few years back, the group forecast that cocoa prices would fall soon. That was right after one of the CEOs boasted he had secured his cocoa supply and price for two years just before the meeting with a long term contract.
What's the endgame? Considering the lack of take-off in the last seven years, geopolitical forecasting will always remain a (fun, for forecasters like me) small niche within foreign affairs reporting, which is itself a small niche in media reporting. In acute "crisis times" or "election times" there will be acute demand, but no willingness to pay and no systematic advantage of good forecasters over the general crowd. Superforecasters are indeed useful mostly for the 6m-2y advance frame, because that's when nobody else is interested in the question. And they're not very useful when deep liquid markets already exist.
Thanks for the counterbalancing opinion! I think I agree with you on the general point that GJP shouldn't be discounted, but disagree on some of the specifics.
> All platforms struggle retaining good talent: forecasting doesn't pay
Then they could pay more. Hindenburg Research or 538, or don't seem like it has problems making its research public.
If Good Judgment had been willing to bet large amounts on prediction markets (such as those on the Ukraine invasion or on the Olympics), it could have both made a lot of money and revealed its forecasts (after correcting the prices to what they think is correct). One could imagine other models as well. They would require a different ecosystem, though.
> And all forecasting markets struggle with regulatory problems, chains that need to be broken, but no one knows how for now
People do know how. One can either do it the hard way, like Kalshi, or one can go outside the US jurisdiction, or into crypto.
> Unless someone figures out how to make a big news site with tons of good content and content related marketing on it, targeted to consumers of geopolitical forecasts and (maybe, partly) paywalled.
I notice that 538 could afford to make really good political forecasts and making them public. Likewise with Hindenburg Research, or Wikileaks. This doesn't seem like it would require new physics, just trying a bunch of new stuff.
> It's a chicken and egg problem. Clients say "give me interesting targeted forecasts now, then I'll see if I pay you later". Platforms say "give me money now and I'll build your targeted forecasts".
The solution to this is to pay based on performance. "Give me money later in proportion to how much our forecasters beat your current best guess numerical forecast" (and perhaps we pay you money, or don't accept the contract, if we don't think we can improve much on your initial probabilities).
> the group forecast that cocoa prices would fall soon. That was right after one of the CEOs boasted he had secured his cocoa supply and price for two years just before the meeting with a long term contract.
Uuf.
> Considering the lack of take-off in the last seven years, geopolitical forecasting will always remain a ... small niche within foreign affairs reporting, which is itself a small niche in media reporting.
One could also have said "Considering the lack of take-off in the last seven years, geopolitical forecasting should do something different". Forecasting was geopolitical forecasting because it was seeded by IARPA, but then the intelligence community didn't want to pay for more of it. So it could stop being geopolitical forecasting and instead try to forecast other fields of importance that people are interested in (sociopolitical forecasting? philanthropic forecasting? business forecasting? etc.)
True that, we forecasters never seem to be able to think big enough.
It's in our genes, as they say.
Forecasters would have told Trump after the first primaries "it makes no sense for you to run for president. You're a longshot, and even if you make it, you will find it is a terrible experience both for you and the nation." Yet he ran, he won, and it was an all around terrible experience.
Forecasters would have told Putin "Don't invade Ukraine. Your chances of success are a long shot, and you will find it a terrible experience for you and Russia." Yet he invaded.
Forecasting doesn't have much policy sway these days. You advocate expanding it to other fields beyond geopolitics. But if we don't even manage to make a difference in geopolitics, can we be sure about our success chances in other fields?
I was thinking of 538 too. Nate Silver does marry excellent forecasting ability and excellent marketing ability. The combination of those skills is possible and powerful, but rare apparently. Maybe forecasting markets take off. If they do, it will be due to genius in marketing crowd forecasting insight to the millions of general public out here, as superior to (or at least as important as) the popular journalistic/expert interview formats. The forecasters themselves may or may not have much insight into the marketing angle, since sometimes they will be at odds with it. For instance when good marketing would require unwarranted contrarianism or pandering to the crowd.