Forecasting Newsletter: January 2021
Highlights
1. Veteran PredictIt trader writes a pretty good guide on how to make money on prediction markets.
2. Metaculus and Hypermind both have new COVID-19 forecasting tournaments.
3. I created a search engine for probabilities.
Index
Highlights
Prediction Markets & Forecasting Platforms
In The News
Long Content
Hard To Categorize
You can also view this post on the EA Forum.
Prediction Markets & Forecasting Platforms
Hypermind is an American-French forecasting platform with a somewhat outdated and clunky interface. They have a new COVID-19 Recovery contest with $7000 in promised prizes so far, with the amount set to increase as more questions get added. The contest is sponsored by the Open Philanthropy Project. Hypermind is somewhat difficult to navigate, so you might only be able to find the contest if you create an account and look around.
Metaculus has a new COVID-19 Forecasting contest. From the description:
“The goal of this project is to provide probabilistic predictions of the U.S. COVID-19 outbreak to support public health decision making at the federal and state level.
At the end of each month we will share a summary report with the Council of State and Territorial Epidemiologists, members of the Centers for Disease Control and Prevention, all members of MIDAS (Modeling of Infectious Disease Agent Study), and make this report available for public consumption.”
Metaculus also published an open letter on the urgent need for expanded surveillance and forecasting of novel SAR-CoV-2 variants. An opinion piece saying the same thing is also available on The Hill:
“Efforts are already being made to characterize and understand the infectivity properties and immunological consequences of these new variants. However, as was the case at the start of the pandemic, most countries remain extraordinarily uncertain as to (1) the extent to which these novel variants are spreading and (2) the likelihood as to whether and when these new variants will become predominant.
Unfortunately, these issues do not appear as if they will be extensively addressed in the immediate future. The U.S. CDC, for instance, is currently only planning on having each state send it ‘at least 10 samples’ on a biweekly basis for sequencing and further characterization. This is woefully inadequate genomic surveillance — we are in the dark.
We are calling for a massive increase in genomic sequencing, monitoring, data sharing, and probabilistic forecasting so we can have a detailed understanding of where these new variants are circulating and how rapidly they are increasing as a proportion of all cases.”
On the negative side, Metaculus’s current editor could use some improvement. For example, consider the following aggregate prediction on the state of the art performance on the SuperGLUE AI benchmark:
The current state of the art performance is 90.3 in the SuperGLUE benchmark, making it extremely unlikely that the end result will fall below that number. But, the Metaculus’s aggregate prediction gives a 25% chance to the state of the art falling below that number at question resolution time. This is because the Metaculus interface makes it annoying, or directly impossible, to create one-sided tails.
Omen announced an integration with API3. This will allow for obtaining generally superior resolutions for the price of almost any cryptocurrency. However, Omen is seeing very low trade volumes and very low numbers of active questions.
In contrast, Polymarket has been doing quite well with regards to trade volume. They resolved some of their presidential succession questions, and have probably managed to keep some of the new users in the aftermath. For a while, Polymarket was “an unlimited passive income stream for people who still have their frontal lobes intact” (source). For example, Will Joe Biden be inaugurated as President of the USA on January 20th, 2021? traded at 85 to 93% (!).
But now—unlike in the previous edition of this newsletter—I don’t think that there are any markets which are egregiously wrong after taking into account fees and the hassle of moving relatively small amounts of money into Polymarket. That said, the “No” position on Will Donald Trump be President of the USA on March 31, 2021? is still trading at 97 to 98% (after fees). And, a 2 to 3% return per month, particularly if compounded for a year, still looks pretty good.
Augur, a more decentralized cryptocurrency-based prediction market, has also successfully resolved various US election questions, and has also done better in terms of volume, particularly since its new interface, catnip.exchange, sprung up. However, I haven’t been following them closely.
In other news, I created a search engine for probabilities. It currently aggregates forecasts from PredictIt, Polymarket, Omen, Metaculus, Good Judgment Open, CSET-foretell, Elicit, PredictionBook (through Elicit) and Hypermind. You can access a demo here, or browse a GitHub repository and find out the location of selected API endpoints here. To get a feel of how it works, I suggest searching for “Trump”, “China”, or “semiconductors”. Tentatively, I'll keep both the search engine and the json/csv endpoints updated once a day for the next month. I consider this to be in very early beta: comments and suggestions are welcome.
In the News
Forecasting the New Administration’s Impact on Defense. Despite being quite badly formatted, this piece by a former vice president of combat avionics at Northrop Grumman provides deep expertise and insight on the future shape of US defense spending under the Biden administration. The piece doesn't provide explicit probabilities, but it does give a sense of which scenarios are most likely and which are most worth paying attention to.
Vox looks back at their forecasts from 2020, and they compare favorably to Metaculus’s (source). Vox also offers new predictions for 2021.
Radar technology that could revolutionize hurricane forecasts hits major setback. The US’s National Science Foundation considered the price tag of $70 million insufficiently justified.
“An airborne phased-array radar system consists of thousands of transmitters and receivers spread across four square arrays strategically placed on an aircraft’s fuselage. They scan the sky and “can provide unprecedented detailed observations of the dynamics and microphysics of high-impact storms,” according to an NCAR fact sheet. The data collected by the phased-array radar, when integrated into computer models, could improve forecasts for hurricanes and other hazards investigated by aircraft, including non-hurricane severe weather and winter storms.”
Airports explore new ways to forecast travel amid the pandemic by looking at new indicators, such as the number of people who search for the opening times of the Statue of Liberty, or for rental cars.
AI Startup Sees Opportunity Forecasting Pandemic-Era Consumer Demand using proxies which other companies don’t yet use as much, such as internet searches.
Betting Against QAnon proved particularly profitable for some of the PredicIt traders.
Superforecasters have a look at the end of Covid in Britain.
Hard to Categorize
A small US city deliberates about paying for an expansion to a gunfire locator which would not only detect and report shots fired, but also direct police officers to areas where incidents are predicted to be likely to happen. See also: Minority Report.
Rootclaim is a site which comes up with Bayesian calculations for public interest questions. For example, here is their page on the source of COVID-19: they start with a reasonable prior and then legibly update their initial prediction with each piece of evidence they consider. That said, their conclusion differs from that of Metaculus and from that of casual discussion between several superforecasters on Twitter.
Metaculus user Ege Erdil has produced a heatmap of predicted locations for World War 3 putting together the results of two questions: If there is a WW3, what latitude will it start in? and If there's a WW3, what longitude will it start in? . The source code used to produce the image below is available here. Because the latitude and longitude are given as separate variables, the code uses some kernel wizardry to try to find their degree of correlation, which might introduce some mistakes.
Another Metaculus user and top 50 forecaster, SimonM, has created a page, Metaculus Extras, which presents various statistics about the platform, such as a list of top comments, an h-index (!), and a timeline of Metaculus community predictions.
Long Content
A new paper (summary) tries to quantify by how much entrepreneurs are overconfident when presenting forecasts to potential investors. The authors found ~15% overconfidence in founder CEOs, and ~27% for non-founder CEOs.
“Over the last decade, the toll of dengue fever has increased in New Caledonia, raising questions about the future of the disease in this French island territory located in the South Pacific. Climate has a strong influence on dengue through its influence on the ecology of the vector and the viral cycle. Several studies have explored the link between climate and dengue in New Caledonia, with the aim of explaining and predicting dengue outbreaks. None of these studies have explored the possible outcome climate change will have on the risk of dengue fever in New Caledonia. This is the goal of this study, through projections of rainfall and temperature and the selection of an appropriate prediction target for our statistical model, we assess the climate-induced risk of dengue outbreaks up to the 2100 horizon. We prove that the inter-annual risk of dengue outbreaks in New Caledonia will raise, according to all the greenhouse gas emission scenarios and according to the high emission scenario, dengue fever will become an endemic disease in New Caledonia.”
A recent working paper by the Federal Reserve Bank of Philadelphia introduces a class of disagreement measures for probability distribution forecasts based on the Wasserstein metric (also known as the Earth mover's distance).
Against essential and accidental complexity.
“In the classic 1986 essay, No Silver Bullet, Fred Brooks argued that there is, in some sense, not that much that can be done to improve programmer productivity. His line of reasoning is that programming tasks contain a core of essential/conceptual complexity that's fundamentally not amenable to attack by any potential advances in technology (such as languages or tooling). He then uses an Ahmdahl's law argument, saying that because 1/X of complexity is essential, it's impossible to ever get more than a factor of X improvement via technological improvements. Towards the end of the essay, Brooks claims that at least 1/2 (most) of complexity in programming is essential, bounding the potential improvement remaining for all technological programming innovations combined to, at most, a factor of 2. Let's see how this essential complexity claim holds(...)”
Mining the silver lining of the Trump presidency: A retrospective look at the Trump tweets’ markets in PredictIt.
“For the better part of the last four politically insane years, a community of gamblers wagered stupid amounts of money betting on a simple question: How many times would Donald Trump tweet this week? The game ended for us before it ended for the President, but now that it’s completely over, I feel this tiny corner of Internet weirdness deserves some remembrance. After all, there are very few other people on this planet that understood Trump’s twitter habits – and by extension, Trump himself – more than the people who bet on them.”
That same author has what seems to be a pretty good guide on how to bet on prediction markets if one is optimizing for making money. Something I personally was doing wrong was holding until the end and considering prices in isolation, that is, asking myself “is this price wrong?” rather than “is this price the most wrong it will be?”
The UK's National Risk Register is a document which "provides information on the most significant risks that could occur in the next two years and which could have a wide range of impacts on the UK." Besides this, it also contains a pretty good categorization scheme for risks.
Metaculus lore tells of a legendary comment by user @travisfisher. Written on Jan 24, 2020, under Will the world population increase every year for the next decade?, it reads:
The Wuhan Coronavirus is looking like a pandemic event that could be serious enough to threaten this outcome.
Note to the future: All links are added automatically to the Internet Archive. In case of link rot, go there and input the dead link.
Probability should not be introduced through games of chance. These games are artificial, & give the impression that probability is mostly objective & irreducible (aleatoric). The real problems we face almost always require probability that is subjective & reducible (epistemic).
Source: @maosbot