Why would we expect that prediction markets where one outcome results in all parties dying and the prediction market and money in general being destroyed are remotely meaningful or informative assessments of AI x-risk? Even on Manifund, I've seen people saying "free internet points, if I'm wrong I'm dead" when betting against AI extinction. Why would people put up real money betting on a position they literally cannot win?
Like there are reasons other than AI doom for e.g. the 10% reduction in population over 5 years by 2100, but the numbers you get are going to be excluding by far the most likely cause of the predicted outcome in a highly misleading way.
Those would be really nice, but yeah, I don't expect to get ones which are super nicely entangled with the core risk model. I think you've got to get into the gears, and reality doesn't have to be a friendly shape for nice forecasting, and in fact seems not to be for the most important questions.
You can do some things like "AIs will convergently seek self-preservation" or "humans will recklessly hand over control of lots of things for convenience" or "massive increases in AI R&D spending", but those seem behind us already, even if called in advance, and aren't extremely close to the main cruxes of the serious doom argument.
(the first is a little close, and I stake a fair amount of epistemic points on AI memetics turning towards power-seeking in some not fun ways over the next year. and i guess "line keeps going up" on benchmarks rather than hitting a wall is somewhat linked, though i'd guess most other models also predict this.)
Updating on latent variables is real and great when possible, but some latent variables don't show empirically testable signs until it's too late. Your post argues that this can work, but I don't see an argument that reality must always give you nice evidence in this form about everything important, and I suspect if you try to write a post examining that hypothesis fairly you'll find it's unfortunately not something you can rely on.
One fun related conversation is Yud and Paul Christiano trying to find a bettable crux on their fairly different beliefs, it's pretty difficult to find a divergence between a model which predicts one thing and the same thing until a point where humans get disempowered and die which shows up in the data, which forces you to rely on building a generative model of the actual dynamics at play and seeing which is more natural/simple/fitting with general world models, not getting to run nice tests.
To be clear, this sucks! It's wonderful when you can do empiricism. But reality doesn't have to give you the wonderful easy thing.
(also, even if you get some updates here, which you do to at least some small extent, this doesn't remove the fact that you can get horribly misleading prediction market numbers due to the everyone dies on one side of it effect)
> He then got disposed of in a helicopter accident soon afterwards.
It was a private jet, with the other Wagner executives on board, leaving Moscow; likely a bomb at 28,000 feet. I found it clarifying.
He simply stopped the March on Moscow after destroying five RAF aircraft, and a few weeks later was traveling around Russia like nothing had happened, while Kremlin spokesman Peskov said "No, we are not tracking his [Prigozhin’s] whereabouts, have no possibilities and desire to do so". That seemed weird, but Prigozhin was a propaganda king first and military leader second. Evidently he viewed his March on Moscow as part of that theatre, and then Putin worked to convince him, falsely, that the two of them had come to an understanding. (I'm sure Prigozhin also knew the march was very risky, but the Kremlin was sidelining him by absorbing Wagner into MoD, which was also risky for him.) Perhaps the intended audience of Peskov's quote was precisely those executives on the plane.
When I have the cash I'll cut you a check.
Thx mate
Blocked for AI reply
Why would we expect that prediction markets where one outcome results in all parties dying and the prediction market and money in general being destroyed are remotely meaningful or informative assessments of AI x-risk? Even on Manifund, I've seen people saying "free internet points, if I'm wrong I'm dead" when betting against AI extinction. Why would people put up real money betting on a position they literally cannot win?
Like there are reasons other than AI doom for e.g. the 10% reduction in population over 5 years by 2100, but the numbers you get are going to be excluding by far the most likely cause of the predicted outcome in a highly misleading way.
Because you can have markets on falsifiable indicators of risk before the risk happen. See <https://nunosempere.com/blog/2023/05/11/updating-under-anthropic-effects/> on the general point. Btw if you can't find falsifiable indicators for your beliefs that's a terrible sign.
Those would be really nice, but yeah, I don't expect to get ones which are super nicely entangled with the core risk model. I think you've got to get into the gears, and reality doesn't have to be a friendly shape for nice forecasting, and in fact seems not to be for the most important questions.
You can do some things like "AIs will convergently seek self-preservation" or "humans will recklessly hand over control of lots of things for convenience" or "massive increases in AI R&D spending", but those seem behind us already, even if called in advance, and aren't extremely close to the main cruxes of the serious doom argument.
(the first is a little close, and I stake a fair amount of epistemic points on AI memetics turning towards power-seeking in some not fun ways over the next year. and i guess "line keeps going up" on benchmarks rather than hitting a wall is somewhat linked, though i'd guess most other models also predict this.)
Updating on latent variables is real and great when possible, but some latent variables don't show empirically testable signs until it's too late. Your post argues that this can work, but I don't see an argument that reality must always give you nice evidence in this form about everything important, and I suspect if you try to write a post examining that hypothesis fairly you'll find it's unfortunately not something you can rely on.
One fun related conversation is Yud and Paul Christiano trying to find a bettable crux on their fairly different beliefs, it's pretty difficult to find a divergence between a model which predicts one thing and the same thing until a point where humans get disempowered and die which shows up in the data, which forces you to rely on building a generative model of the actual dynamics at play and seeing which is more natural/simple/fitting with general world models, not getting to run nice tests.
To be clear, this sucks! It's wonderful when you can do empiricism. But reality doesn't have to give you the wonderful easy thing.
(also, even if you get some updates here, which you do to at least some small extent, this doesn't remove the fact that you can get horribly misleading prediction market numbers due to the everyone dies on one side of it effect)
Good article!
> He then got disposed of in a helicopter accident soon afterwards.
It was a private jet, with the other Wagner executives on board, leaving Moscow; likely a bomb at 28,000 feet. I found it clarifying.
He simply stopped the March on Moscow after destroying five RAF aircraft, and a few weeks later was traveling around Russia like nothing had happened, while Kremlin spokesman Peskov said "No, we are not tracking his [Prigozhin’s] whereabouts, have no possibilities and desire to do so". That seemed weird, but Prigozhin was a propaganda king first and military leader second. Evidently he viewed his March on Moscow as part of that theatre, and then Putin worked to convince him, falsely, that the two of them had come to an understanding. (I'm sure Prigozhin also knew the march was very risky, but the Kremlin was sidelining him by absorbing Wagner into MoD, which was also risky for him.) Perhaps the intended audience of Peskov's quote was precisely those executives on the plane.
Thanks for the clarification/correction!