AI Safety organizations constantly examine how to market “x-risk” messages to targeted audiences based on - political party affiliation, age group, gender, educational level, and residency.
Congrats with your well-researched blogposts. We're happy that you are keenly studying our work.
Clearly, you don't believe that AI can cause human extinction. Are you actually right about that? If you are right about that, we're the first to stop our campaign, and personally (as ERO's founder) I'm happy to start sustainable energy engineering and climate activism again (I made about four times with the engineering part what I'm making now if that's relevant). To make that happen, could you please send us the paper that convincingly shows that human-level AI has a 0% chance of leading to human extinction? That the four independent ways that e.g. Dan Hendrycks found (https://arxiv.org/abs/2306.12001) that could lead to extinction are all wrong?
Unfortunately, that paper doesn't exist. And not for lack of discussion. As you've noted yourself, there is plenty of debate around whether AI can lead to human extinction or not. Given that this is the case, one would presume that there are decent arguments from those working on AI that their invention can obviously (to think of the absurd idea!) never lead to human extinction, because of reasons, A, B, and C. There's a small problem: reasons A, B, and C are missing. I've met many AI researchers, both in industry and academia, who actually agree that yes, our invention could lead to human extinction. Some of them have signed the open statement earlier this year, reading that "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." If that sounds scary, it's because these people are scared. They are scared because they think that yes, AI can actually lead to human extinction. (1/2)
Your argument is disingenuous. You are demanding proof of a negative. Meanwhile, AI Risk hypotheses are mired in philosophy and thought experiments instead of falsifiable proofs by experimentation. The paper you reference is riddled with "AI could" and "AI may" extrapolations. Run actual experiments and come back with results.
I appreciate your intentions here and assume good faith. But I think the problem with your approach is more that when ideas that are uncertain and lack consensus among the experts are floated (and in this case intentionally popularized) among the public, it often leads to special interests eventually hijacking the narratives (often) leading to undesirable outcomes. Thus, I think you'd be far better off spending the capital you have access to in furthering research and trying to establish consensus among (at least the good faith) experts.
I also understand that you might feel consensus among experts can't be established in the time frame you desire. And the hope maybe is to use the resulting panic to slow down everything to allow time for the consensus to emerge. But I would just caution that once the public makes up it's mind about a topic (as you might be aware from your time with climate activism), it's very hard to change their opinions. And that might lead to a prolonged state of stasis and slow down. This prolonged stasis can have it's own set of negative second order consequences as you might well be aware (depriving developing economies of progress, preventing (possible) progress towards other fruitful ventures like mitigating climate change, etc.)
Furthermore, I would also caution that the resulting panic in the public could also be leveraged by bad actors/special interests to slow down progress for other (potentially good) actors (who might want to try and develop solutions) while continuing to make progress themselves. Note that even the resulting panic and chaos can easily be leveraged in this way without having to really need any real regulation. All this is to say that the consequences of your actions can also lead to existential risk (i.e increase the probability of risk 1 from the paper you cite). And it's unlikely that you'll be able to give arguments that prove with 100% certainty that your actions wont lead to an increase on any of those risks.
I do realize that these (incomplete and simple) arguments are unlikely to convince you and that you have probably heard these arguments from other people. But I hope by reinforcing these arguments, I'm at least able to bring home some source of uncertainty about the outcomes that your actions might actually achieve. Reminding oneself of the inherent uncertainties involved here is super important to ensure that we don't get carried away in our own arguments and keep questioning our every decision.
> To make that happen, could you please send us the paper that convincingly shows that human-level AI has a 0% chance of leading to human extinction?
This is an impossible thing to ask for, presumably you see that. The argument isn't to search for perfect certainty, which doesn't exist in this world about anything, but rather a cost-benefit analysis that would tell us if you are correct.
Meanwhile, I personally would love a model of what you think will actually happen (asked for here https://www.strangeloopcanon.com/p/the-price-of-progress) as do several others who have also asked for this, not repeated multiple arguments from authority about weighted beliefs from experts or AI engineer surveys.
The best study of the extinction risk of AI by 2100 asked 88 superforecasters, people selected by a track record of making better predictions than you or me. Those 88 superforecasters give a risk of 0.38% of human extinction from AI by 2100.
Of course, you know that, don't you? But it doesn't suit your narrative, so you'd prefer we ignore it, so you can keep inflicting your nightmares on the body politic.
I do know that. 88 superforecasters can't replace humanity, though. AI researchers give estimates that are a lot higher. Is that because they know something the superforecasters don't? Or is it because they're worse at putting probabilities on future events? I don't know. Do you?
In any case, 0.38% chance of human extinction is still higher than climate (likely), nuclear war (likely), asteroid strikes (definitely) and supervolcanoes (definitely). Statistically, it's 8e9* .0038 = 30 million deaths. That's only counting current humans, not current animals, future humans, future animals, or just the tragedy of lost history. It's not a small number, actually.
Also, I don't know why they estimate 0.38%. If it would be me doing the estimating, I would guess that a significant part of why we might survive is _because_ of awareness. Awareness is awesome: it's the start of us doing all kinds of risk-reducing things. Without awareness, there would be no AI Safety research, no Bletchley process, no evals, nothing. And we're only at the beginning. If we survive, I think there's a high probability that's because of some kind of measure someone has implemented because at least they were aware of the fucking problem. That should be step zero in any risk management strategy and I'm not going to apologize for helping to bring it about.
Other AI researchers do not believe we should worry about human extinction because of AI. While in many cases admitting that human-level AI can cause extinction, they tend to give responses such as "don't worry: that's far in the future! And we're not even sure if we'll ever make it haha." When you ask how far that is exactly, they might say "well maybe as much as twenty years!" Those researchers don't tend to think about the fact that we might well need twenty years, or more, to debate the issue in society (which took three decades plus for climate), do the required research to somehow find a safe solution, and implement that solution.
So thanks for your 'exposé'. We're in no way hiding the fact that we think there should be a societal debate about the human extinction risk of AI, nor have we ever hidden this. If you think there is nothing to be concerned of, please forward us that paper. (2/2)
Predicating your condition on proving a negative just equivocated your AI Risk with dangerous turnips. Because you haven't shown a paper that shows a 0% chance that turnips cause human extinction.
There are very clear simple arguments I can spell out for why AI has a high chance of causing human extinction. If you disagree AI has a high chance of causing human extinction, you need to poke holes in those arguments. Curiously people who deny x-risk from AI never do this.
I expect you to know the basic line of argumentation already, but the reason AI will likely kill us all is
1. We will get smarter than human AI soon (within the next few decades)
2. Agents smarter than us can get their desired goals even if those goals contradict ours.
3. Most goals, if optimized hard enough, end up killing us.
4. We currently have no idea how to put any goal into an AI, so when we train AIs they end up with some random inscrutable goal.
5. We all die.
Again, I've never seen anyone seriously engage with these points. I challenge you to do it. Give object level responses, don't make meta-points.
If your argument has the longest history, out of any argument, of being completely wrong, at what point do you take a moment to self-reflect about what makes you so special where every other prognosticator of doom has failed?
I find your position generic, and detrimental to any concept of AI safety because it glorifies your virtue signaling at the cost of numbing the public; spending precious attention when it could be used to detail real harms when they arise.
I find none of your points reasonable, and yet mine has the entire empirical weight of record. Grift on!
(You ignored my arguments and made a meta level point... ) My arguments don't have any history of being wrong. They are novel arguments. They only have a history of being wrong if you clump together all arguments pertaining to human extinction as the same argument. Which is obviously EXTREMELY SILLY.
If an asteroid the size of the moon was hurling towards earth and was 1 minute away, you could make the exact same argument you're making right now.
"The asteroid is coming! Its going to blow up the earth and we're all gonna die!! Look in the sky, you can see it!! We're gonna die!!!"
"Well, actually, your argument has the longest history, out of any argument of being wrong...... Grift on!"
Hi, just wanted to say as a normal person, I really appreciate the depth you went into with this post and all of your other posts. That’s all! They’re so informative and a great deep dive.
AI may be intelligent, but it doesn't have any will. It isn't driven by emotions. Animals, including humans, eat, sleep, breath and reproduce by sex. We don't have the same driving force/incentives. Computers just need electricity, it has no need for status, honor or any of the things that cause humans to fight each other. And why would it want to take over the world? Is it rational to want to control humans?
If it start to feel things like humans do, it may become dangerous. But it is always possible to ask it to prove it's conclusions by showing us how they reached it.To sShow us the logic behind it's conclusion, or provide evidence for what it says.
It can create disinformation, but it can also help us sort out disinformation. Now that we don't have a common perception of reality we may use AI to help us overcome the polarization between us.
Congrats with your well-researched blogposts. We're happy that you are keenly studying our work.
Clearly, you don't believe that AI can cause human extinction. Are you actually right about that? If you are right about that, we're the first to stop our campaign, and personally (as ERO's founder) I'm happy to start sustainable energy engineering and climate activism again (I made about four times with the engineering part what I'm making now if that's relevant). To make that happen, could you please send us the paper that convincingly shows that human-level AI has a 0% chance of leading to human extinction? That the four independent ways that e.g. Dan Hendrycks found (https://arxiv.org/abs/2306.12001) that could lead to extinction are all wrong?
Unfortunately, that paper doesn't exist. And not for lack of discussion. As you've noted yourself, there is plenty of debate around whether AI can lead to human extinction or not. Given that this is the case, one would presume that there are decent arguments from those working on AI that their invention can obviously (to think of the absurd idea!) never lead to human extinction, because of reasons, A, B, and C. There's a small problem: reasons A, B, and C are missing. I've met many AI researchers, both in industry and academia, who actually agree that yes, our invention could lead to human extinction. Some of them have signed the open statement earlier this year, reading that "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." If that sounds scary, it's because these people are scared. They are scared because they think that yes, AI can actually lead to human extinction. (1/2)
Your argument is disingenuous. You are demanding proof of a negative. Meanwhile, AI Risk hypotheses are mired in philosophy and thought experiments instead of falsifiable proofs by experimentation. The paper you reference is riddled with "AI could" and "AI may" extrapolations. Run actual experiments and come back with results.
I appreciate your intentions here and assume good faith. But I think the problem with your approach is more that when ideas that are uncertain and lack consensus among the experts are floated (and in this case intentionally popularized) among the public, it often leads to special interests eventually hijacking the narratives (often) leading to undesirable outcomes. Thus, I think you'd be far better off spending the capital you have access to in furthering research and trying to establish consensus among (at least the good faith) experts.
I also understand that you might feel consensus among experts can't be established in the time frame you desire. And the hope maybe is to use the resulting panic to slow down everything to allow time for the consensus to emerge. But I would just caution that once the public makes up it's mind about a topic (as you might be aware from your time with climate activism), it's very hard to change their opinions. And that might lead to a prolonged state of stasis and slow down. This prolonged stasis can have it's own set of negative second order consequences as you might well be aware (depriving developing economies of progress, preventing (possible) progress towards other fruitful ventures like mitigating climate change, etc.)
Furthermore, I would also caution that the resulting panic in the public could also be leveraged by bad actors/special interests to slow down progress for other (potentially good) actors (who might want to try and develop solutions) while continuing to make progress themselves. Note that even the resulting panic and chaos can easily be leveraged in this way without having to really need any real regulation. All this is to say that the consequences of your actions can also lead to existential risk (i.e increase the probability of risk 1 from the paper you cite). And it's unlikely that you'll be able to give arguments that prove with 100% certainty that your actions wont lead to an increase on any of those risks.
I do realize that these (incomplete and simple) arguments are unlikely to convince you and that you have probably heard these arguments from other people. But I hope by reinforcing these arguments, I'm at least able to bring home some source of uncertainty about the outcomes that your actions might actually achieve. Reminding oneself of the inherent uncertainties involved here is super important to ensure that we don't get carried away in our own arguments and keep questioning our every decision.
> To make that happen, could you please send us the paper that convincingly shows that human-level AI has a 0% chance of leading to human extinction?
This is an impossible thing to ask for, presumably you see that. The argument isn't to search for perfect certainty, which doesn't exist in this world about anything, but rather a cost-benefit analysis that would tell us if you are correct.
Meanwhile, I personally would love a model of what you think will actually happen (asked for here https://www.strangeloopcanon.com/p/the-price-of-progress) as do several others who have also asked for this, not repeated multiple arguments from authority about weighted beliefs from experts or AI engineer surveys.
The best study of the extinction risk of AI by 2100 asked 88 superforecasters, people selected by a track record of making better predictions than you or me. Those 88 superforecasters give a risk of 0.38% of human extinction from AI by 2100.
Of course, you know that, don't you? But it doesn't suit your narrative, so you'd prefer we ignore it, so you can keep inflicting your nightmares on the body politic.
https://bigthink.com/the-future/ai-extinction-predictions-superforecasting/
I do know that. 88 superforecasters can't replace humanity, though. AI researchers give estimates that are a lot higher. Is that because they know something the superforecasters don't? Or is it because they're worse at putting probabilities on future events? I don't know. Do you?
In any case, 0.38% chance of human extinction is still higher than climate (likely), nuclear war (likely), asteroid strikes (definitely) and supervolcanoes (definitely). Statistically, it's 8e9* .0038 = 30 million deaths. That's only counting current humans, not current animals, future humans, future animals, or just the tragedy of lost history. It's not a small number, actually.
Also, I don't know why they estimate 0.38%. If it would be me doing the estimating, I would guess that a significant part of why we might survive is _because_ of awareness. Awareness is awesome: it's the start of us doing all kinds of risk-reducing things. Without awareness, there would be no AI Safety research, no Bletchley process, no evals, nothing. And we're only at the beginning. If we survive, I think there's a high probability that's because of some kind of measure someone has implemented because at least they were aware of the fucking problem. That should be step zero in any risk management strategy and I'm not going to apologize for helping to bring it about.
Other AI researchers do not believe we should worry about human extinction because of AI. While in many cases admitting that human-level AI can cause extinction, they tend to give responses such as "don't worry: that's far in the future! And we're not even sure if we'll ever make it haha." When you ask how far that is exactly, they might say "well maybe as much as twenty years!" Those researchers don't tend to think about the fact that we might well need twenty years, or more, to debate the issue in society (which took three decades plus for climate), do the required research to somehow find a safe solution, and implement that solution.
So thanks for your 'exposé'. We're in no way hiding the fact that we think there should be a societal debate about the human extinction risk of AI, nor have we ever hidden this. If you think there is nothing to be concerned of, please forward us that paper. (2/2)
Predicating your condition on proving a negative just equivocated your AI Risk with dangerous turnips. Because you haven't shown a paper that shows a 0% chance that turnips cause human extinction.
There are very clear simple arguments I can spell out for why AI has a high chance of causing human extinction. If you disagree AI has a high chance of causing human extinction, you need to poke holes in those arguments. Curiously people who deny x-risk from AI never do this.
I expect you to know the basic line of argumentation already, but the reason AI will likely kill us all is
1. We will get smarter than human AI soon (within the next few decades)
2. Agents smarter than us can get their desired goals even if those goals contradict ours.
3. Most goals, if optimized hard enough, end up killing us.
4. We currently have no idea how to put any goal into an AI, so when we train AIs they end up with some random inscrutable goal.
5. We all die.
Again, I've never seen anyone seriously engage with these points. I challenge you to do it. Give object level responses, don't make meta-points.
If your argument has the longest history, out of any argument, of being completely wrong, at what point do you take a moment to self-reflect about what makes you so special where every other prognosticator of doom has failed?
I find your position generic, and detrimental to any concept of AI safety because it glorifies your virtue signaling at the cost of numbing the public; spending precious attention when it could be used to detail real harms when they arise.
I find none of your points reasonable, and yet mine has the entire empirical weight of record. Grift on!
(You ignored my arguments and made a meta level point... ) My arguments don't have any history of being wrong. They are novel arguments. They only have a history of being wrong if you clump together all arguments pertaining to human extinction as the same argument. Which is obviously EXTREMELY SILLY.
If an asteroid the size of the moon was hurling towards earth and was 1 minute away, you could make the exact same argument you're making right now.
"The asteroid is coming! Its going to blow up the earth and we're all gonna die!! Look in the sky, you can see it!! We're gonna die!!!"
"Well, actually, your argument has the longest history, out of any argument of being wrong...... Grift on!"
Reflect on that...
Hi, just wanted to say as a normal person, I really appreciate the depth you went into with this post and all of your other posts. That’s all! They’re so informative and a great deep dive.
Thank you so much! I'm glad you found it useful.
AI may be intelligent, but it doesn't have any will. It isn't driven by emotions. Animals, including humans, eat, sleep, breath and reproduce by sex. We don't have the same driving force/incentives. Computers just need electricity, it has no need for status, honor or any of the things that cause humans to fight each other. And why would it want to take over the world? Is it rational to want to control humans?
If it start to feel things like humans do, it may become dangerous. But it is always possible to ask it to prove it's conclusions by showing us how they reached it.To sShow us the logic behind it's conclusion, or provide evidence for what it says.
It can create disinformation, but it can also help us sort out disinformation. Now that we don't have a common perception of reality we may use AI to help us overcome the polarization between us.
I believe that humans are going to extinct because of
1) environment degradation
2) resource depletion
3) technologic stagnation and education degradation
with much higher probability than hypotetical risc from "mad" or "rogue" AI.
Stopping or slowing progress is the most dangerous, almost suicidal idea of some rich people from rich countries.
I expect AI to be made. Get to superintelligence. And destroy the world. (probably with nanobots)
A lot of people expect Jesus Christ to return to judge the living and the dead. What's your point?