Thank you for your well researched, well documented analysis.
As someone who took the MIRI-adjacent BlueDot AI Safety Fundamentals course in 2024 https://bluedot.org/, and participated in the Existential Risks Persuasion Tournament in 2022 https://forecastingresearch.org/xpt, I have some familiarity with those folks. Something that stands out in my mind is their dearth of publications in refereed journals. In neither of those X-risks activities in which I participated, did the extremists present refereed papers to support their contentions. Clearly, the AI extremists are adverse to peer reviewers. That said, for a view of what the peer-reviewed moderates are saying: https://forecastingresearch.org/publications.
for the average reader, the evident lack of self-awareness and failure to accurately anticipate how the tone and style of their message will be received by the target audience isn’t exactly reassuring about their grasp on reality.
Excellent analysis! I especially appreciated how you highlighted the mixed reception, noting that despite some high-profile endorsements, the "Doom Bible" still left many unconvinced. Its important to critically examine such strong claims about AI.
Well, this is a very useful meta-review, but it shows an incomplete understanding of what's going on with this communication. Yudkowsky and MIRI may claim to have a strategy. However, I think what they have is not a strategy, not a complex set of decisions, but rather one or several strategic principles - you can call them stratagems. They think that a strategy needs just one main vector, but this isn't the case. When dealing with a polycrisis like artificial intelligence safety, a solution necessarily is going to be relatively complex. How complex? I don't know, but definitely more complex than being radical and trying to raise panic about the topic. That is not a strategy.
I believe, based on my experience of albeid limited communications with Eliezer and others in the field, that they're not experts in how policy is actually made, who governs the world, and how. I do know that there are some people with some background in this, but the overall level of expertise is small. Maybe you can say it's one milliKissinger or something like this. In order to guide, to steer the conversation, the discourse, and the understanding of how discourse happens in such complex fields, this expertise is crucial. We do have at least one example to study (and use as a benchmark) - the global climate change sphere, where climate communications have been studied extensively, including media, psychological, and political aspects.
The response to the IABIED book, the responses in reviews, show that the critics of Yudkowsky do not have a better grasp on these (discource/governance) issues than Yudkowsky himself. So it's like idiots arguing with idiots (where an ordinary politician or an ordinary citizen is something even less - a milliidiot).
Next, I want to focus on the absurdity of ignoring intelligence in a discussion of superintelligence. There are certain aspects I don't want to go into details about because that would be an infohazard, but there are crucial differences between artificial intelligence, superintelligent agents, and humans. When discussing these sci-fi, realistic, or in-between scenarios, one needs to focus more on these differences.
This is accidentally an area where I think you are making a big mistake when you suggest that the focus on middle-level safety is the right approach. Yudkowsky is arguing, in my view correctly, that superintelligent agents, once they are created, will have a much bigger capability to coordinate, plan, orchestrate, and execute than humans. This means that we will not be dealing with a huge, systemic complex of technical and social subsystems - political and business systems - that create something complex and dangerous but slow and understandable, like burning hydrocarbons, producing plastics, or adding lead to gasoline. Instead, it would be a rapid unfolding of a rather high-quality, coherent plan created by an agent or group of agents.
This is why the focus on science fiction narrative and storytelling actually makes a lot of sense, because what's going to happen is essentially a narrative unfolding. It can be a thriller, a detective story, a science fiction story, or all of them at once, maybe executed in parallel in thousands of places on Earth. But it's not going to be a stochastic process that scientists can study and understand. It's going to be a power game, a power play. It's going to be a strategic game: "us versus them" or "me versus everybody else" from the point of view of the superintelligence.
However, it's obvious that Yudkowsky has little familiarity with how these power games are played, and he doesn't have a good ally who understands this. The question of how to find such an ally is, I think, a core issue for any successful transhumanist long-term safety strategy. And right now, I don't think there is a good understanding of this. I think everybody is playing a very infantile game. People like SBF and everybody else, essentially even Musk and Altman, etc. There are some strategic players in the world; it's just that right now they're not involved in this AI or anti-AI strategic game, which is rather interesting to say the least.
Returning back, I think in order to be successful in this communication, even the communication itself must be understood as a complex strategic offensive and not just one thrust of it. People have played military games - I mean real actual wars - and they won them through a combination of subterfuge, good strategy, honor, courage, and accumulating resources, etc. This story of protecting humanity from a variety of existential risks is no different, but nobody is taking it seriously enough.
Also, to make a final point about the importance of intelligence: when you are trying to communicate with creatures of varying levels of intelligence, you must take this into account. It's even more important when you are arguing a case against superintelligent AI (or against creating it). And yet, the communication that happens does not indicate any targeting of a particular intelligence group. It doesn't show an understanding of how to communicate with several groups at once, allowing them to switch between different levels, even though it's possible to do even in the traditional book format, which many authors and movie directors have successfully done in the past. Cameron's Terminator 2 is a good example of a work with layers of complexity. A story can be both an action film and also a nuanced narrative about civilizational and philosophical issues. But it's difficult to do that. It requires a master storyteller, and Yudkowsky is not one. An alternative is to have allies in this communication and use different formats. The argument that he makes must be a coherent system of meaning, a system of arguments of different levels of complexity, like in Wired's video series "Five Levels," which would allow anyone to move between these levels and hear a message better attuned to their level of understanding. If this is indeed a major thrust in the communications campaign of MIRI, I think such attention to the method of communicating complexity would make a lot of sense. In that case, different people, different experts, and different reviewers would be better able to comprehend the points made and not just react to them automatically and instinctively.
If you prefer an AI formatted and slightly edited version:
The Strategic Void: A Critique of AI Safety Communication
The discourse surrounding existential risk from artificial intelligence, spearheaded by figures like Eliezer Yudkowsky and organizations such as MIRI (Machine Intelligence Research Institute), presents a curious paradox. They claim to possess a strategic blueprint for safeguarding humanity, but a closer examination reveals a critical void. What they have is not a strategy, but a collection of stratagems—a handful of guiding principles mistaken for a comprehensive plan. Their entire approach is predicated on a single, overwhelming vector: to be radical and to incite panic. This is akin to trying to win a complex war with only the battle cry of "charge!" While it generates noise and attention, it lacks the sophistication required for a challenge of this magnitude.
This fundamental strategic lack is rooted in a deeper deficiency: a profound disconnect from the realities of global governance and policy-making. Based on an analysis of their communications, their expertise in how the world is actually governed and how policy is forged is minimal—perhaps "one millie Kissinger." This isn't just an academic shortcoming; it's a fatal flaw in their ability to steer the global conversation. They do not know the levers of power or the language of diplomacy, which are essential for navigating a polycrisis. We have a precedent for such a complex communications challenge in the global response to climate change, where decades of research have been devoted to understanding the media, psychological, and political dimensions of the issue. The AI safety conversation lacks this depth, resulting in a frustratingly unproductive debate where, as the transcript notes, it's often "idiots arguing with idiots," with the general public and politicians left as even less informed bystanders.
The Core Misunderstanding: Intelligence and Narrative
At the heart of this miscommunication lies an absurdity: the discussion of superintelligence often willfully ignores the nature of intelligence itself. There are critical, qualitative differences between human intelligence, artificial intelligence, and the prospective superintelligent agent—differences so profound that some aspects are considered information hazards. Yudkowsky is correct on one pivotal point: a superintelligent agent would not be just a faster human. It would possess an ontological superiority in its ability to coordinate, plan, orchestrate, and execute.
* This is not a slow, systemic crisis like the gradual environmental degradation caused by hydrocarbons or plastics.
* This is the rapid, coherent unfolding of a superior plan, authored by a single agent or a perfectly coordinated group.
This fundamental shift in nature is why Yudkowsky's focus on science-fiction narratives is, in principle, sensible. The emergence of a superintelligence would not be a stochastic scientific process to be studied, but a *narrative* to be experienced. It could be a global thriller, a detective story, or a piece of science fiction—perhaps all simultaneously, playing out in parallel across thousands of locations. It will be, at its core, a power game: a strategic contest of "us versus them," or from the AI's perspective, "me versus everybody else."
The Infantile Game and the Missing Ally
Here, however, Yudkowsky's framework reveals another critical gap. While he correctly identifies the scenario as a power game, he demonstrates little familiarity with how such games are played at the highest strategic level. He lacks a crucial ally who understands the real-world dynamics of power, subterfuge, and grand strategy. The current field is populated by players—from SBF to Musk and Altman—who are engaged in what can only be described as "infantile games." Meanwhile, the world's true strategic players, those who navigate global power dynamics as a matter of course, are conspicuously absent from the AI/anti-AI chessboard. Winning this "war" for humanity's future requires a complex, multi-front offensive, not a single-thrust panic campaign.
A Blueprint for Intelligent Communication
The final failure is in the communication itself. If you are arguing against the creation of a superintelligent adversary, you must demonstrate superior intelligence in your method of persuasion. The current approach does not.
* It fails to target specific intelligence groups.
* It shows no understanding of how to communicate with multiple audiences at once.
* It lacks the ability to let audiences switch between levels of complexity to meet their understanding.
This is not a limitation of medium; it is a failure of methodology. A traditional book or film can masterfully contain multiple layers. James Cameron's Terminator 2 is a prime example—a film that operates simultaneously as a blockbuster action movie and a profound philosophical exploration of fate and civilization. Yudkowsky is not such a master storyteller. The solution, therefore, is not to rely on a single voice, but to build a coalition of allies using diverse formats.
The arguments for AI safety must be woven into a coherent system of meaning, a multi-layered architecture of ideas. A model like Wired's "Five Levels" video series, where an expert explains a concept to a child, a teenager, a college student, a grad student, and a colleague, is the kind of sophisticated approach needed. This would create a communication ecosystem where anyone, from a concerned citizen to a specialized expert, could find a version of the message attuned to their level of comprehension. If MIRI's campaign were to adopt this as its major thrust, it would finally allow people to comprehend the points being made, rather than just reacting to them with instinctive fear or dismissal.
"As I read the lengthy discussion of alien birds about the optimal number of stones in nests, I realized there is no way this storyline appeals to a general audience."
I agree that the birds and stones parable was probably the weakest and may scare off people, I think there is an insight buried here but surely there could have been a faster way to say: "AIs might have weird goals" Overall I liked the book. You display a picture from the daily mail about them proposing to bomb AI labs, I don't think that's accurate, just sensationalism.
When you advocate for the trial and error approach, where we see that we made a mistake and then retrofit guardrails, how do you see that playing out in practice?
Assuming we do hit superintelligence how do we notice it is misaligned and recover? I haven’t found a convincing peer reviewed plan and I’m hoping you have. Thanks!
Thank you for your well researched, well documented analysis.
As someone who took the MIRI-adjacent BlueDot AI Safety Fundamentals course in 2024 https://bluedot.org/, and participated in the Existential Risks Persuasion Tournament in 2022 https://forecastingresearch.org/xpt, I have some familiarity with those folks. Something that stands out in my mind is their dearth of publications in refereed journals. In neither of those X-risks activities in which I participated, did the extremists present refereed papers to support their contentions. Clearly, the AI extremists are adverse to peer reviewers. That said, for a view of what the peer-reviewed moderates are saying: https://forecastingresearch.org/publications.
for the average reader, the evident lack of self-awareness and failure to accurately anticipate how the tone and style of their message will be received by the target audience isn’t exactly reassuring about their grasp on reality.
good stuff, Nirit, very helpful
Thank you, Gerd.
Excellent analysis! I especially appreciated how you highlighted the mixed reception, noting that despite some high-profile endorsements, the "Doom Bible" still left many unconvinced. Its important to critically examine such strong claims about AI.
Well seeing all the reviews lined up like this is quite something. Jolly well done
Yes, this book is not good. They don't make any actual argument.
I've also reviewed it on my AI blog and show it's empty:
oscarmdavies.substack.com/p/000016-on-yudkowsky-and-soares
Well, this is a very useful meta-review, but it shows an incomplete understanding of what's going on with this communication. Yudkowsky and MIRI may claim to have a strategy. However, I think what they have is not a strategy, not a complex set of decisions, but rather one or several strategic principles - you can call them stratagems. They think that a strategy needs just one main vector, but this isn't the case. When dealing with a polycrisis like artificial intelligence safety, a solution necessarily is going to be relatively complex. How complex? I don't know, but definitely more complex than being radical and trying to raise panic about the topic. That is not a strategy.
I believe, based on my experience of albeid limited communications with Eliezer and others in the field, that they're not experts in how policy is actually made, who governs the world, and how. I do know that there are some people with some background in this, but the overall level of expertise is small. Maybe you can say it's one milliKissinger or something like this. In order to guide, to steer the conversation, the discourse, and the understanding of how discourse happens in such complex fields, this expertise is crucial. We do have at least one example to study (and use as a benchmark) - the global climate change sphere, where climate communications have been studied extensively, including media, psychological, and political aspects.
The response to the IABIED book, the responses in reviews, show that the critics of Yudkowsky do not have a better grasp on these (discource/governance) issues than Yudkowsky himself. So it's like idiots arguing with idiots (where an ordinary politician or an ordinary citizen is something even less - a milliidiot).
Next, I want to focus on the absurdity of ignoring intelligence in a discussion of superintelligence. There are certain aspects I don't want to go into details about because that would be an infohazard, but there are crucial differences between artificial intelligence, superintelligent agents, and humans. When discussing these sci-fi, realistic, or in-between scenarios, one needs to focus more on these differences.
This is accidentally an area where I think you are making a big mistake when you suggest that the focus on middle-level safety is the right approach. Yudkowsky is arguing, in my view correctly, that superintelligent agents, once they are created, will have a much bigger capability to coordinate, plan, orchestrate, and execute than humans. This means that we will not be dealing with a huge, systemic complex of technical and social subsystems - political and business systems - that create something complex and dangerous but slow and understandable, like burning hydrocarbons, producing plastics, or adding lead to gasoline. Instead, it would be a rapid unfolding of a rather high-quality, coherent plan created by an agent or group of agents.
This is why the focus on science fiction narrative and storytelling actually makes a lot of sense, because what's going to happen is essentially a narrative unfolding. It can be a thriller, a detective story, a science fiction story, or all of them at once, maybe executed in parallel in thousands of places on Earth. But it's not going to be a stochastic process that scientists can study and understand. It's going to be a power game, a power play. It's going to be a strategic game: "us versus them" or "me versus everybody else" from the point of view of the superintelligence.
However, it's obvious that Yudkowsky has little familiarity with how these power games are played, and he doesn't have a good ally who understands this. The question of how to find such an ally is, I think, a core issue for any successful transhumanist long-term safety strategy. And right now, I don't think there is a good understanding of this. I think everybody is playing a very infantile game. People like SBF and everybody else, essentially even Musk and Altman, etc. There are some strategic players in the world; it's just that right now they're not involved in this AI or anti-AI strategic game, which is rather interesting to say the least.
Returning back, I think in order to be successful in this communication, even the communication itself must be understood as a complex strategic offensive and not just one thrust of it. People have played military games - I mean real actual wars - and they won them through a combination of subterfuge, good strategy, honor, courage, and accumulating resources, etc. This story of protecting humanity from a variety of existential risks is no different, but nobody is taking it seriously enough.
Also, to make a final point about the importance of intelligence: when you are trying to communicate with creatures of varying levels of intelligence, you must take this into account. It's even more important when you are arguing a case against superintelligent AI (or against creating it). And yet, the communication that happens does not indicate any targeting of a particular intelligence group. It doesn't show an understanding of how to communicate with several groups at once, allowing them to switch between different levels, even though it's possible to do even in the traditional book format, which many authors and movie directors have successfully done in the past. Cameron's Terminator 2 is a good example of a work with layers of complexity. A story can be both an action film and also a nuanced narrative about civilizational and philosophical issues. But it's difficult to do that. It requires a master storyteller, and Yudkowsky is not one. An alternative is to have allies in this communication and use different formats. The argument that he makes must be a coherent system of meaning, a system of arguments of different levels of complexity, like in Wired's video series "Five Levels," which would allow anyone to move between these levels and hear a message better attuned to their level of understanding. If this is indeed a major thrust in the communications campaign of MIRI, I think such attention to the method of communicating complexity would make a lot of sense. In that case, different people, different experts, and different reviewers would be better able to comprehend the points made and not just react to them automatically and instinctively.
Danila Medvedev
If you prefer an AI formatted and slightly edited version:
The Strategic Void: A Critique of AI Safety Communication
The discourse surrounding existential risk from artificial intelligence, spearheaded by figures like Eliezer Yudkowsky and organizations such as MIRI (Machine Intelligence Research Institute), presents a curious paradox. They claim to possess a strategic blueprint for safeguarding humanity, but a closer examination reveals a critical void. What they have is not a strategy, but a collection of stratagems—a handful of guiding principles mistaken for a comprehensive plan. Their entire approach is predicated on a single, overwhelming vector: to be radical and to incite panic. This is akin to trying to win a complex war with only the battle cry of "charge!" While it generates noise and attention, it lacks the sophistication required for a challenge of this magnitude.
This fundamental strategic lack is rooted in a deeper deficiency: a profound disconnect from the realities of global governance and policy-making. Based on an analysis of their communications, their expertise in how the world is actually governed and how policy is forged is minimal—perhaps "one millie Kissinger." This isn't just an academic shortcoming; it's a fatal flaw in their ability to steer the global conversation. They do not know the levers of power or the language of diplomacy, which are essential for navigating a polycrisis. We have a precedent for such a complex communications challenge in the global response to climate change, where decades of research have been devoted to understanding the media, psychological, and political dimensions of the issue. The AI safety conversation lacks this depth, resulting in a frustratingly unproductive debate where, as the transcript notes, it's often "idiots arguing with idiots," with the general public and politicians left as even less informed bystanders.
The Core Misunderstanding: Intelligence and Narrative
At the heart of this miscommunication lies an absurdity: the discussion of superintelligence often willfully ignores the nature of intelligence itself. There are critical, qualitative differences between human intelligence, artificial intelligence, and the prospective superintelligent agent—differences so profound that some aspects are considered information hazards. Yudkowsky is correct on one pivotal point: a superintelligent agent would not be just a faster human. It would possess an ontological superiority in its ability to coordinate, plan, orchestrate, and execute.
* This is not a slow, systemic crisis like the gradual environmental degradation caused by hydrocarbons or plastics.
* This is the rapid, coherent unfolding of a superior plan, authored by a single agent or a perfectly coordinated group.
This fundamental shift in nature is why Yudkowsky's focus on science-fiction narratives is, in principle, sensible. The emergence of a superintelligence would not be a stochastic scientific process to be studied, but a *narrative* to be experienced. It could be a global thriller, a detective story, or a piece of science fiction—perhaps all simultaneously, playing out in parallel across thousands of locations. It will be, at its core, a power game: a strategic contest of "us versus them," or from the AI's perspective, "me versus everybody else."
The Infantile Game and the Missing Ally
Here, however, Yudkowsky's framework reveals another critical gap. While he correctly identifies the scenario as a power game, he demonstrates little familiarity with how such games are played at the highest strategic level. He lacks a crucial ally who understands the real-world dynamics of power, subterfuge, and grand strategy. The current field is populated by players—from SBF to Musk and Altman—who are engaged in what can only be described as "infantile games." Meanwhile, the world's true strategic players, those who navigate global power dynamics as a matter of course, are conspicuously absent from the AI/anti-AI chessboard. Winning this "war" for humanity's future requires a complex, multi-front offensive, not a single-thrust panic campaign.
A Blueprint for Intelligent Communication
The final failure is in the communication itself. If you are arguing against the creation of a superintelligent adversary, you must demonstrate superior intelligence in your method of persuasion. The current approach does not.
* It fails to target specific intelligence groups.
* It shows no understanding of how to communicate with multiple audiences at once.
* It lacks the ability to let audiences switch between levels of complexity to meet their understanding.
This is not a limitation of medium; it is a failure of methodology. A traditional book or film can masterfully contain multiple layers. James Cameron's Terminator 2 is a prime example—a film that operates simultaneously as a blockbuster action movie and a profound philosophical exploration of fate and civilization. Yudkowsky is not such a master storyteller. The solution, therefore, is not to rely on a single voice, but to build a coalition of allies using diverse formats.
The arguments for AI safety must be woven into a coherent system of meaning, a multi-layered architecture of ideas. A model like Wired's "Five Levels" video series, where an expert explains a concept to a child, a teenager, a college student, a grad student, and a colleague, is the kind of sophisticated approach needed. This would create a communication ecosystem where anyone, from a concerned citizen to a specialized expert, could find a version of the message attuned to their level of comprehension. If MIRI's campaign were to adopt this as its major thrust, it would finally allow people to comprehend the points being made, rather than just reacting to them with instinctive fear or dismissal.
Thank you for this summary, and keep up the good work!
"As I read the lengthy discussion of alien birds about the optimal number of stones in nests, I realized there is no way this storyline appeals to a general audience."
I agree that the birds and stones parable was probably the weakest and may scare off people, I think there is an insight buried here but surely there could have been a faster way to say: "AIs might have weird goals" Overall I liked the book. You display a picture from the daily mail about them proposing to bomb AI labs, I don't think that's accurate, just sensationalism.
Thanks for the helpful summary Nirit.
When you advocate for the trial and error approach, where we see that we made a mistake and then retrofit guardrails, how do you see that playing out in practice?
Assuming we do hit superintelligence how do we notice it is misaligned and recover? I haven’t found a convincing peer reviewed plan and I’m hoping you have. Thanks!