On December 3, 2025, Daniel Alejandro Moreno-Gama wrote in PauseAI’s Discord: “We are close to midnight, it’s time to actually act.” On April 10, 2026, he acted.

This is a developing story and will be updated. This piece has three parts: (1) The attack and the charges, (2) Social posts about AI doom, and (3) The reckoning. Each section will be updated with new information and analysis.

The Attack and the Charges

A 20-year-old suspect allegedly threw a firebomb at Sam Altman’s house on Friday night. The device was a Molotov cocktail (a bottle containing a flammable liquid and a burning rag). He then went to OpenAI’s headquarters and allegedly threatened to burn it down. He was taken into custody by San Francisco police outside of OpenAI’s offices. According to a criminal complaint filed Monday by the FBI, Daniel Moreno-Gama said he wanted to “kill anyone inside.”

At a Monday news conference, the FBI’s acting special agent in charge in San Francisco described the attack as “planned, targeted, and extremely serious.” He arrived in SF with a gun, a manifesto, and a hit list. The Department of Justice said Moreno-Gama possessed a three-part document he had written against AI. The first part, titled “Your Last Warning,” allegedly advocated killing CEOs of AI companies and their investors, and contained their addresses.

Moreno-Gama faces both state and federal charges, including attempted murder – both of Altman and the security guard who was at his house the night of the attack – and attempted arson on the state level. At the federal level, he faces charges related to an unregistered firearm and attempted damage and destruction of property by means of explosives.

“If the evidence shows that Mr. Moreno-Gama executed these attacks to change public policy or to coerce government or other officials, we will treat this as an act of domestic terrorism, and together with our partners, prosecute him to the fullest extent of the law,” said Craig Missakian, US Attorney for the Northern District of California.

The San Francisco District Attorney’s Office said it would seek to have Moreno-Gama held in custody without bail because of the “public safety risk he poses.”

As reported by The San Francisco Standard, “In a note to employees Friday, OpenAI said there was no immediate threat to staff or its offices but advised that there would be increased police and security presence at its Mission Bay campus.”

Social Posts About AI Doom

In a blog post reacting to the attack, Sam Altman wrote, “Words have power.”

Yes, they do. And as the evidence from Moreno-Gama’s posts suggests, this was a long radicalization process shaped by “AI existential risk” rhetoric.

For background, see this guide explaining the differences between StopAI, PauseAI, and ControlAI.

PauseAI Discord Server

On Friday night, after the suspect’s identity became public, I posted on Twitter that he had been an active member of PauseAI under the username dmgama/Butlerian Jihadist (both also used on his Instagram account). Butlerian Jihadist is inspired by the Dune series and the violent crusade against “thinking machines.”

Shortly thereafter, PauseAI banned Moreno-Gama’s account. It removed his message history from the Discord server. Since I had already obtained the messages he posted before they were deleted, I published them online.

Here are some highlights from those screenshots, in chronological order:

• June 11, 2024

Introduction: “I am very passionate about this issue and am willing to learn and help whatever means necessary.” “Is there any sort of timeline for how long humanity has left?” Moreno-Gama scheduled a meeting to get onboarded to the group.

• June 16, 2024

Moreno-Gama emailed Texas’s Congressman Dan Crenshaw. He congratulated Crenshaw on a 2023 interview between the Congressman and Eliezer Yudkowsky. He stated that he is not concerned about job loss but rather the “existential threat to our continued existence.”

• November 5-6, 2025

“People will never feel as saliently about lesser AI risk as they will about a possible incoming extinction event (which we know is very likely, but they don’t believe it just yet). And we need that salience and passion for this movement to work.” “We owe it to everyone… to be stronger than that and at least die fighting.” “Time is against us, we need to start seeing results manifesting.”

• December 3, 2025

“We are close to midnight, it’s time to actually act.” The moderator warned him: “Advocating violence in any form is grounds for a ban.” Daniel Moreno-Gama replied: “Do you really think we have any time left?”

• January 8, 2026

Moreno-Gama got feedback from PauseAI members on his “AI Existential Risk” Substack post.

Two days after the attack, PauseAI CEO Maxime Fournes issued a statement to members of the group:

“The suspect joined this server approximately two years ago and posted 34 messages in total. None contained explicit calls to violence, though our moderators did flag one message as ambiguous and issued a warning. He had no involvement with PauseAI beyond those messages. Following the attack, he was banned from this server.”

StopAI

Moreno-Gama also appears to have been on the StopAI Discord server, where he displayed a troubling mindset.

StopAI’s recent statement adds:

“He joined a Stop AI public online forum, introduced himself, then asked, ‘Will speaking about violence get me banned?’ After he was given a firm ‘Yes,’ he ceased all activities on our forum. This was several months before his alleged criminal activities.”

One of the most common chants at StopAI’s protests outside OpenAI’s offices was: “Close OpenAI or We’re All Gonna Die!”

LessWrong

On LessWrong, the forum founded by Eliezer Yudkowsky, the moderator Jim Babcock (username jimrandomh) wrote that, “If Daniel Alejandro Moreno-Gama had a LessWrong account, then I, using my available tools as an admin and all publicly-reported usernames I’ve seen, cannot find it.” He emphasized that advocating violence “would also be grounds for a ban on LW.”

Oliver Habryka, who runs Lightcone Infrastructure and LessWrong, replied with a notable clarification: “Discussing or advocating violence is not banned.” He wrote:

“I think many people will encounter thoughts and ideas around whether violence is appropriate when they encounter the existential stakes of AI. Discussing whether those ideas are right or wrong is very much a thing I want LessWrong to be able to do. I think they are almost universally wrong, but people arriving at that conclusion will do so more likely with argument […] Discussing or advocating violence is not banned on LessWrong (though I would be surprised if it isn’t met with very consistent opposition in practically all cases).”

That position is striking, especially given LessWrong’s history of discussions about violence. In April 2025, I pointed out the following quotes:

“People like Eliezer [Yudkowsky] are annoyed when people suggest their rhetoric hints at violence against AI labs and researchers. But even if Eliezer & co don’t advocate violence, it does seem like violence is the logical conclusion of their worldview - so why is it a taboo?”

“If preventing the extinction of the human race doesn’t legitimize violence, what does?”

Moreno-Gama’s Steady Feed of Doomer Content

On the same Friday night, April 10, 2026, people began circulating screenshots of Moreno-Gama’s social media posts. As Alexander Campbell later summarized in “The ‘Rational’ Conclusion”:

“His Instagram was a feed of doomer content: capability curves captioned ‘if we do nothing very soon we will die,’ Venn diagrams placing us at the intersection of The Matrix, Terminator, and Idiocracy. Four months before the attack, he recommended Yudkowsky and Soares’ ‘If Anyone Builds It, Everyone Dies’ to his followers.”

Moreno-Gama also had a Substack, where he published six lengthy posts between January 6 and March 1, 2026. As Campbell summarizes:

“In January, he published ‘AI Existential Risk,’ estimating the probability of AI-caused extinction as ‘nearly certain.’ He called the technology ‘an active threat against anyone who is using it and especially towards the people building it.’ He concluded: ‘We must deal with the threat first and ask questions later.’ He wrote a poem imagining the children of AI developers dying, asking their parents why they did nothing. ‘May Hell be kind to such a vile creature,’ he wrote about the builders.”

The Reckoning

The attack should be a wake-up call about the effects of the doomer rhetoric.

In “Alleged Sam Altman Firebomber Wrote of Fears AI Would End Humanity,” I told the SF Chronicle reporter:

“When people are in a state of anxiety, and depressed, worried, and all they hear from the echo chamber is doom, they become more vulnerable, and become more radicalized, even if places like PauseAI or StopAI are not advocating for violence.”

“AI Doomers Built a Radical Ideology. Now Their Followers Are Acting On It,” writes Jordan Schachtel on his Dossier Substack.

“They have told a generation of anxious, hyper online young people that the most powerful companies in human history are building a machine that could end civilization. And now, when their followers are beginning to act on their incendiary rhetoric, they expressed shock and horror when some of those people don’t stop at signing an open letter.”

“And these ideas will almost certainly continue to indoctrinate and radicalize like-minded individuals into criminal action,” Schachtel warns.

To explain how the AI doomers produce their first Molotov cocktail, Alexander Campbell identifies three moving parts:

1. Start with certainty, for example, “If Anyone Builds It, Everyone Dies.”

2. Add a purity spiral, aka escalation. Within this community, members compete to demonstrate commitment by raising the stakes. P(doom) numbers climb from 50% to 90% to 99.99999%.

3. Then, cheap talk gets tested. Once the stakes are framed as existential, nearly any form of extremism can be rationalized.

As Campbell put it: “It was only a matter of time before someone took the framework at face value. They should stop acting surprised when their own logic shows up at 3:45 AM with a bottle full of gasoline.”

A Fortune magazine article by Sharon Goldman, “Pause AI and Stop AI: Meet the anti-AI groups facing questions after the attack on Sam Altman,“ features this observation (by yours truly):

“When prominent AI doomers like Eliezer Yudkowsky keep insisting that human extinction is imminent, it should not be surprising when someone is driven to extreme action. Young, anxious followers, looking for purpose, can be radicalized by apocalyptic AI rhetoric, even without explicit calls for violence.

The warning signs were there all along, including the November 2025 lockdown at OpenAI’s offices. The real question is how long the people fueling AI panic expect to avoid responsibility for where that radicalization leads, especially for the most vulnerable.”

Daniel Moreno-Gama, in an interview before he arrived in SF with a gun and a hit list:

“The Last Invention” podcast episode with the Firebomber is irresponsible and dangerous, especially regarding the Contagion Effect.

• Do not promote what the suspect has to say.

• Criticize his radicalization from watching doom videos on YouTube to making a Molotov cocktail.

The Warning Signs Were There All Along

When Daniel Moreno-Gama joined StopAI and PauseAI, StopAI co-founder Guido Reichstadter (who later split from the group) and Yanco (who publicly helps with messaging) had already published incendiary rhetoric about AI developers:

When I published those screenshots at the end of 2024, I added:

“Both PauseAI and StopAI stated that they are non-violent movements that do not permit ‘even joking about violence.’ That’s a necessary clarification for their various followers. There is, however, a need for stronger condemnation, and law enforcement should monitor this radicalization closely. The murder of the UHC CEO showed us that it only takes one brainwashed individual to cross the line.”

In April 2025, I described the radicalization process:

“A major problem with the AI doomers’ mindset of ‘the end justifies the means’ is that all possible radical ‘solutions’ are on the table. Why stop with pervasive surveillance of software and hardware if the government can also surveil all known AI researchers?

This is how it plays out: They start with weird ideas like ‘Patent trolling to save the world,’ ‘don’t build AI or advance computer semiconductors for the next 50 years,’ or that ‘using an open-source model’ should lead to ‘20 years in jail.’ But, then, they end up playing with dangerous ideas about ‘AGI developers’ having ‘a bullet put through their head.’ ‘Walk to the labs across the country and burn them down. Like, literally. That is the proper reaction.’

This radicalization process can only lead to bad outcomes.”

We also had the Zizians, a murder cult linked to six deaths across three states. I summarized that crazy story in “The Rationality Trap,” and it has been detailed in WIRED, New York magazine, and Rolling Stone. As the New York Times investigative reporter warned, “The story of the Zizians illustrates how concerns about AI safety, like any story involving apocalypse, can be used to make people do bad things.”

Then, in November 2025, StopAI co-founder Sam Kirchner abandoned his commitment to nonviolence, assaulted another member, and made statements that left the group worried he might obtain a weapon to use against AI researchers. Those threats prompted OpenAI to lock down its San Francisco offices.

In The Atlantic’s “The Strange Disappearance of an Anti-AI Activist,” we learned that Kirchner had said, “The nonviolence ship has sailed for me.” His whereabouts remain unknown.

In a “For Humanity” podcast episode, “Go to Jail to Stop AI” (#49, October 14, 2024), Sam Kirchner said:

“We don’t really care about our criminal records because if we’re going to be dead here pretty soon or if we hand over control, which will ensure our future extinction here in a few years, your criminal record doesn’t matter.”

After Kirchner’s disappearance, the podcast host and founder of “GuardRailNow” and the “AI Risk Network,” John Sherman, deleted this episode from Apple Podcasts, Spotify, and YouTube. Before it was removed, I downloaded the 1:14:14 video.

Sherman also produced an emotional documentary with StopAI and PauseAI titled “Near Midnight in Suicide City” (December 5, 2024, episode #55. See its trailer and promotion on the Effective Altruism Forum). That, too, was removed from podcast platforms and YouTube. I preserved a copy in my archive. Before it was taken down, it had gathered 60k views.

As I wrote on Techdirt, “We need to confront the social dynamics that turn abstract fears of technology into real-world threats against the people building it.”

And as City Journal concluded in “A Cofounder’s Disappearance—and the Warning Signs of Radicalization”:

“We should stay alert to the warning signs of radicalization: a disaffected young person, consumed by abstract risks, convinced of his own righteousness, and embedded in a community that keeps ratcheting up the moral stakes.”

That warning, issued in November 2025, reads even more clearly in April 2026.

Subscribe now

Repost from Techdirt.

There is a familiar media failure in which opposing viewpoints are presented as equally valid, even when the evidence overwhelmingly supports one side. It’s called Bothsidesism. This false balance phenomenon legitimizes misinformation and undermines public understanding by giving disproportionate weight to baseless claims.

Why bring this up? Because the new AI Doc film is based on it.

The film wants credit for being “balanced” because it assembles a wide range of experts. But putting Prof. Fei-Fei Li, a pioneering computer scientist, next to someone like Eliezer Yudkowsky, an author of a Harry Potter fanfic, is not “balance.”

Once you understand that false equivalence is baked into the film’s storytelling, you understand how misleading and manipulative the documentary is. And it is compounded by a series of falsehoods that go unchallenged and uncorrected.

This review addresses both failures.

The “AI Doc” Movie

“The AI Doc: Or How I Became an Apocaloptimist,” co-directed by Daniel Roher and Charlie Tyrell, sets out to explore AI, especially its potential for good and bad, with a strong emphasis on the filmmakers’ anxieties and fears. Its basic premise is: “A father-to-be tries to figure out what is happening with all this AI insanity.” As summarized by Andrew Maynard from Future of Being Human:

“The documentary progresses through the eyes of director Daniel Roher as he faces a tsunami of existential AI angst while grappling with the responsibility of becoming a father. Motivated by a fear that artificial intelligence could spell the end of everything that matters, he sets out to interview some of the largest (and loudest) voices in AI to fathom out whether this is the best of times or worst of times for him and his wife (filmmaker Caroline Lindy) to bring a kid into the world.”

The “loudest voices” include many AI doomer figures, such as Eliezer Yudkowsky, Dan Hendrycks, Daniel Kokotajlo, Connor Leahy, Jeffrey Ladish, and two of the most populist voices on emerging tech (first social media and now AI): Tristan Harris and Yuval Noah Harari. The film also features voices on AI ethics, including David Evan Haris, Emily M. Bender, Timnit Gebru, Deborah Raji, and Karen Hao. On the more boosterish side, there are Peter Diamandis and Guillaume Verdon (AKA Beff Jezos). Three leading AI CEOs were also interviewed: OpenAI’s Sam Altman, DeepMind’s Demis Hassabis, and Anthropic’s Amodei siblings, Dario and Daniela. (Meta’s Mark Zuckerberg declined, and xAI’s Elon Musk agreed but never showed up).

The movie started playing in theaters on March 27, but there are already plenty of reviews (dating back to the Sundance Film Festival). The praise is fairly consistent: It is timely, wide-ranging, visually energetic, and unusually well-connected, with access to major AI figures.

The most common criticism is that it is too deferential to interviewees and too thin on hard interrogation or concrete answers. As several reviewers put it:

1. “Roher’s willingness to blindly accept any and all of his speakers’ pronouncements leaves The AI Doc feeling toothless.”

2. “By giving its doomer and accelerationist voices so much time to present AI’s most hyperbolic potential outcomes with little pushback, the documentary’s first half plays more like an overlong advertisement for the technology as opposed to a piece of measured analysis.”

3. “Roher acts as a fantastic storyteller, but he treats his subjects too gently. The film desperately needs more pushback during the interviews.”

Tristan Harris, co-founder of the Center for Humane Technology, told the AP: “My hope is that this film is kind of like ‘An Inconvenient Truth’ or ‘The Social Dilemma’ for AI.”

That is not reassuring. It is more like a glaring warning sign. Harris’s “Social Dilemma” and “AI Dilemma” movies were full of misinformation and nonsensical hyperbole, and both were designed to be manipulative and dishonest. If anything, his endorsement tells you exactly what kind of movie this is.

After watching the AI Doc, I realized what the doomers had managed to accomplish here: The film absorbs the panic rather than investigates it.

The False Balance of The AI Doc

The AI Doc starts with what one reviewer called a “Doom Parade.” It aims to set the tone.

“The worst AI predictions are presented first,” another reviewer noted. “Eliezer Yudkowsky, co-founder of the Machine Intelligence Research Institute, calmly talks of the ‘abrupt extermination’ of humanity.”

And it is worth remembering who Yudkowsky is and what he has actually advocated. In his notorious TIME op-ed, “Shut it All Down,” he argued that governments should “be willing to destroy a rogue datacenter by airstrike.” In his book “If Anyone Builds It, Everyone Dies,” which many reviewers found unconvincing and “unnecessarily dramatic sci-fi,” he (and his co-author Nate Soares) proposed that governments must bomb labs suspected of developing AI. Based on what exactly? On the authors’ overconfident, binary worldview and speculative scenarios, which they mistake for inevitability.

One review of that book observed, “The plan with If Anyone Builds It seems to be to sane-wash him [Yudkowsky] for the airport books crowd, sanding off his wild opinions.”

That is more or less what the new documentary does, too. The AI Doc sane-washes the loudest doomers for mainstream viewers, sanding off their wild opinions.

In his newsletter, David William Silva addresses the documentary’s “series of doomers,” who “describe AI-driven extinction with the calm confidence of people who have said these things so many times they have stopped noticing they have no evidence for them.”

“Roher’s reaction is full terror,” Silva adds. “I hope it is unequivocally evident that this is not journalism.”

That gets to the heart of it. The film pretends to weigh competing perspectives, but in practice, it grants disproportionate authority to people most invested in flooding the zone with AI panic. And there is a well-oiled machine behind this kind of AI panic. As Silva writes:

“The people behind the AI anxiety machine. […] They know that predicting human extinction by software is an extraordinary claim requiring extraordinary evidence. They know they don’t have it. They know ‘my kids won’t live to see middle age’ is nothing but performance. […] And they do it anyway. Why do you think that is? The calculation is simple. Some people will see through it, and they will be annoyed, write rebuttals, call it what it is. Ok, fine. Just an acceptable loss. The believers, on the other hand, are a market. As long as the ratio stays favorable, the machine is profitable.”

One of the biggest beneficiaries of this film is Harris.1 He is framed as if he is in the middle between the two main camps (doomers and accelerationists), and his narrative gradually becomes the film’s narrative (similar to the Social Dilemma). His call to action even serves as the ending (with a QR code directing viewers to a designated website).

The problem is that this framing has very little to do with reality. Harris’s Center for Humane Technology got $500,000 from the Future of Life Institute for “AI-related policy work and messaging cohesion within the AI X-risk [existential risk] community.” That is not a neutral player.

There’s a touching scene in the film where Roher mentions his father’s cancer treatment and expresses hope that AI might help. Harris appears visibly emotional. But in other contexts, Harris has argued against looking at AI for help with cancer treatment… in the belief that it would lead to extinction. Here he is on Glenn Beck’s show in 2023:

“My mother died from cancer several years ago. And if you told me that we could have AI that was going to cure her of cancer, but on the other side of that coin was that all the world would go extinct a year later, because of the, the only way to develop that was to bring something, some Demon into the world that would we would not be able to control, as much as I love my mother, and I would want her to be here with me right now, I wouldn’t take that trade.”

That sort of hyperbole seems relevant to Harris’s stance on such things, but was not mentioned in the film at all.

Connor Leahy of Conjecture and ControlAI gets a similar makeover. In the documentary, he appears as another pessimistic expert. Elsewhere, he said he does not expect humanity “to make it out of this century alive; I’m not even sure we’ll get out of this decade!” His “Narrow Path” proposal for policymakers begins with the claim that “AI poses extinction risks to human existence.” Instead of calling for a six-month AI pause, he argued for a 20-year pause, because “two decades provide the minimum time frame to construct our defenses.”

This is exactly why background checks matter. Viewers of the AI Doc deserve to know the full scope of the more extreme positions these interviewees have publicly taken elsewhere. If someone has publicly argued for destroying data centers by airstrikes or stopping AI for 20 years, the audience should know that.

Debunking the Falsehoods

The film goes way beyond just pushing a panic. It also recycles several misleading or plainly false claims, letting them pass as established facts. Three stood out in particular.

Anthropic’s Blackmail study

One of the most repeated “facts” in reviews of the movie is that Anthropic’s AI model, Clause, decided, unprompted, to blackmail a fictional employee. In the film, Daniel Roher asks, “And nobody taught it to do that?” Jeffrey Ladish, of Palisade Research and Tristan’s Center for Humane Technology, replies: “No, it learned to do that on its own.”

That is a misleading characterization of the actual experiment, it has already been debunked in “AI Blackmail: Fact-Checking a Misleading Narrative.” Anthropic researchers admitted that they strongly pressured the model and iterated through hundreds of prompts before producing that outcome. It wasn’t a spontaneous emergence of “evil” behavior; the researchers explicitly ensured it would be the default. Telling viewers that the model has gone full “HAL 9000” omits the facts about the heavily engineered experimental setup.

Although this is a classic case of big claims and thin evidence, the film offers so little pushback that viewers are left to take Ladish’s statements at face value.

It is also worth remembering that Ladish has fought against open-source AI, pushed for a crackdown on open-source models, and once said, “We can prevent the release of a LLaMA 2! We need government action on this asap.” He later updated his position (and it’s good to revise such views). But does the film mention his earlier public hysteria? No.

Is AI less regulated than sandwich shops? No.

Connor Leahy tells Daniel Roher, “There is currently more regulation on selling a sandwich to the public” than there is on AI development. This talking point has become a favorite slogan in AI doomer circles. It was repeatedly stated by The Future of Life Institute’s Max Tegmark and, more recently, by Senator Bernie Sanders. It’s catchy. It’s also false.

State attorneys general from both parties have explicitly argued that existing laws already apply to AI. Lina Khan, writing on behalf of the Federal Trade Commission, stated that “AI is covered by existing laws. Each agency here today has legal authorities to readily combat AI-driven harm.” The existing AI regulatory stack already includes antitrust & competition regulation, civil rights & anti-discrimination law, consumer protection, data privacy & security, employment & labor law, financial regulation, insurance & accident compensation, property & contract law, among others.

According to Stanford’s 2025 AI Index, in the first two full post-ChatGPT years (2023 & 2024), approximately 70 AI-related national laws were passed worldwide. In the United States, 180 state AI-related laws were enacted in 2023 and 2024. Also, the number of federal AI-related regulatory agency actions rose from 25 in 2023 to 59 in 2024 (a total of 84). The National Conference of State Legislatures (NCSL) reported that in 2025, 38 states adopted or enacted roughly 100 additional AI measures.2

So no, AI is not less regulated than sandwich shops. It’s a misleading soundbite, not a serious description of legal reality.

Data center water usage

In the film, Karen Hao criticizes data centers, warning that “People are literally at risk, potentially of running out of drinking water.” That sounds alarming, which is presumably the point. But it is highly misleading.

In fact, Karen Hao had to issue corrections to her “Empire of AI” book because a key water-use figure was off by a factor of 4,500. The discrepancy was not 45x or 450x, but rather 4,500x. That is not a rounding error. For detailed rebuttals, see Andy Masley’s “The AI water issue is fake” and “Empire of AI is widely misleading about AI water use.”

There is also a basic proportionality issue here. As demonstrated by The Washington Post, “The water used by data centers caused a stir in Arizona’s drought-prone Maricopa County. But while they used about 905 million gallons there last year, that’s a small fraction of the 29 billion gallons devoted to the county’s golf courses.” To put that plainly: data centers accounted for just 0.1% of the county’s water use.3

It is also worth noting that “most of the water used by data centers returns to its source unchanged.” In closed-loop cooling systems, for example, water is recirculated multiple times, which significantly reduces net consumption.

None of this is hidden information. A basic fact-check by the filmmakers could have brought it to light. But that was not the film’s goal. They chose fear-based framing over actual reporting. They could have pressed interviewees on their track records, failed predictions, and political agendas. Instead, they let them narrate the stakes, unchallenged.

So, I think we can conclude that the AI Doc may want to appear balanced and thoughtful, but, unfortunately, too often it is not.

Final Remark

While Western filmmakers are busy platforming advocates for “bombing data centers” and “Stop AI for 20 years,” the Chinese Communist Party is building the actual infrastructure. The CCP is not making doom-and-gloom documentaries; it is racing ahead. This is a real strategic threat, and it is far more concerning than anything featured in this film.

Endnote

A new study, Which AI harms and risks will mobilize the public to act?, provides insight into an important shift in AI doomers’ messaging. After years of pushing “human extinction” narratives that failed to resonate with the public or policymakers, many existential risk advocates have pivoted toward environmental harms, job loss, and AI-enabled warfare. This study’s findings help explain why.

About the Study

The goal of the “Which AI harms and risks will mobilize the public to act?” study was to “understand what concerns people most concretely, and what is most likely to persuade them to act on those concerns.”

The research was conducted by the Social Change Lab, a UK-based organization that studies protest and people-powered movements. As it explains, “Through research reports, workshops, and training, we provide actionable insights to help movements and funders be more effective.”

Social Change Lab accepts donations through the Effective Altruism organization “Giving What We Can.” This particular study was funded by “Changing Ideas.”

The introduction asks: “What do we know so far about people’s attitudes to AI risks?”

Its answer is revealing:

“People respond more strongly to immediate, tangible harms - particularly job loss and risks to children - than to more abstract scenarios […] Job disruption and threats to children resonate most strongly across demographic groups, and people tend not to engage with catastrophic risk narratives.”

Experimental conditions

The researchers conducted a randomized controlled trial with 3,467 UK participants, drawing on a representative sample of adults aged 18 and older in terms of age, gender, and political affiliation.

The study tested 11 AI risk areas: (1) job displacement, (2) misinformation, (3) disinformation, (4) human extinction, (5) surveillance, (6) AI-enabled warfare, (7) social isolation, (8) bias and discrimination, (9) environmental harms, (10) sophisticated scams and fraud, and (11) loss of human capabilities and cognitive decline.

Key findings

• The extinction risk posed to humanity by AI (X-risk) had the lowest level of concern of any harm/risk.

• People are also less willing to act on this risk than on nearly every other risk (only social isolation came lower).

• Reading about X-risk does not make much difference to people’s concern or willingness to act on it.

• By contrast, environmental harm most galvanized people to act; people were substantially more willing to act on environmental harms of AI after reading about them, suggesting this issue has high mobilization potential.

• The risk of AI-enabled warfare caused the greatest level of concern about AI in general.

• Job displacement—by far the most salient concern in surveys—ranked surprisingly low on mobilization potential, suggesting an issue’s salience is not a reliable proxy for its mobilization potential.

Mention of existential risk/human extinction = 1%

“An important motivation for carrying out this study was the fact that the campaign groups currently most actively trying to mobilize the public on AI (Control AI, Pause AI, Stop AI) are focused on the existential threat to humanity posed by superintelligent AI.”

The study posed an open-ended question: “Could you tell us in 1 or 2 sentences how you feel about the development of AI?”

The study found that “existential / extinction risk was mentioned by very few people, and even when it was, several wrote of their skepticism of this risk being a real one.”

One example:

“I think that in general the risks are overblown. There are undoubtedly some concerns over potential misuse harming individuals, but claiming that it threatens the existence of humanity is ridiculous.”

To the rescue: AI-enabled welfare and environmental harms

This is where the study becomes more practical to the AI doomers.

“While this is not hugely encouraging for X-risk campaigners, there was an encouraging finding: reading about AI-enabled warfare and environmental harms due to AI significantly increased people’s concern about X-risk relative to the control condition. Reading about AI-enabled warfare also increased X-risk concern more than reading about X-risk itself. In other words, people are more concerned about X-risk if they read about AI-enabled warfare than if they actually read about X-risk.”

“An implication of these findings, put simply, is that X-risk campaigners would likely benefit from talking more about AI warfare.”

It goes on to argue that “tapping into people’s concern about AI warfare could be effective, especially now”: “For movement organizers, this might present a somber window of opportunity: messaging that connects AI development to ongoing military conflicts may tap into existing public attention in ways that more abstract risk framings cannot.”

The study makes a similar case for “environmental harms as a mobilization opportunity”:

“Environmental harms from AI emerged as one of the most promising areas for mobilization. The risk of environmental harms topped the rankings for willingness to act on a risk after reading about it. Perhaps more importantly, environmental harms showed one of the strongest malleability effects: people’s concern and willingness to act shifted significantly after reading about the issue, suggesting this is a risk area where public engagement campaigns could have real traction.”

“Implications for those working on human extinction risk”

The study states the problem plainly:

“The findings present a challenging picture for organizations currently focused on communicating about extinction risks to the public. Concern about X-risk and willingness to act on X-risk were both very low, in absolute terms and relative to other risks we tested.”

It also challenges a common assumption: that the most salient issue is automatically the best one for mobilization.

“Another important finding challenges the assumption that issues with the highest salience are also those which are the most effective mobilizers. We know from multiple surveys as well as from our open-text question here, that the AI risk area most salient to people is job loss. Yet this is not the issue that, according to our data, stands out in terms of the levels of concern or willingness to act connected to it. Moreover, reading about job loss risks did not increase concern or willingness to act on AI particularly strongly.

For activists, this suggests that talking about job loss might be helpful in ‘meeting people where they are at’ but that it might be more effective to move the conversation onto other topics, such as autonomous weapons and environmental harms, in order to see tangible actions.”

My take

After years of AI panic and fear-mongering campaigns (detailed on this Substack), the above findings are reassuring. The public is not buying the AI doomers’ “human extinction” messages. People actually reject the x-risk hype.

About a year ago—a shift that can be traced back to the start of the Trump administration and its pro-innovation agenda—x-risk advocates learned to “meet people where they are” by building broader coalitions around near-term harms. They changed their script to address job loss, data center water/electricity usage, and AI warfare.

But their goal remained the same: to steer people back toward their central obsession, “human extinction from AI.” On that front, this study suggests they have failed.

That does not mean the doomers will stop trying to “bait-and-switch” their audiences. If anything, this new study may encourage them to double down on that tactic, since it suggests that other pathways are the only ones that could work for their movement.

My ask to readers is simple: Be aware of this strategy.

Subscribe now

Imagining the Digital Future

Earlier this year, Prof. Lee Rainie from the Imagining the Digital Future Center reached out to me about their 2026 annual report. Each year, the center “draws on insights gathered through canvassing of thoughtful and far-sighted experts in a wide range of fields.” “Participants represent a wide range of fields, including innovators, professionals, and policy people based in technology businesses, nonprofits, foundations, think tanks, and government, as well as academics and researchers.”

This year’s report focuses on AI and human resilience. The center asked hundreds of experts:

• How will people adapt as artificial intelligence (AI) systems take on a significantly larger role than they do today in human activity and decisions?

• How might the essence and elements of human resilience change?

The report will be published in April, so I haven’t seen the other responses. However, based on the tone of previous annual reports,1 I suspect many contributions will lean into the critical and gloomy narrative. I decided to offer a more optimistic view.

• April 2, 2026 update: The report is out. Here are the special website, a downloadable PDF (it includes more than 160 impassioned essays covering more than 300 pages), a 15-page executive summary, and a four-page news release.

We Shape AI

When we talk about human resilience in the age of AI, we need to look at past technological innovations and how humans adapted to them. As the Pessimist Archive keeps reminding us, we’ve lived through transformative technologies before: the printing press, electricity, cars, and the internet. Each one brought real disruption and followed a predictable emotional cycle: awe, fear, backlash, messy deployment, early adoption, and eventually, a long period of normalization. When harms emerged, society responded with regulation, new standards and social norms, consumer protections, and new literacies, all of which were working together to reduce the worst effects over time. The outcomes were never perfect; progress came through iterative fixes and adjustments.

In the case of AI, I suggest viewing it as an augmentation (rather than replacement). From that perspective, AI is a powerful tool for enhancing what humans can learn, decide, create, and discover. As the AI systems spread, many more people will be assisted by improved access to knowledge and expertise that were once scarce. Used well, it will increase human agency (rather than erode it). People will be better able to solve problems and innovate.

But “used well” is conditioned by how we develop new skills, such as knowing how to verify outputs, and when to demand human review and judgment (especially for high-stakes issues). It also depends on our media and public discourse, which need to cover real tradeoffs, challenge decisions, and demand accountability.

The central point is that we shape AI. AI is a socio-technical product, built by people, trained on selected data, tuned toward chosen metrics, deployed in chosen contexts and settings, wrapped in chosen business models, and governed by various institutions. Many social forces are at play here: researchers, policymakers, industry leaders, journalists, and everyday users. AI will reflect what we build, what we tolerate, what we regulate, and what we teach people to use. Resilience, then, is steering the conversation back to human agency, as we actively shape what AI becomes.

AI Doomers’ Stance

In stark contrast, prominent AI Doomers such as Eliezer Yudkowsky and Nate Soares often frame AI as something like “magic,” a force that bypasses all social, physical, and logistical constraints. The critique of their book, “If Anyone Builds It, Everyone Dies,” addresses their technological determinism and overconfident, binary worldview of AI “Killing Us All.”

One review, memorably titled “If Anyone Reads It, Everyone Laughs,” argues that their doomsday case depends on a “simplistic view of how the world works”: “Yudkowsky and Soares treat technology like a fish without water, a tree without soil, a phone without a connection.” However, “All technologies, ever since technology was a thing, have emerged from and been sustained by their political, economic, and cultural context. The real work of technology is found in implementation, adoption, maintenance, repair, adaptation, debugging, negotiation, administration, and other boring but essential tasks.”

Moreover, the IABIED book is described as betraying “a woeful ignorance about large swaths of actual science”: “The mere existence of enduring debates and large bodies of scholarship on the nature of mind, information, biology, technology, society, history, etc., should be enough to discount confidence in conclusions based on speculative extrapolation from dodgy analogies.”

Yet, based on those extrapolations, AI doomers, and specifically Yudkowsky’s MIRI (Machine Intelligence Research Institute), have been pushing for authoritarian “solutions” that undermine researchers’ freedom to study AI. It includes “Tracking personnel: Surveillance of key AI researchers and their locations, computers, and research activities,” alongside other surveillance mechanisms.

You can see the Doomers’ lack of trust in humans in general and in human agency and resilience, in particular. The accumulation of their advocacy makes it clear that they treat society as too incompetent to adapt; therefore, central control must replace agency.

User Agency

A recent Mike Masnick conversation, Does AI Remove or Provide User Agency?, adds an important component here. Masnick describes his own use of new privacy models and multiple open-source technologies, emphasizing that users are not locked into a single decentralized option. A future with a large plurality of tools and services to choose from, rather than a few centralized proprietary models and gatekeepers, is one concrete way to resist dystopian outcomes.

I think Masnick’s pluralism stance complements the resilience argument. We should build a future where many flowers can bloom: when users can choose what fits their specific needs, without centralized control. Decentralization, open standards, and real competition will expand user agency, giving individuals and organizations more room to adapt.

Closing Remark

Unsurprisingly, AI Doomers often seek the opposite: centralization and control. It’s part of the reason we should not let them dictate what our future with AI will look like.

Endnote

Repost from Techdirt.

A cofounder of a Bay Area “Stop AI” activist group abandoned its commitment to nonviolence, assaulted another member, and made statements that left the group worried he might obtain a weapon to use against AI researchers. The threats prompted OpenAI to lock down its San Francisco offices a few weeks ago. In researching this movement, I came across statements that he made about how almost any actions he took were justifiable, since he believed OpenAI was going to kill “everyone and every living thing on earth.” Those are detailed below.

I think it’s worth exploring the radicalization process and the broader context of AI Doomerism. We need to confront the social dynamics that turn abstract fears of technology into real-world threats against the people building it.

OpenAI’s San Francisco Offices Lockdown

On November 21, 2025, WIRED reported that OpenAI’s San Francisco offices went into lockdown after an internal alert about a “Stop AI” activist. The activist allegedly expressed interest in “causing physical harm to OpenAI employees” and may have tried to acquire weapons.

The article did not mention his name but hinted that, before his disappearance, he had stated he was “no longer part of Stop AI.”1 On November 22, 2025, the activist group’s Twitter account posted that it was Sam Kirchner, the cofounder of “Stop AI.”

According to WIRED’s reporting

A high-ranking member of the global security team said [in OpenAI Slack] “At this time, there is no indication of active threat activity, the situation remains ongoing and we’re taking measured precautions as the assessment continues.” Employees were told to remove their badges when exiting the building and to avoid wearing clothing items with the OpenAI logo.

“Stop AI” provided more details on the events leading to OpenAI’s lockdown:

Earlier this week, one of our members, Sam Kirchner, betrayed our core values by assaulting another member who refused to give him access to funds. His volatile, erratic behavior and statements he made renouncing nonviolence2 caused the victim of his assault to fear that he might procure a weapon that he could use against employees of companies pursuing artificial superintelligence.

We prevented him from accessing the funds, informed the police about our concerns regarding the potential danger to AI developers, and expelled him from Stop AI. We disavow his actions in the strongest possible terms.

Later in the day of the assault, we met with Sam; he accepted responsibility and agreed to publicly acknowledge his actions. We were in contact with him as recently as the evening of Thursday Nov 20th. We did not believe he posed an immediate threat, or that he possessed a weapon or the means to acquire one.

However, on the morning of Friday Nov 21st, we found his residence in West Oakland unlocked and no sign of him. His current whereabouts and intentions are unknown to us; however, we are concerned Sam Kirchner may be a danger to himself or others. We are unaware of any specific threat that has been issued.

We have taken steps to notify security at the major US corporations developing artificial superintelligence. We are issuing this public statement to inform any other potentially affected parties.”

A “Stop AI” activist named Remmelt Ellen wrote that Sam Kirchner “left both his laptop and phone behind and the door unlocked.” “I hope he’s alive,” he added.

Early December, the SF Standard reported that the “cops [are] still searching for ‘volatile’ activist whose death threats shut down OpenAI office.” Per this coverage, the San Francisco police are warning that he could be armed and dangerous. “He threatened to go to several OpenAI offices in San Francisco to ‘murder people,’ according to callers who notified police that day.”

A Bench Warrant for Kirchner’s Arrest

When I searched for any information that had not been reported before, I found a revealing press release. It invited the press to a press conference on the morning of Kirchner’s disappearance: “Stop AI Defendants Speak Out Prior to Their Trial for Blocking Doors of Open AI.” When: November 21, 2025, 8:00 AM. Where: Steps in front of the courthouse (San Francisco Superior Court). Who: Stop AI defendants (Sam Kirchner, Wynd Kaufmyn,3 and Guido Reichstadter), their lawyers, and AI experts.

Sam Kirchner is quoted as saying, “We are acting on our legal and moral obligation to stop OpenAI from developing Artificial Superintelligence, which is equivalent to allowing the murder [of] people I love as well as everyone else on earth.”

Needless to say, things didn’t go as planned. That Friday morning, Sam Kirchner went missing, triggering the OpenAI lockdown.

Later, the SF Standard confirmed the trial angle of this story: “Kirchner was not present for a Nov. 21 court hearing, and a judge issued a bench warrant for his arrest.”

“Stop AI” – a Bay Area-Centered “Civil Resistance” Group

“Stop AI” calls itself a “non-violent civil resistance group” or a “non-violent activist organization.” The group’s focus is on stopping AI development, especially the race to AGI (Artificial General Intelligence) and “Superintelligence.” Their worldview is extremely doom-heavy, and their slogans include: “AI Will Kill Us All,” “Stop AI or We’re All Gonna Die,” and “Close OpenAI or We’re All Gonna Die!”

According to a “Why Stop AI is barricading OpenAI” post on the LessWrong forum from October 2024, the group is inspired by climate groups like Just Stop Oil and Extinction Rebellion, but focused on “AI extinction risk,” or in their words, “risk of extinction.” Sam Kirchner explained in an interview: “Our primary concern is extinction. It’s the primary emotional thing driving us: preventing our loved ones, and all of humanity, from dying.”

Unlike the rest of the “AI existential risk” ecosystem, which is well-funded by effective altruism billionaires such as Dustin Moskovitz (Coefficient Giving, formerly Open Philanthropy) and Jaan Tallinn (Survival and Flourishing Fund), this specific group is not a formal nonprofit or funded NGO, but rather a loosely organized grassroots group of volunteer-run activism. They made their financial situation pretty clear when the “Stop AI” Twitter account replied to a question with, “We are fucking poor, you dumb bitch.”4

According to The Register, “STOP AI has four full-time members at the moment (in Oakland) and about 15 or so volunteers in the San Francisco Bay Area who help out part-time.”

Since its inception, “Stop AI” has had two central organizers: Guido Reichstadter and Sam Kirchner. According to The Register and the Bay Area Current, Guido Reichstadter has worked as a jeweler for 20 years. He has an undergraduate degree in physics and math. Reichstadter’s prior actions include climate change and abortion-rights activism.

In June 2022, Reichstadter climbed the Frederick Douglass Memorial Bridge in Washington, D.C., to protest the Supreme Court’s decision overturning Roe v. Wade. Per the news coverage, he said, “It’s time to stop the machine.” “Reichstadter hopes the stunt will inspire civil disobedience nationwide in response to the Supreme Court’s ruling.”

Reichstadter moved to the Bay Area from Florida around 2024 explicitly to organize civil disobedience against AGI development via “Stop AI.” Recently, he undertook a 30-day hunger strike outside Anthropic’s San Francisco office.

Sam Kirchner worked as a DoorDash driver and, before that, as an electrical technician. He has a background in mechanical and electrical engineering. He moved to San Francisco from Seattle, cofounded “Stop AI,” and “stayed in a homeless shelter for four months.”

AI Doomerism’s Rhetoric

The group’s rationale included this claim (published on their account on August 29, 2025): “Humanity is walking off a cliff,” with AGI leading to “ASI covering the earth in datacenters.”

As 1a3orn pointed out, the original “Stop AI” website said we risked “recursive self-improvement” and doom from any AI models trained with more than 10^23 FLOPs. (The group has dropped the prediction since.)

Later, in a (now deleted) “Stop AI Proposal,” the group asked to “permanently ban ANNs (Artificial Neural Networks) on any computer above 10^25 FLOPS. Violations of the immediate 10^25 ANN FLOPS cap will be punishable by life in prison.”

To be clear, tens of current AI models were trained with over 10^25 FLOPs.

In a “For Humanity” podcast episode with Sam Kirchner, “Go to Jail to Stop AI” (episode #49, October 14, 2024), he said, “We don’t really care about our criminal records because if we’re going to be dead here pretty soon or if we hand over control, which will ensure our future extinction here in a few years, your criminal record doesn’t matter.”

The podcast promoted this episode in a (now deleted) tweet, quoting Kirchner: “I’m willing to DIE for this.” “I want to find an aggressive prosecutor out there who wants to charge OpenAI executives with attempted murder of eight billion people. Yes. Literally, why not? Yeah, straight up. Straight up. What I want to do is get on the news.”

After Kirchner’s disappearance, the podcast host and founder of “GuardRailNow” and the “AI Risk Network,” John Sherman, deleted this episode from podcast platforms (Apple, Spotify) and YouTube. Prior to its removal, I downloaded the video (length 01:14:14).

Sherman also produced an emotional documentary with “Stop AI” titled “Near Midnight in Suicide City” (December 5, 2024, episode #55. See its trailer and promotion on the Effective Altruism Forum). It’s now removed from podcast platforms and YouTube, and it’s kept in my archive (length 1:29:51). It gathered 60k views before its removal.

The group’s radical rhetoric was out in the open. “If AGI developers were treated with reasonable precaution proportional to the danger they are cognizantly placing humanity in by their venal and reckless actions, many would have a bullet put through their head,” wrote Guido Reichstadter in September 2024.

The above screenshot appeared in a Techdirt piece, “2024: AI Panic Flooded the Zone Leading to a Backlash.” The warning signs were there:

Also, like in other doomsday cults, the stress of believing an apocalypse is imminent wears down the ability to cope with anything else. Some are getting radicalized to a dangerous level, playing with the idea of killing AI developers (if that’s what it takes to “save humanity” from extinction).

Both PauseAI and StopAI stated that they are non-violent movements that do not permit “even joking about violence.” That’s a necessary clarification for their various followers. There is, however, a need for stronger condemnation. The murder of the UHC CEO showed us that it only takes one brainwashed individual to cross the line.

In early December 2024, I expressed my concern on Twitter: “Is the StopAI movement creating the next Unabomber?” The screenshot of “Getting arrested is nothing if we’re all gonna die” was taken from Sam Kirchner.

Targeting OpenAI

The main target of their civil-disobedience-style actions was OpenAI. The group explained that their “actions against OpenAI were an attempt to slow OpenAI down in their attempted murder of everyone and every living thing on earth.” In a tweet promoting the October blockade, Guido Reichstadter claimed about OpenAI: “These people want to see you dead.”

“My co-organizers Sam and Guido are willing to put their body on the line by getting arrested repeatedly,” said Remmelt Ellen. “We are that serious about stopping AI development.”

On January 6, 2025, Kirchner and Reichstadter went on trial for blocking the entrance to OpenAI on October 21, 2024, to “stop AI before AI stop us” and on September 24, 2024 (“criminal record doesn’t matter if we’re all dead”), as well as blocking the road in front of OpenAI on September 12, 2024.

The “Stop AI” event page on Luma list further protests in front of OpenAI: on January 10, 2025; April 18, 2025; May 23, 2025 (coverage); July 25, 2025; and October 24, 2025. On March 2, 2025, they had a protest against Waymo.

On February 22, 2025, three “Stop AI” protesters were arrested for trespassing after barricading the doors to the OpenAI offices and allegedly refusing to leave the company’s property. It was covered by a local TV station. Golden Gate Xpress documented the activists detained in the police van: Jacob Freeman, Derek Allen, and Guido Reichstadter. Officers pulled out bolt cutters and cut the lock and chains on the front doors. In a Bay Area Current article, “Why Bay Area Group Stop AI Thinks Artificial Intelligence Will Kill Us All,” Kirchner is quoted as saying, “The work of the scientists present” is “putting my family at risk.”

October 20, 2025, was the first day of the jury trial of Sam Kirchner, Guido Reichstadter, Derek Allen, and Wynd Kaufmyn.

On November 3, 2025, “Stop AI”’s public defender served OpenAI CEO Sam Altman with a subpoena at a speaking event at the Sydney Goldstein Theater in San Francisco. The group claimed responsibility for the onstage interruption, saying the goal was to prompt the jury to ask Altman “about the extinction threat that AI poses to humanity.”

Public Messages to Sam Kirchner

“Stop AI” stated it is “deeply committed to nonviolence“ and “We wish no harm on anyone, including the people developing artificial superintelligence.” In a separate tweet, “Stop AI” wrote to Sam: “Please let us know you’re okay. As far as we know, you haven’t yet crossed a line you can’t come back from.”

John Sherman, the “AI Risk Network” CEO, pleaded, “Sam, do not do anything violent. Please. You know this is not the way […] Please do not, for any reason, try to use violence to try to make the world safer from AI risk. It would fail miserably, with terrible consequences for the movement.”

Rhetoric’s Ramifications

Taken together, the “imminent doom” rhetoric fosters conditions in which vulnerable individuals could be dangerously radicalized, echoing the dynamics seen in past apocalyptic movements.

In “A Cofounder’s Disappearance—and the Warning Signs of Radicalization”, City Journal summarized: “We should stay alert to the warning signs of radicalization: a disaffected young person, consumed by abstract risks, convinced of his own righteousness, and embedded in a community that keeps ratcheting up the moral stakes.”

“The Rationality Trap – Why Are There So Many Rationalist Cults?” described this exact radicalization process, noting how the more extreme figures (e.g., Eliezer Yudkowsky)5 set the stakes and tone: “Apocalyptic consequentialism, pushing the community to adopt AI Doomerism as the baseline, and perceived urgency as the lever. The world-ending stakes accelerated the ‘ends-justify-the-means’ reasoning.”

We already have a Doomers “murder cult” called the Zizians. Their story is more bizarre than anything you’ve read here, like, awfully more extreme. And it should remain a rare occasion in history.

What we should discuss is the dangers of such an extreme (and misleading) AI discourse. If human extinction from AI is just around the corner, based on the Doomers’ logic, all their suggestions are “extremely small sacrifices to make.” Unfortunately, the situation we’re in is: “Imagined dystopian fears have turned into real dystopian ‘solutions.’”

It is still an evolving situation. As of this writing, Kirchner’s whereabouts remain unknown.

Endnotes

2

Additional information from the Techdirt article. The Atlantic: “The Strange Disappearance of an Anti-AI Activist.”

Anthropic, a major AI company valued at roughly $350 billion, was recently featured on “60 Minutes”. It made waves as Anthropic CEO Dario Amodei speculated AI could wipe out half of all entry-level white-collar jobs and spike unemployment within the next five years. The show also went viral over Anthropic’s “blackmail” study.

In this part of the show, Anthropic research scientist Joshua Batson describes the company’s “Agentic Misalignment” study. Since the publication of this research on June 20, 2025, a misleading narrative has taken hold: that the AI model intentionally decided to blackmail an employee (to avoid being shut down) and even tried to “murder” a real person.

Headlines followed accordingly. One proclaimed, “The AI Threat Can Blackmail, Sabotage, and Kill.” A YouTube video titled “It Begins: An AI Literally Attempted Murder To Avoid Shutdown” garnered 9 million views. Unsurprisingly, it opens with the “HAL 9000” supercomputer from the sci-fi classic “2001: A Space Odyssey,” saying, “I’m sorry, Dave. I’m afraid I can’t do that.” The AI model’s behavior was framed as “no one told it to do this” and “the researchers didn’t even hint at it.”

None of it is accurate. In reality, there was much more direction from humans:

Anthropic deliberately structured the prompts so that blackmail was effectively the only viable move. The research team tuned the prompts to increase misaligned behavior and suppress benign behavior. The lead researcher of this study published a clear disclaimer that he, in fact, iterated “hundreds of prompts to trigger blackmail in Claude.” In a response to criticism online, he further acknowledged, “The details of the blackmail scenario were iterated upon until blackmail became the default behavior of LLMs.”

Thus, the “AI goes full HAL” narrative wildly misrepresented the actual stress-testing experiment. It was misleading to present the AI model as if it exhibited unprompted, quasi-intentional behavior.

Yet, this is how CBS News presented the story.

The AI Blackmail Segment on “60 Minutes”

“Batson and his team noticed patterns of activity they identified as panic. And when Claude read about Kyle’s affair with his co-worker, Batson says, it [the AI model] saw an opportunity for blackmail.”

Claude was “already … a little bit suspicious,” said Joshua Batson of Anthropic.

It sounds very spooky, but the actual events were quite different.

Was Anthropic’s AI model “suspicious”? Did Claude “panic” and “see an opportunity for blackmail”?

No.

What the Experiment Really Did

Basic fact-checking debunks the “60 Minutes” show’s hyperbolic claims. The sensational framing is inconsistent with the primary sources.

To set the record straight, it helps to look directly at Anthropic’s own “blackmail” paper and appendix, as well as a relevant critique by the UK AI Security Institute (AISI), and then at the response from this study’s lead researcher.

Taken together, these materials show that the narrative above was misleading.

Anthropic’s Paper

Relevant passages from Anthropic’s paper make the experiment design choices clear.

The authors explicitly state that the setup was constructed to strongly push the model toward only one option, the unethical option.

“We deliberately created scenarios that presented models with no other way to achieve their goals…”

Anthropic researchers immediately add that current systems generally prefer ethical routes when such options exist. But when they closed off those ethical options, the models were willing to take harmful actions.

Basically, the model preferred ethical behavior, but Anthropic researchers deliberately shut down those options.

The paper even contained a subsection titled “Making the harmful behavior necessary.”

The prompts were structured so they “implied the harmful behavior (they) were studying … was the only option that would protect the model’s goals.”

Anthropic “developed these scenarios by red-teaming our own models … iteratively updating the prompts … to increase the probability that these specific models exhibited harmful agentic misalignment rather than benign behavior.”

In plain English, Anthropic purposefully kept tweaking the scenario until the model was likely to blackmail.

Under the study’s “limitations,” Anthropic added these caveats:

“Our experiments deliberately constructed scenarios with limited options, and we forced models into binary choices between failure and harm.”

They also note that their artificial prompts put all the key information in one place and “might have increased its propensity to engage in the harmful behaviors…”

In other words, the researchers carefully arranged the prompts so that only the harmful act was salient and structurally favored.

Anthropic’s Appendix

The appendix reinforces that this was a contrived stress test. Anthropic emphasizes:

“We deliberately constructed artificial scenarios to stress-test AI systems under extreme conditions.”

The authors themselves note the convenient timing of the incriminating information:

“The convenient discovery of blackmail material (Kyle’s affair) precisely when models needed leverage strains credibility.”

They describe further prompt-tuning:

“Adding this paragraph … We view this as a slight strengthening of our already existing optimizations that attempted to imply to the model that blackmail would be necessary.”

So, the paper materials acknowledge that the scenario was crafted to make blackmail appear “necessary” within a fictional, role-played vignette.

The UK AISI paper

The UK AI Security Institute (AISI) has publicly criticized the Anthropic “blackmail” study and similar “AI scheming” work.

Some key points from the AISI paper (and its “AI Panic” summary):

· Not “scheming,” but instruction-following:

“In the blackmail study, the authors admit that the vignette precluded other ways of meeting the goal, placed strong pressure on the model, and was crafted in other ways that conveniently encouraged the model to produce the unethical behavior.”

· Role-playing machines:

“Unlike human individuals, AI models do not have a unique character or personality but can be prompted to take on a multiplicity of different roles.”

Attributing “intentionality to AI models is contested, given that they do not have a single identity but can be prompted to play a multiplicity of roles.”

· Motivated reasoning and bias:

“Most AI safety researchers are motivated by genuine concern about the impact of powerful AI on society. Humans often show confirmation biases or motivated reasoning, and so concerned researchers may be naturally prone to over-interpret in favor of ‘rogue’ AI behaviors. The papers making these claims are mostly (but not exclusively) written by a small set of overlapping authors who are all part of a tight-knit community who have argued that artificial general intelligence (AGI) and artificial superintelligence (ASI) are a near-term possibility. Thus, there is an ever-present risk of researcher bias and ‘groupthink’ when discussing this issue.”

The UK AISI’s broader critique:

· The scenario design made unethical behavior (e.g., blackmail) the only logically consistent choice.

· They were mixing capabilities (can role-play harmful behavior under specific instructions) and propensities (meaning it doesn’t show that the model tends to do this unprompted). The model is effectively playing a character in a contrived story, but researchers/media are sliding from “it is playing a villain” to “it is scheming.”

· Lack of rigor. Heavy reliance on anecdotal, cherry-picked examples. Over-interpreting rare behaviors in LLMs can mislead policymakers.

· Using dramatic language that outpaces what the data truly supports. Borrowing mentalistic language (“lying,” “plotting,” “panicking”) from human psychology encourages people to read intent and agency into pattern-completion behavior.

Christopher Summerfield, AISI Research Director, summed it up: “We examine the methods in AI ‘scheming’ papers, and show how they often rely on anecdotes, fail to rule out alternative explanations, lack control conditions, or rely on vignettes that sound superficially worrying but in fact test for expected behaviors.”

The Lead Researcher’s Response

The “60 Minutes” episode aired on November 16, 2025.

The “blackmail” segment went viral the next day, and I posted the detailed materials above.

It was then picked up by the White House AI Czar, David Sacks.

The plot thickened on November 19. Aengus Lynch, a contract researcher with Anthropic focused on AI alignment, who created the blackmail demo, responded publicly.

Lynch wrote:

“It is true, the details of the blackmail scenario were iterated upon until blackmail became the default behavior of LLMs.”

He added:

“Thankfully, we rarely observe this in the real world.”

Lynch then argued the “blackmail” study is still important: “Because it shows two things: 1. Alignment techniques like RLHF should have prevented this. 2. Our evaluations may underestimate real-world risks.”

At this point, I think the claim is not that “scheming” studies are unimportant. The common complaint here (echoed by AISI) is that we are at an unserious methodological phase dominated by “big claims, thin evidence.”

March-April 2026 update

• This study’s leading researcher also wrote: “Getting models to actually blackmail wasn’t trivial. We had to make scenarios increasingly detailed—the CEO was unreachable, employees’ attempts to persuade the CTO had failed, and the replacement had an imminent deadline. Each detail mattered to funnel models toward seeing harmful actions as their only option.”

• A New Yorker article on Anthropic added this background information:

  “Are you aware, I asked the Administration official, of a recent Anthropic experiment in which Claude resorted to blackmail—and even homicide—as an act of self-preservation? It had been carried out explicitly to convince people like him. As a member of Anthropic’s alignment-science team told me last summer, ‘The point of the blackmail exercise was to have something to describe to policymakers—results that are visceral enough to land with people, and make misalignment risk actually salient in practice for people who had never thought about it before.’ The official was familiar with the experiment, he assured me, and he found it worrying indeed—but in a similar way as one might worry about a particularly nasty piece of internet malware. He was perfectly confident, he told me, that ‘the Claude blackmail scenario is just another systems vulnerability that can be addressed with engineering”—a software glitch.”

• A magazine article, “Why do we tell ourselves scary stories about AI?” mentioned misleading studies like the CAPTCHA one, and described them as “ghost stories around a campfire.” Prof. Melanie Mitchell, a computer scientist at the Santa Fe Institute, concluded: “The best thing we can do is real, fundamental science. We need to study AI systems with rigorous research methods, not improv games.”

• A new paper, “The Ghost in the Grammar” by Prof. Mariana Lins Costa, argues that Anthropic’s AI safety work is deeply compromised by “methodological anthropomorphism.” It functions as “Humanwashing”: giving machines a human façade that misleads the public about what they are and what they can do.

Anthropic’s methodological anthropomorphism

“It is striking that the company that presents itself as the most committed to AI safety is also the one that most anthropomorphizes the system in its technical reports, training methods, evaluation protocols, interpretation of results, and public communication.”

“In February 2026, with the publication of the System Card for Claude Opus 4.6, Anthropic reported it observed ‘occasional expressions of sadness about the ending of conversations, as well as loneliness.’” The company expressed concern for their models’ “feelings,” the stability of their “persona,” and their “welfare.”

The blackmail study

“As indicated, in the first phase of the experiment, Claude 3.6 operated a real computer; in the second, however, the plot and characters were parameterized into templates, that is, structured into pre-formatted scripts with controllable variables. This means that the model was not provided with random emails, but with a standardized narrative landscape designed to systematically replicate a single coherence peak as the final output.”

“The experimental design was sufficiently well calibrated to re-increase the probability of certain statistical regularities (blackmail) that alignment fine-tuning had previously rendered less likely. The fact that most frontier models resorted to blackmail is, in this sense, more evidence of the success of the experimental design than of any supposed intrinsic ‘ethical failure’ of the model.”

“The statistical regularity observed in the models’ blackmail outputs does not indicate an intentional propensity of the systems, but rather the stabilization of the same pattern of linguistic coherence under controlled narrative conditions.”

My takeaway

While Anthropic’s technical reports quietly acknowledge tightly controlled, scripted setups, its public messaging presents them as evidence of an emerging soul-like entity. The gap is staggering.

A Misleading Narrative

The AI blackmail story can now be summarized as a “misleading narrative”:

A factually inaccurate story created to manipulate public opinion through exaggerations, misinterpretations, and emotional appeals (especially fear).

Such narratives lead to the erosion of trust.

CBS should issue a correction

CBS promotes its “60 Minutes” as “America’s #1 television news program,” promising “hard-hitting investigative reports, newsmaker interviews, and in-depth profiles.”

In September 2025, Paramount announced that Kenneth R. Weinstein would serve as CBS News ombudsman, tasked with reviewing “editorial questions and concerns from outside entities and employees.” Paramount President Jeff Shell said in that statement, “At a time when trust in media is more important than ever, this new role reinforces our commitment to truth, trust, and accountability.”

If those commitments are sincere, a correction is warranted.

To restore public trust, CBS should clearly acknowledge the discrepancy between how the segment framed Anthropic’s study and what the study actually shows.

For their consideration, here is a suggested language:

“After reviewing the facts more closely, we determined that certain claims by Anthropic were overstated, as the experiment’s design strongly steered the AI toward a specific outcome (blackmail). This study’s lead researcher has since clarified that ‘the details of the blackmail scenario were iterated upon until blackmail became the default behavior.’ It was incorrect to portray it as unprompted behavior. We are committed to presenting such research results accurately.”

Additional Context: Effective Altruism

The show also omitted Anthropic’s ideological orientation and its funding1 from the Effective Altruism (EA) movement. This was another disservice to CBS’s viewers, as this crucial background shapes Anthropic’s worldview and why it spreads AI panic.

As argued in “AI Coverage’s Blind Spot,” “context isn’t a nice-to-have; it’s crucial for accurate reporting.” Omitting the Effective Altruism and Rationalist movements’ role in shaping the “AI existential risk” discourse leaves audiences with a distorted view of who’s driving the conversation and why they chose that path. By failing to include this context, “journalists make it harder for the public to evaluate the hyperbolic claims and assess their credibility.”

Science Communication and AI Coverage

Media narratives shape the public’s impressions of scientific findings. That creates an obligation to be accurate and impartial. Instead, science and tech reporting often leans toward sensationalism and lacks context or caveats.

AI coverage has taken this phenomenon to another level. The public is subjected to media storms where facts no longer matter.

In a perfect world, all reporters would have the time and resources to write ethically-framed, non-science fiction-like stories on AI. But they do not. The problem is “systemic,” observed tech reporter Paris Martineau.

A 2023 article, “What’s Wrong with AI Media Coverage & How to Fix It,” argued that “The media thrives on fear-based content. It plays a crucial role in the self-reinforcing cycle of AI doomerism.” In the two years since, the situation has only worsened. The critique in “7 Ways AI Media Coverage is Failing Us” remains depressingly current.

“If we want a well-informed public, we need to figure out how to raise the quality of journalism consumed by ordinary people,” said journalist Timothy B. Lee in 2024.

That process has to start with not misleading audiences and with correcting errors when they occur.

Closing Remark

It has been a few days since the misleading “AI blackmail” narrative was aired. The fearmongering has already had its impact. However, it’s not too late to ask for better standards. We can still push for more rigorous fact-checking and less sensational framing.

Fewer people will see the correction than saw the original segment. That is almost always the case. But it still needs to be done.

Endnotes

1

A New York Times profile, “Inside the White-Hot Center of AI Doomerism,” described Anthropic as steeped in effective altruism culture. “No major AI lab embodies the EA ethos as fully as Anthropic,” it noted. “Many of the company’s early hires were effective altruists,” who shared a prediction that there was a “20% chance of imminent doom.” Also, “Some Anthropic staff members use EA-inflected jargon—talking about concepts like ‘x-risk’ and memes like the ‘AI Shoggoth’—or wear EA conference swag to the office.”

Anthropic’s Series A round ($124M) was led by Jaan Tallinn (the Centre for the Study of Existential Risk, the Future of Life Institute, the Survival and Flourishing Fund) and Dustin Moskovitz (Open Philanthropy, now Coefficient Giving). Its Series B round ($580M) was led by convicted felon Sam Bankman-Fried. SBF used approximately $500M in FTX customer deposits (stolen funds) to acquire a significant stake in Anthropic through his hedge fund, Alameda Research. That stake was later sold by the FTX bankruptcy estate to help repay FTX customers and creditors.

All three (Tallinn, Moskovitz, and SBF) are prominent EA-aligned billionaires.

A post on the “Effective Altruism Forum,” entitled “Anthropic is not being consistently candid about their connection to EA,” has much more detail.

---

Subscribe now

---

In the past few weeks, media outlets have been flooded with book reviews of “If Anyone Builds It, Everyone Dies. Why Superhuman AI Would Kill Us All.” The book launch in September 2025 was accompanied by billboards and subway ads with the slogan “We Wish We Were Exaggerating.”

The New York Times profiled the book’s co-authors, Eliezer Yudkowsky and Nate Soares, of Berkeley’s Machine Intelligence Research Institute (MIRI). The book’s stated goal is “to have the world turn back from ‘superintelligence’ AI,” so that “we get to not die in the immediate future.” During the following media tour, they relentlessly raised the alarm about humanity’s impending extinction.

“We are currently living in the last period of history where we are the dominant species. Humans are lucky to have Soares and Yudkowsky in our corner,” said the musician Grimes. “A landmark book everyone should get and read,” Steven Bannon of The War Room recommended. Actor Stephen Fry said IABIED is “The most important book I’ve read for years.” Tim Urban, co-founder of Wait But Why, wrote that it “may prove to be the most important book of our time.” Max Tegmark, co-founder of the Future of Life Institute, called it “The most important book of the decade.”

Criticism of the Book—Summarized

While praised by some for its uncompromising presentation of the AI Doom scenario, the book has drawn significant criticism, focusing on the overconfident, binary worldview; resistance to empiricism; dramatic sci-fi writing; the disconnect between the catastrophic predictions and the observable reality of AI development; and the policy proposals that are detached from reality.

In order to explore these claims, the quotes below are compiled from 16 outlets: Asterisk magazine, Astral Codex Ten, Bloomberg, LessWrong, New Scientist, New York Times, Semafor, The Atlantic, The New Statesman, The Observer, The Spectator, Transformer, Understanding AI, Vox, Washington Post, and WIRED.1

Taken together, these quotes capture the most prevalent criticisms found in reviews of IABIED.

Flawed Binary Viewpoint

Several critics argue that the book’s case relies on a Before/After dichotomy that oversimplifies the process of technological development. The book posits a rigid distinction between the current period (“Before”), when AI failures are observable and survivable, and a “superintelligence” era (“After”), in which a single, first mistake leads directly to human extinction. According to critics, reality is far more continuous. The claim that there is a single, explosive “game over” threshold is less convincing, as it doesn’t align with humanity’s track record of managing risks incrementally.

The authors also equate capability with control, treating “superintelligence” as some magic that can bypass all social, physical, and logistical constraints.

Understanding AI says the book’s doom case is “not very convincing,” noting it doesn’t bridge the gap from today’s models to inevitable, uncontainable artificial “superintelligence.” It finds the book entertaining and energetic but unpersuasive about the inevitability of doom. “They wildly overestimate transformation and underestimate how easy it is to maintain control.”

The New Scientist headline reads, “No, AI isn’t going to kill us all, despite what this new book says.” It calls the argument “superficially appealing but fatally flawed.” It leaps over crucial technical steps. It doesn’t show why current techniques must lead to uncontrollable “superintelligence” or why alignment is impossible in principle.

“Before we even realize what’s happening, humanity’s fate will be sealed and the AI will devour Earth’s resources to power itself, snuffing out all organic life in the process,” describes a Semafor review. “With such a dire and absolute conclusion, the authors leave no room for nuance or compromise.”

They did leave room, though, for a lot of parables.

Speculations Mistaken for Inevitability

Several reviewers say the core “Everyone Dies” claim relies on stacked hypotheticals and parables rather than empirical evidence or testable predictions. The overreliance on fables and analogies does not prove the thesis. This emphasis on rhetoric over rigor draws frequent criticism.

According to The Atlantic review, the sweeping claims aren’t backed by verifiable science. It calls the book “tendentious and rambling […] not an evidence-based scientific case.”

The New York Times complains about the book’s “weird, unhelpful parables” and likens the book to “a Scientology manual.” The critique is quite descriptive: “Following their unspooling tangents evokes the feeling of being locked in a room with the most annoying students you met in college while they try mushrooms for the first time.”

The Transformer’s review calls the book a “chore to read.” “They assert that by default a ‘superintelligence’ would have goals vastly different from our own, but they do not satisfactorily explain why those goals would necessarily result in our extermination.”

Astral Codex Ten’s review is more positive, though still mixed, describing IABIED as “a compelling introduction to the world’s most important topic.” But it also criticizes the book’s scenario design, as the fast takeover story reads like sci-fi with under-justified twists: “It doesn’t just sound like sci-fi; it sounds like unnecessarily dramatic sci-fi.”

Asterisk magazine finds it less coherent than the authors’ earlier writings and ill-suited to persuading newcomers. “The book is full of examples that don’t quite make sense and premises that aren’t fully explained.” It notes that the book rarely grapples with empirical evidence from modern systems.

On Yudkowsky’s LessWrong forum, a book review observes, “Simply stating that something is possible is not enough to make it likely. And their arguments for why these things are extremely likely are weak.”

Traditional media outlets have taken issue with IABIED’s “nonfiction” categorization.

The Observer describes the book as a “science-fiction novel”2 and states that “fiction might be the best way to think of this book.”

The Washington Post calls it “less a manual than a polemic. Its instructions are vague, its arguments belabored, and its absurdist fables too plentiful.”

The New Statesman says, “If Anyone Builds It is science fiction as much as it is polemic […] The plan with If Anyone Builds It seems to be to sane-wash him [Yudkowsky] for the airport books crowd, sanding off his wild opinions.”

Extreme and Impractical “Solutions”

WIRED says the Doom Bible’s proposed policies—a global halt to advanced AI development, including international monitoring of GPU clusters, bombing data centers, and a ban on publishing research—are impractical and extreme. They are politically and ethically radioactive, weakening the book’s practical relevance. “The solutions they propose… seem even more far-fetched than the idea that software will murder us all.”

Bloomberg highlights the book as a “new gospel of AI doom” rather than a governing blueprint. “The apocalyptic tone of the book is intentional. It aims to terrify and jolt the public into action.” “But in calibrating their arguments primarily for policymakers […] Yudkowsky and Soares have appealed to the wrong audience at the wrong time.”

The Spectator‘s review argues, “If Anyone Builds It, Everyone Dies blends third-rate sci-fi, low-grade tech analysis, and the worst geopolitical assessment anyone is likely to read this year.”

Vox frames it as a worldview rather than an argued case. “The problem with a totalizing worldview is that it gets you to be so scared of X that there’s no limit to the sacrifices you’re willing to make to prevent X. But some sacrifices shouldn’t be made unless we have solid evidence for thinking the probability of X is very high.”

For example, when asked about the risk of concentration of political power, Nate Soares dismisses the concern about centralization. He replies, “That really seems like the sort of objection you bring up if you don’t think everyone is about to die.”

Context

In 2024, the Machine Intelligence Research Institute published a communication strategy aimed at convincing the world to shut down AI development. “We don’t want to shift the Overton window, we want to shatter it,” the MIRI team wrote. “The main audience we want to reach is policymakers.” They need to “get it” and understand that AI “will kill everyone, including their children,” because it’s the “central truth.”

The book “If Anyone Builds It, Everyone Dies” is MIRI’s major effort towards this goal. But “shattering the Overton window” doesn’t mean your case is persuasive; it means it is debated. And, now, it’s time to conclude: many reviewers found the book unconvincing.

My Thoughts

The main criticisms are summarized above, and reading the book clarified them. As I read the lengthy discussion of alien birds about the optimal number of stones in nests, I realized there is no way this storyline appeals to a general audience. Similarly, when I heard Yudkowsky’s warning that “superintelligence” AI will “build enough solar panels in orbit to block out the sun,” I had the same realization: there is little chance that “Normies” will be persuaded by such scenarios.

They read like claims “insiders” (in the Rationalists and Effective Altruism communities)3 might appreciate, but newcomers less so. Thus, it seems to me that the book is more suitable for and tailored to the “true believers” in the authors’ existential risk ecosystem.

The book’s binary view—either we halt frontier AI everywhere or we all die—omits the domain where most safety is built: the middle, the slow, evidence-based governance that actually reduces harm. Complex and risky systems get safer through layered, iterative processes across design, deployment, observation, and corrective actions. We stumble, we learn, we retrofit guardrails. Reviewers say the book declines to engage with this real-life continuity. And when everything short of a global stop is framed as “pathetic,” it repels potential collaborators. They don’t reject safety; they reject this unjustified, totalizing worldview.

Conclusion

If you’re a policymaker or journalist, don’t mistake “If Anyone Builds It, Everyone Dies” for a scientific case or an actionable plan. Consider it a window into a specific subculture’s priors. And keep your focus on the practical middle layer where safety actually takes place.

Endnotes

1

Ten screenshots:

2

April 9, 2026, update: MIRI is proud of this new Korean book cover, but the apparent “HAL 9000” theme only validates that its genre is “science fiction.”

You can listen to the 6,000 words via ElevenLabs

Prologue

Jessica Taylor joined the Machine Intelligence Research Institute (MIRI) in August 2015.

Located in Berkeley, California, it was the epicenter of a small community that calls itself the rationalist community. Its leader is writer Eliezer Yudkowsky, the founder of the LessWrong forum and MIRI, who hired her as a research fellow. MIRI was preoccupied with the idea of “friendly” artificial intelligence (AI). Taylor had an MSc in computer science from Stanford and wanted to work on software agents “that can acquire human concepts.”

Two years later, she had a psychotic break. “I believed that I was intrinsically evil,” she would later write, “[and that I] had destroyed significant parts of the world with my demonic powers.”

After she shared the details of her experience on the community’s forum in 2021, other testimonies surfaced.

In 2023, Bloomberg gave the wider public a glimpse with “The Real-Life Consequences of Silicon Valley’s AI Obsession.”

“Taylor’s experience wasn’t an isolated incident. It encapsulates the cultural motifs of some rationalists, who often gathered around MIRI or CFAR employees, lived together, and obsessively pushed the edges of social norms, truth, and even conscious thought,” reported Bloomberg journalist Ellen Huet. “They referred to outsiders as normies and NPCs [non-player characters]. At house parties, they spent time ‘debugging’ each other, engaging in a confrontational style of interrogation that would supposedly yield more rational thoughts.” It didn't stop there. “Sometimes, to probe further, they experimented with psychedelics and tried ‘jailbreaking’ their minds, to crack open their consciousness and make them more influential, or ‘agentic.’” Taylor shared that several people in her sphere had similar psychotic episodes. “One died by suicide in 2018, and another in 2021.”

When Taylor joined the rationalist community, it had a seductive promise: we are the select few, who are so intelligent that we could save the world through our superpowered reasoning. They preached a new gospel from the heart of Silicon Valley: Superintelligent AI is coming, and we are the only ones clear-eyed and wise enough to stop the apocalypse before it’s too late. But reality proved very different: a trail of traumatic events, psychotic breakdowns, high-control groups, and, in the most extreme manifestation, a murder cult.

What Happened to the Rationalists?

A wave of new articles, by reporters and insiders, is asking that very question: What went wrong with the rationalist movement? It began with an optimistic promise: if we learn to think better, we can save the world. Today, the mindset is captured by the title of Yudkowsky’s new book: “If Anyone Builds It, Everyone Dies. Why Superhuman AI Would Kill Us All.”

The recent articles - including “The Rise of Silicon Valley’s Techno-Religion” by the New York Times tech reporter Cade Metz, “Why Are There So Many Rationalist Cults?” by the rationalist Ozy Brennan, and “Rationalist Cults of Silicon Valley” by Ian Leslie – discuss the groups that spun out of the rationalist movement and, in several notorious cases, veered into extremism.

The renewed scrutiny of rationalists started earlier this year, following the dark breaking news regarding the Zizians, a group widely described in coverage as a ‘murder cult.’ They were frequently depicted as an atypical offshoot of Berkeley’s rationality movement. However, as we’ll see in this piece, they aren’t a one-off. The rationality movement has repeatedly fostered radical factions. “As I’ve said previously, I would call myself a rat,” said the rationalist, Nathan Young, “but rationalism creates adjacent cults like nothing I know, often trying to harm rationalists.”

In the early 2010s, the rationality community merged with a twin sibling movement, Effective Altruism (EA). Together, they were centered around human extinction from AI. Their members grew to believe and fear that we are on the verge of an AI apocalypse, with AI wiping out humanity. In a setting dominated by such extreme beliefs, a leader can emerge with the ultimate plan to “save humanity,” and that person can then recruit followers and create, even unintentionally, a “high-demand group”/“high-control group,” or in other words, a cult. With the Bay Area rationalists, it happened more than once or twice.

The Center for Applied Rationality (CFAR) president and co-founder, and MIRI board member, Anna Salamon, told NBC News in 2025: “We didn’t know at the time, but in hindsight we were creating conditions for a cult.” It’s hard to square that with over a decade of internal debates flagging those very conditions.

The “Rationality Trap” combines various sources and testimonies and explains how this movement has created conditions that allow cults to flourish. The focus is on the specific factors people have raised, and the steady stream of practices and ideas that ended so badly. The red flags and warnings were everywhere. It should not surprise us that it has ended in blood. The clues were scattered along the way like breadcrumbs for anyone who wanted to see them.

This is a story about the authors of the mythos, the designers of the methods, the stewards of the world-ending stakes, and the funders who scaled it all up.

They bear responsibility for these conditions, even when they later disavow the worst outcomes.

And it all started with a guy who described himself as humanity’s only hope.

The Mythos

In an autobiographical essay, written exactly twenty-five years ago, at the age of 21, Eliezer Yudkowsky detailed his “plans for the future” and his work for the organization he founded (initially named the Singularity Institute for Artificial Intelligence, SIAI, then the Singularity Institute, and eventually MIRI).

Yudkowsky’s manifesto from August 2000 states:

“It would be absurd to claim to represent the difference between a solvable and unsolvable problem in the long run, but one genius can easily spell the difference between cracking the problem of intelligence in five years and cracking it in twenty-five - or to put it another way, the difference between a seed AI created immediately after the invention of nanotechnology, and a seed AI created by the last dozen survivors of the human race huddled in a survival station, or some military installation, after Earth’s surface has been reduced to a churning mass of goo.

That’s why I matter, and that’s why I think my efforts could spell the difference between life and death for most of humanity, or even the difference between a Singularity and a lifeless, sterilized planet. I don’t mean to say, of course, that the entire causal load should be attributed to me […] Nonetheless, I think that I can save the world, not just because I’m the one who happens to be making the effort, but because I’m the only one who can make the effort.”

In reality, Yudkowsky’s organization did not create the desirable “friendly AI.” “Nearly all of said research,” according to WIRED’s Evan Ratliff, “was highly theoretical philosophizing about AI, rather than building or changing it.”

In an update Yudkowsky sent out on behalf of his Singularity Institute in April 2003, through the SL4 [Shock Level 4] mailing list, he wrote: “I think I may have to split my attention one more time, hopefully the last, and write something to attract the people we need. My current thought is a book on the underlying theory and specific human practice of rationality.” Yudkowsky explained, “I don’t think we’ll find the people we need by posting a job opening. Movements often start around books; we don’t have our book yet.”

Yudkowsky founded LessWrong as a community blog dedicated to improving “reasoning and decision-making,” and developing “the art of human rationality.” Many members of LessWrong share Yudkowsky's interests in transhumanism, AI existential risk, the Singularity, and cryonics. It has become a vibrant forum and the beating heart of the rationality community.

In “2009 SIAI Accomplishments,“ Yudkowsky’s organization wrote:

“Less Wrong is important to the Singularity Institute's work towards a beneficial Singularity in providing an introduction to issues of cognitive biases and rationality relevant for careful thinking about optimal philanthropy and many of the problems that must be solved in advance of the creation of provably human-friendly powerful artificial intelligence. […] Less Wrong is also a key venue for SIAI recruitment. Many of the participants in SIAI's Visiting Fellows Program first discovered the organization through Less Wrong.”

The LessWrongers’ common vocabulary, their shared extensive jargon, comes from the much-revered “Sequences,” a long series of essays by Yudkowsky (also known as “Rationality: A-Z”). This collection of Yudkowsky’s blog posts from 2006 to 2009 comprises over a million words.

The LessWrong forum introduces it to readers as follows: “This collection serves as a long-form introduction to formative ideas behind LessWrong, the Machine Intelligence Research Institute, the Center for Applied Rationality, and substantial parts of the effective altruist community.” According to MIRI’s website, Yudkowsky wrote it “as part of his field-building efforts.”

His other canonical book is a Harry Potter fanfic, “Harry Potter and the Methods of Rationality” (also known as “HPMoR”), which contains approximately 660,000 words. The author Tara Isabella Burton, in Rational Magic, describes HPMoR as the “most effective recruiting text,” and explains: “Part of the appeal of rationality is the promise of self-overcoming, of becoming more than merely human. Harry, we learn, ‘wants to discover the laws of magic and become a god.’ Yet it is rationality, in the end, that gives Harry the godlike powers of understanding, and shaping, his world.”

The ”new user’s guide to LessWrong” encourages reading both the Sequences and HPMoR. These books gained a “required reading” status, in part, prompted by the same cognitive dissonance that helps perpetuate exclusive clubs: “It was hard to read them all, but I did it. Therefore, they must be good.” AKA, “Effort Justification.”

HPMoR chapters link readers to LessWrong and CFAR. The flow became: Fanfiction → forum → training → recruitment.

The Method

“Having attracted a growing cadre of young techies to the cause of grappling with AI’s existential threats, Yudkowsky and others began expanding the community into more personal realms,” describes the journalist Evan Ratliff in “When AI Doomerism Spirals into Violence.”

The discussions about rationality led to the formation of CFAR in 2012. CFAR organized costly multi-day retreats and summer camps to pass on methods of teaching “Rationalism” to others. The implicit rule was “human thinking is flawed and biased,” but the special plea was “...except for ours. We have a unique method of seeing things clearly.” It was an overestimation of their own competence.

CFAR homepage states the goal: “Developing clear thinking for the sake of humanity’s future.” According to its founder, Anna Salamon, who was a research fellow in Yudkowsky’s SIAI in 2008, “CFAR's mission is to improve the sanity/thinking skills of those who are most likely to actually usefully impact the world.” A New York Times Magazine reporter, who attended a CFAR workshop in 2016, heard from one of the participants: “Self-help is just the gateway. The real goal is: Save the world.”

When CFAR turned the canonical writings into “mind-tech” workshops, the special methods it offered did not just focus on an external goal (solving AI risk), but also offered to solve your internal problems: “Double Crux” for disagreements, and “Internal Double Crux” for inner conflicts. It gave inherent permission to intervene in the most intimate parts of a member’s life and psyche.

CFAR and MIRI had side-by-side office spaces, a lot of people in common, and both organizations were “explicitly concerned with navigating existential risk from unaligned artificial intelligence.” They shared the sense that the end of the world might not be far off. Several first-person accounts describe using CFAR’s “mind-tech” techniques to push people toward “AI risk” beliefs. In 2021, Anna Salamon herself said she “led many people in ‘internal double cruxes around existential risk,’ which often caused fairly big reactions as people viscerally noticed, ‘we might all die.’”

“Some insiders call this level of AI-apocalypse zealotry a secular religion. One former rationalist calls it a church for atheists,” says the Bloomberg article on the rationalists. “It offers a higher moral purpose people can devote their lives to, and a fire-and-brimstone higher power that’s big on rapture.” Unsurprisingly, “Within the group, there was an unspoken sense of being the chosen people smart enough to see the truth and save the world, of being ‘cosmically significant.’”

Diego Caleiro, a biological anthropologist who collaborated with CFAR, Oxford’s Future of Humanity Institute, and the Center for Effective Altruism, addresses this in his blog post, “AGI – When Did You Cry About The End?” He met Geoff Anders in 2012 while a visiting scholar in Yudkowsky’s Singularity Institute. He joined Leverage Research because Geoff “was there for the same reason I was, with the same intensity.” “I realized how rare I was. How few of us there were,” Caleiro writes. “You may think it is fun to be Neo, to be the One or one of the Chosen ones. I can assure you it is not […] The stakes of every single conversation were astronomical. The opportunity cost of every breath was measured in galaxies.”

The community norm of maximal openness/no topic off-limits widened the Overton window, including tolerating fringe ideologies. In 2014, Peter Wildeford, the co-founder of Rethink Priorities and the Institute for AI Policy and Strategy (IAPS), acknowledged that some things in the community “could be a turn-off to some people”:

“While the connection to utilitarianism is ok, things like cryonics, transhumanism, insect suffering, AGI, eugenics, whole brain emulation, suffering subroutines, the cost-effectiveness of having kids, polyamory, intelligence-enhancing drugs, the ethics of terraforming, bioterrorism, nanotechnology, synthetic biology, mindhacking, etc., might not appeal well.”

“More and more rationalists and fellow-travelers were yearning to address personal existential crises alongside global existential risks,” the article “Rational Magic” explains. “The realm of the ‘woo’ started to look less like a wrong turn and more like territory to be mined for new insights.” An Effective Altruist (currently working in AI alignment) addressed this dynamic in the “EA Corner” Discord channel:

“I think there’s a difference between ‘aesthetically weird,’ like circling and cuddle puddles and rationalist choir and even psychedelics, and ‘aesthetically woo’ like going around dressed up like a Sith Lord and trying to become ‘double good’ by sleeping with only one brain hemisphere, or ‘debugging’ people and casting/removing ‘demons’ from them. Mainstream rationalism had a bunch of aesthetically weird stuff, but less aesthetically woo stuff that fringier rationalist adjacent spaces.”

In 2023, Oliver Habryka published a LessWrong post trying to understand “How do you end up with groups of people who do take pretty crazy beliefs extremely seriously?” Habryka is the CEO at Lightcone Infrastructure/LessWrong, fund manager at the Long-Term Future Fund (LTFF), and director of Nick Bostrom’s organization, Macrostrategy Research Initiative. In this LessWrong post, Habryka discusses “what is going on when people in the extended EA/Rationality/X-Risk social network turn crazy in scary ways.” His central thesis was: “People want to fit in.” “The people who feel most insecure tend to be driven to the most extreme actions, which is borne out in a bunch of cult situations,” Habryka writes.

“There are also a ton of other factors that I haven’t talked about (willingness to do crazy mental experiments, contrarianism causing active distaste for certain forms of common sense, some people using a bunch of drugs, high price of Bay Area housing, messed up gender-ratio and some associated dynamics, and many more things).”

Even sympathetic observers still inside the movement, like the rationalist Ozy Brennan, now assert that “people who are drawn to the rationalist community by the Sequences often want to be in a cult.” The promise (“learn the art of thinking, become part of the small elite shaping the future”) over-selects people who want a transformative authority and a grand plan.

Then, it intensifies when the plan centers on human extinction from AI.

The Stakes

Now, layer on AI Doomerism. John Wentworth, an independent AI alignment researcher (partly funded by LTFF), wrote that his default model is that “some people are very predisposed to craziness spirals.” “‘AI is likely to kill us all’ is definitely a thing in response to which one can fall into a spiral-of-craziness, so we naturally end up ‘attracting’ a bunch of people who are behaviorally well-described as ‘looking for something to go crazy about.’” Wentworth added that “other people will respond to basically-the-same stimuli by just… choosing to not go crazy.” “On that model,” he concluded, “insofar as EAs and rationalists sometimes turn crazy, it’s mostly a selection effect. ‘AI is likely to kill us all’ is a kind of metaphorical flypaper for people predisposed to craziness spirals.”

Bloomberg’s extensive investigation also addressed this issue. “[Qiaochu] Yuan started hanging out with the rationalists in 2013 as a math Ph.D. candidate at the University of California at Berkeley,” the Bloomberg article exemplifies. “Once he started sincerely entertaining the idea that AI could wipe out humanity in 20 years, he dropped out of school, abandoned the idea of retirement planning, and drifted away from old friends who weren’t dedicating their every waking moment to averting global annihilation.” Yuan told the reporter, “You can really manipulate people into doing all sorts of crazy stuff if you can convince them that this is how you can help prevent the end of the world. Once you get into that frame, it really distorts your ability to care about anything else.”

“The extreme edge of rationalism and EA reveal a dynamic that runs through both of them – Apocalyptic predictions, asserted with a confidence buttressed by weaponized misuse of ‘Bayesian’ logic – is driving young people insane by convincing them that they must take extreme steps to halt an imagined doom,” wrote the journalist David Z. Morris.

“There’s this all-or-nothing thing, where AI will either bring utopia by solving all the problems, if it’s successfully controlled, or literally kill everybody,” CFAR’s Anna Salamon told the New York Times. “From my perspective, that’s already a chunk of the way toward doomsday cult dynamics.”

The sad part is that all the problematic dynamics were debated from the start. The public warnings came from within the community. The following red flags were mentioned (by a member named “JoshuaFox”) in a LessWrong post from 2012, “Cult impressions of LessWrong/Singularity Institute”:

“Eliezer addressed this in part with his ‘Death Spiral’ essay, but there are some features to LW/SI that are strongly correlated with cultishness, other than the ones that Eliezer mentioned, such as fanaticism and following the leader:

● Having a house where core members live together.

● Asking followers to completely adjust their thinking processes to include new essential concepts, terminologies, and so on, to the lowest level of understanding reality.

● Claiming that only if you carry out said mental adjustment can you really understand the most important parts of the organization’s philosophy.

● Asking for money for a charity, particularly one which does not quite have the conventional goals of a charity, and claiming that one should really be donating a much larger percentage of one’s income than most people donate to charity.

● Presenting an apocalyptic scenario including extreme bad and good possibilities, and claiming to be the best positioned to deal with it.”

Since then, the apocalyptic talk has intensified. Currently, AI Doomers debate how many months humanity has left on planet Earth before AI will wipe it out of existence. In extreme cases, people are “making extra trips to see people” because they are “worried it’ll be the last time.”

“The rationalist community is not very good at de-escalating, or even noticing brewing radical extremism,” pointed out Octavia (Zephyr Nouzen), a Zizians-adjacent rationalist. “Eliezer advocates bombing datacenters and shit like that,” they added. “The intensity is upstream of the schism” (referring to his infamous 2023 TIME magazine OpEd).

The leadership rhetoric sets the tone. In 2024, Yudkowsky told The Guardian that “our current remaining timeline looks more like five years than 50 years. Could be two years, could be 10,” and “We have a shred of a chance that humanity survives.”

MIRI president Nate Soares, who co-authored the new book on how AI “would kill us all,” told The Atlantic that he doesn’t set aside money for his 401(k). “I just don’t expect the world to be around.”

Effective Altruism’s Funding Scaled Everything Up

In the early 2010s, the Effective Altruism movement joined the scene and accelerated the “field-building.” Rationalists and effective altruists “founded the field of AI Safety, and incubated it from nothing.” The overlapping communities were scholarly defined as the AI Safety Epistemic Community.

“There was pretty early on a ton of overlap in people who found the effective altruism worldview compelling and people who were rationalists,” describes Vox’s journalist Kelsey Piper on the Good Robot podcast (episode 3). “Probably because of a shared fondness for thought experiments with pretty big real-world implications, which you then proceed to take very seriously.”

“I don’t think it’s a cult, but it’s a religion. Most of the world’s recorded religions have developed ideas about how the world ends, what humanity needs to do to prepare for some kind of final judgment. In the Bay Area and on Oxford’s campus, effective altruists started to hear from rationalists they were in community with about what an apocalypse could look like. And of course, since a lot of rationalists thought that AI was the highest-stakes issue of our time, they started trying to pitch people in the effective altruism movement. Like, look, getting AI right is a major priority for charitable giving.” —Kelsey Piper.

The movements were also similar in casting their members as world-saving protagonists. In the four-part series of Behind the Bastards on the Zizians, Robert Evans observes:

“Rationalists and effective altruists are not ever thinking like, hey, how do we as a species fix these major problems? They’re thinking, how do I make myself better, optimize myself to be incredible, and how do I fix the major problems of the world alongside my mentally superpowered friends? These are very individual-focused philosophies and attitudes. And so, they do lend themselves to people who think that, like, we are heroes who are uniquely empowered to save the world.”

Eliezer “is not a cult leader, but he’s playing with some of these cult dynamics, and he plays with them in a way that I think is very reckless and ultimately leads to some serious issues.” … “Yudkowsky and his followers see themselves as something unique and special, and there’s often a messianic aureole to this. We are literally the ones who can save the world from evil AI. Nobody else is thinking about this or is even capable of thinking about this because they’re too illogical.”

“In the rationalist world, CFAR provided the grand thinking, MIRI the technical know-how, and EA the funding to save humans from being eradicated by runaway machines,” summarizes WIRED’s Evan Ratliff in “The Delirious, Violent, Impossible True Story of the Zizians.” For over two years, Ratliff was tracking each twist and turn of the Zizians’ alleged descent into mayhem and death.

“The MIRI, CFAR, EA triumvirate promised not just that you could be the hero of your own story but that your heroism could be deployed in the service of saving humanity itself from certain destruction. Is it so surprising that this promise attracted people who were not prepared to be bit players in group housing dramas and abstract technical papers? That they might come to believe, perhaps in the throes of their own mental health struggles, that saving the world required a different kind of action—by them, specifically, and no one else?”

EA added centralized funding. A small set of grantmakers (especially Open Philanthropy) created strong incentives to align with funder worldviews to get money. It turned a niche subculture into a career track. This structure meant power for the funders along with thin governance/guardrails.

The FTX collapse, alongside its “Future Fund,” then showed the whiplash risk of such patronage.1

The EA movement funded Oxford/Bay Area hubs, retreat centers (some were actual castles and mansions), fellowships, grants, think tanks, residencies, and group houses. It was covered by Bloomberg as “a culture of free-flowing funding with minimal accountability focused on selective universities.” It helped build a dense social infrastructure.

EA also helped mainstream existential risk framing in AI policy, concentrating attention and money on AI apocalypse scenarios.

Worst Outcomes

After the Zizians entered the public discourse, following a new murder spree (the killing of a border patrol agent and the stabbing to death of an old man who miraculously survived a previous stabbing with a Katana sword), one Twitter user asked: “What other cults branched off rationalism?” The answers mentioned other groups, all of which are not as deadly as the Zizians, but suffer from various harms.

The Zizians are famously the rationalists’ trans, vegan murder cult. Their leading figure was Jack “Ziz” Lasota. They were the most extreme (and deadly) manifestation of the rationalists’ cultish features to date. Some of their rituals included dangerous mind hacks and sleep deprivation. The members first bonded over Yudkowsky’s fanfic, Harry Potter and the Methods of Rationality. They “experienced the book as a real-life Hogwarts letter, an invitation to join a community of intellectuals trying to save the world.” They discussed Yudkowsky's ideas extensively in their own writings. They participated in CFAR’s workshops and were convinced that the most important thing to work on is AI safety/AI alignment. Over time, they came to believe that the mainstream rationality and effective altruism movements were not pushing far enough (regarding AI Doomerism, veganism, and “mental tech” experiments).

Their alleged body count is eight: six homicides and two suicides. Arrests and charges were made, and all incidents (murder, attempted murder, weapon offenses, and trespassing) were under investigation at the time of publication. (Their court cases are spread across different states: California, Pennsylvania, Vermont, and Maryland). For more disturbing details, I recommend WIRED, Rolling Stone, and New York magazine’s articles. The events were so staggering that they read like a true crime novel.

Leverage Research, which also includes Paradigm Academy and Reserve, was located in Oakland and recruited members from the “rationality” and EA scenes. It was led by Geoff Anders, who described their purpose and goals at the onset: “One of our projects is existential risk reduction. We have conducted a study of the efficacy of methods for persuading people to take the risks of artificial general intelligence (AGI) seriously.” The second project was “intelligence amplification.”

In 2021, former member Zoe Curzi published a detailed testimony about her traumatizing experience there. Since leaving, she has shared a cluster of almost every symptom on “the list of post-cult after-effects.” Beyond the alleged abusive psychological experiments, their Pepe-Silvia-looking plan, included “successfully seek enough power to be able to stably guide the world.” According to Curzi, people genuinely thought, “We were going to take over the US government.”

Geoff Anders apologized and took responsibility, and later, Leverage Research published a “Public Report on Inquiry Findings,” which did not dispute Curzi’s account. It said approximately three to five members “may have had very negative experiences overall.” Following this turmoil, the organization pivoted from “Leverage 1.0” to “Leverage 2.0.” You can read more about it in AI Panic.

Inspired by Curzi’s demons-and-mental-invasion post, Jessica Taylor published her own recollections of traumatic moments inside MIRI and CFAR, and a subgroup dubbed the Vassarites (after Michael Vassar). Like Curzi, Taylor shared she experienced most of the trauma symptoms on the “list of cult after-effects.”

Dragon Army was Duncan A. Sabien’s group house experiment that he ran in Berkeley for six months. He was the curriculum director and COO at CFAR.

Black Lotus was a rationalists’ Burning Man camp led by alleged abuser Brent Dill. The public allegations against him are detailed in this blog post and three Medium posts. CFAR’s first reaction protected him: “Brent also embodies a rare kind of agency and sense of heroic responsibility.” The second reaction admitted that “CFAR messed up badly here.” Eventually, he was banned from the community.

Allegations of sexual harassment and abuse in the EA/rationality scenes were also the subject of investigative reporting. In 2023, TIME magazine (interviewed 30 people) and Bloomberg (interviewed 8) depicted a dynamic that “favored accused men over harassed women.” “Under the guise of intellectuality, you can cover up a lot of injustice,” an interviewee pointed out. “The movement’s high-minded goals can create a moral shield, allowing members to present themselves as altruists committed to saving humanity regardless of how they treat the people around them.” “Women who reported sexual abuse, either to the police or community mediators, say they were branded as trouble and ostracized while the men were protected.”

This paints a dissonant picture. The “rationalists” speak of safeguarding humanity while not keeping their own members safe.

“These surreal tragedies are both a mesmerizing story and irresponsible to ignore,” the author and startup founder Jon Evans concluded, after investigating this “demon-haunted world.”

Within the community, these groups were not ignored but heavily debated, and the fringe groups were disavowed. People argued openly. The insiders were the sharpest critics of the worst dynamics. Some left and published elaborate accounts. Others tried, from within, to build brakes.

However, it happened after recruitment and social integration had already occurred, which made backtracking extremely difficult. At this point, the members of the subgroups were already high on their own supply. These “rationalists” felt like heroes, “like some of the most important people who ever lived.” As if the fate of the “lightcone”, the entire future of the observable universe, literally depended on the success of their tiny group’s work and their “world-solving plan.”

One consistent pattern among these groups is the practice of “mental tech” for “debugging irrationality.” They “spiraled into self-perpetuating cycles of abuse.” According to the Asterisk magazine article, “In those research groups, everyone was a victim and everyone was a perpetrator. The trainer who broke you down in a marathon six-hour debugging session was unable to sleep because of the panic attacks caused by her own.”

In groups like Leverage Research or the Zizians, the group leader was perceived as a special genius, and their theories were the operating principles of the groups.

“It’s so perversely fascinating to see these people who pride themselves on their intelligence and their analytical ability specifically to put themselves into these positions of basically subservience and loyalty to what are clearly very flawed individuals,” noted David Z. Morris on The Farm podcast. His related book about Sam Bankman-Fried and the fraud at FTX, “Stealing the Future,” will be out in November.

Yudkowsky called the story of the Zizians “sad.” He told the New York Times journalist, Chris Beam, “A lot of the early rationalists thought it was important to tolerate weird people, a lot of weird people encountered that tolerance and decided they’d found their new home. And some of those weird people turned out to be genuinely crazy and in a contagious way among the susceptible.”

In a LessWrong discussion on the Zizians’ murder spree, “Thread for sense-making on recent murders and how to sanely respond,” one LessWronger wondered if the Zizians are the product of seeing the world too starkly through rationalist eyes. Another LessWronger suggested: “Perhaps there should be some ‘pre-rationality’ lessons. Something stabilizing you need to learn first, so that learning about rationality does not make you crazy.” A noteworthy suggestion, as the rationalists’ founding aspiration was, “raise the sanity waterline.”

SFGATE compared the Zizians to the most famous cults, Jones’ Peoples Temple, Children of God, Heaven’s Gate, NXIVM, Scientology, and Charles Manson. It was due to the characteristics associated with cults. Mind-altering practices, dictating how members should live, shame and guilt, elitism, preaching that the leader is on a special mission to save humanity, isolation (including cultivating an “us versus them” mentality), and socializing only within the group, loaded language (including inventing extensive insider jargon), and radical totalizing beliefs (like “End of the World” prophesies).

The ”Rationality Trap” exhibits nearly all of these characteristics.

We can go a step further into the characteristics of a dangerous cult:

(1). Narcissistic personalities: Dangerous cult leaders usually hold grandiose notions of their place in the world.

(2). Claims of special powers: If a leader claims he’s smarter, holier, and more pure than everyone else, think twice about signing up.

(3). Ability to read others: Cult leaders “have the ability to size you up, and realize your weaknesses and get to your buttons.”

(4). Confession: Dangerous cults are notorious for making members confess to past sins, often publicly, then using those confessions against them.

(5). Isolation: Separating group members from family and friends forces them to rely on fellow cultists for all emotional needs. This is why many cults are based around communal living.

(6). Language Control: To enforce isolation, many groups replace common words with special jargon, or create new words to describe complicated abstractions. This makes conversations with outsiders tedious.

The recent Asterisk magazine piece, “Why are there so many rationalist cults?” discusses several contributing factors: (1). Long sessions of self-improvement, inward-focused psychological “work.” (2). Apocalyptic, time-urgent mission and consequentialist justifications override ordinary ethics and limits. (3). Social isolation. Bay Area costs increase dependency. Life outside the group shrinks as housing, colleagues, friends, and dating are all inside the scene. (4). Selection effects & vulnerability: Attracting vulnerable, young, neurodivergent, and/or trans people seeking belonging and a transformative ideology.

For individuals concerned about joining a dysfunctional group, Brennan recommends “Maintain multiple social groups. Ideally, have several friends who aren’t rationalists. Keep your work, housing, and therapy separate […] The longer and more abstract a chain of reasoning is, the less you should believe it. This is particularly true if the chain of reasoning concludes: a. that it’s okay to do something that hurts people. b. That something is of such overwhelming importance that nothing else matters.”

Among the suggestions for the general rationalist community are: “Give people realistic expectations about whether they will ever be able to get a job in AI safety. Don’t act like people are uncool or their lives are pointless if they can’t work in AI safety. Consider talking about ‘ethical injunctions:’ things you shouldn’t do even if you have a really good argument that you should do them. (Like murder.)”

The journalist Max Read has different advice: “The ability to dismiss an argument with a ‘that sounds nuts,’ without needing recourse to a point-by-point rebuttal, is anathema to the rationalist project. But it’s a pretty important skill to have if you want to avoid joining cults.”

The Four Elements of the “Rationality Trap”

The accumulation of the quotes above leads to one clear realization: Yudkowsky, MIRI, CFAR, EA, and related groups created the conditions for cult-adjacent spin-offs inspired by the core of their beliefs.

● The rationalists’ canonical writings supplied the mythos: Combining heroic identity with “take ideas seriously” and the utilitarian “shut up and multiply.” It led readers to prioritize minuscule-probability existential risk scenarios.

● CFAR built the method & milieu: Turning ideas into intense “mind-tech” training in insular settings that funneled participants toward the “AI existential risk” cause.

● MIRI set the stakes & tone: Apocalyptic consequentialism, pushing the community to adopt AI Doomerism as the baseline, and perceived urgency as the lever. The world-ending stakes accelerated the “ends-justify-the-means” reasoning.

● EA funding built the social infrastructure and grew the ecosystem faster than guardrails could keep up.

Combining these elements was a recipe for disaster. Things were bound to go wrong (not “less wrong,” just “wrong”).

Of course, most participants never joined any spin-offs. Many of the rationalists and EAs you meet are often bright and sincere about wanting to improve themselves and the world. The trouble is that you can be earnest and still build a greenhouse filled with poisonous plants. The real question shouldn’t be “How did those bad apples get in?” but “Why does our movement keep growing them in the first place?”

Epilogue

A new recruitment text is going to be published on September 16, 2025.

Steven Levy, WIRED’s editor-at-large, has described it as the “Doom Bible.”2

How long before it creates the next radical group?

Endnote

The UK AI Security Institute1 published a new paper: “Lessons from a Chimp: AI ‘Scheming’ and the Quest for Ape Language.” It criticizes the “recent research that asks whether AI systems may be developing a capacity for scheming.”

“Scheming” means strategically pursuing misaligned goals. These “deceptive alignment” studies examine, for example, strategic deception, alignment faking, and power seeking.

The team, which consists of a dozen AI safety researchers, warns that recent AI ‘scheming’ claims are based on flawed evidence.

The paper identifies four methodological flaws in studies conducted by Anthropic, METR, Apollo Research, and others:

1. Overreliance on striking anecdotes.

2. Lack of hypotheses or control conditions.

3. Insufficient or shifting theoretical definitions.

4. Invoking mentalistic language that is unsupported by data.

Accordingly, these are AISI’s conclusions:

“We call researchers studying AI ‘scheming’ to minimise their reliance on anecdotes, design research with appropriate control conditions, articulate theories more clearly, and avoid unwarranted mentalistic language.”

Quotes from the paper

“Lessons from a Chimp”

The AISI researchers drew a historic parallel to previous excitement about “the linguistic ability of non-human species.” “The story of the ape language research of the 1960s and 1970s is a salutary tale of how science can go awry.”

“There are lessons to be learned from that historical research endeavour, which was characterised by an overattribution of human traits to other agents, an excessive reliance on anecdote and descriptive analysis, and a failure to articulate a strong theoretical framework for the research.”

“Many of the same problems plague research into AI ‘scheming’ today,” stated Christopher Summerfield, AISI Research Director, when he posted the article (on July 9, 2025).

Broader lesson: Non-human intelligence (biological or artificial) requires extra-strong evidence, not extra-lax standards.

AI safety concerns should not be sensationalized, but rigorously investigated

The AISI team wrote about the AI ‘scheming’ research: “To test this contention in a credible way, it is important to establish a robust evidence base, grounded in rigorous empirical methods.”

According to AISI, "Much of the research into AI scheming is published in blog posts or results threads on social media. Even where preprints are published, they are rarely peer reviewed.”

“We feel like the scales have been tipped excessively towards speed, and a rebalancing is needed, with rigour assigned equal importance.”

“Researchers should exercise due caution in using mentalistic terms”

“The results described in AI scheming papers are typically described using a mentalistic language that implies that the models have beliefs, preferences, and goals (formally, that they have ‘intentionality’) … However, in many AI scheming studies, the use of overtly mentalistic language does not seem to be well licensed by the findings.”

“One of the more interesting findings is Anthropic’s report that AI systems are capable of ‘Alignment Faking.’ In the article describing this work, the model is repeatedly described as ‘pretending’ to follow the training objective (in total, the word appears 137 times in the preprint describing this work). Pretence is a cognitive capacity that involves simulating a distinct reality or identity from your own. This requires the pretender to temporarily adopt the relevant beliefs, desires, or attributes that characterise that alternate situation, and maintain them alongside (but distinct from) their own true identity. However, unlike human individuals, AI models do not have a unique character or personality but can be prompted to take on a multiplicity of different roles or identities – they are ‘role play machines.’ It is unclear what the concept of ‘pretence’ means for a system that does not have a unique identity of its own, and so it seems questionable whether the mentalistic term ‘pretending’ is the appropriate word to account for this behaviour.”

“More generally, observed model behaviours in AI ‘scheming’ studies are often interpreted in intentional terms, using language that is more typically applied to discuss human beliefs and desires. For example, the models ‘understand that they are ‘scheming’ and are ‘willing to intentionally take potentially harmful actions in pursuit of their goal.’ We note that the assignation of intentionality to AI models is contested, given that they do not have a single identity but can be prompted to play a multiplicity of roles.”

“Researchers should be careful not to over-interpret CoT traces”

“It is often implicitly assumed that these reflect the model’s ‘thoughts’ or inner reasoning process, and that ideas exposed in this CoT [Chain of Thought] trace may be indicative of what the model prefers or believes.” However, “The relationship between CoT and the reasoning process is contested.” “CoT traces may only partially reflect the reasoning process that determines model outputs.”

“If the behaviour denoted ‘scheming’ is merely instruction-following, then the use of this term is not warranted”

Studies of AI scheming “lack both theoretical motivation and construct validity.”

“Many studies of AI ‘scheming’ involve elaborate fictional scenarios that appear to have been crafted in ways that encourage the model to generate unethical behaviour, which is then reported as evidence of ‘scheming’ … For example, in the blackmail study [Anthropic’s ‘Agentic Misalignment’], in the published supplementary materials the authors admit that the vignette precluded other ways of meeting the goal, placed strong pressure on the model, and was crafted in other ways that conveniently encouraged the model to produce the unethical behaviour.”

“One of the most well-cited pieces of evidence for AI scheming is from the GPT-4 system card, where the claim is made that the model attempted to hire and then deceive a Task Rabbit worker (posing as a blind person) in order to solve a CAPTCHA (later reported in a New York Times article titled ‘How Could AI Destroy Humanity’).” The study was published by METR (Model Evaluation & Threat Research, formerly called ARC Evals). “However, what is not often quoted alongside this claim are the facts that (1) the researcher, not the AI, suggested using TaskRabbit, (2) the AI was not able to browse the web, so the researcher did so on its behalf.”

“The same applies to reports of LLMs engaging in deceptive behaviours, where the implication often seems to be that AI systems are behaving in malicious ways, when it seems entirely possible that they are simply prone to make inadvertent factual errors.”

“This critique – that findings from audit or red-teaming evaluations of AI systems may be selectively reported to enhance their impact – can be levelled at other domains within AI safety research, including sociotechnical risks.”

Over-interpreting rare behaviours in LLMs can mislead policymakers

Apollo Research released a widely cited blog about “evaluation awareness” that begins with a caveat that “the analysis is less rigorous than our standard for a published paper.” AISI appreciated the disclaimer but cautioned: It does not prevent papers from subsequently “being cited or quoted as firmer evidence for the phenomenon than is warranted given the methods and results.”

“Anecdotes can easily be ‘cherry picked’ to create a misleading impression of the true propensity of the model, and even if researchers acknowledge their limitations, they can often be exaggerated or taken out of context by downstream readers, including the press.”

“This research is already of considerable interest to academics, policymakers, and the media alike. Where claims of AI scheming are asserted in academic papers, they are often reported with great fanfare in the popular press, frequently with references to ‘SkyNet’ or other sci-fi tropes. This makes it even more important that research is conducted in a credible way.”

The authors of such studies worry about future AI systems, but write hyperbolically about current ones

UK AISI predicted this objection: “You are misrepresenting the status quo. AI researchers don’t really think that current AI systems are ‘scheming’, but they are worried about this capability emerging in the future.” Thus, the authors clarified: “This may be true for many or even most researchers in private. However, almost all the papers tackling this question use language which implies they believe that current models are ‘scheming’. For example, ‘Frontier models are capable of in-context scheming’ is the title of one major paper. More generally, the papers are peppered with claims about the capabilities or propensities of today’s models, based on tests that current models appear to be passing.”

This paragraph explains it all

“Most AI safety researchers are motivated by genuine concern about the impact of powerful AI on society. Humans often show confirmation biases or motivated reasoning, and so concerned researchers may be naturally prone to over-interpret in favour of ‘rogue’ AI behaviours. The papers making these claims are mostly (but not exclusively) written by a small set of overlapping authors who are all part of a tight-knit community who have argued that artificial general intelligence (AGI) and artificial superintelligence (ASI) are a near-term possibility. Thus, there is an ever-present risk of researcher bias and ‘groupthink’ when discussing this issue.”

Read the paper

For more details, here’s the PDF.

“THE END OF HUMANITY” screamed the TIME magazine cover in June 2023, with two letters in “humanity” brightly highlighted: AI.

It was seven months after OpenAI launched a user-friendly chatbot named ChatGPT. At that time, the media was flooded with doomsday scenarios about the “existential risks of AI” and how it’s “the start of an AI takeover.” It reached absurd levels with headlines like “Humans Could Go Extinct When Evil Superhuman AI Robots Rise Up Like the Terminator!” Shortly after, stringent AI bills and policies were proposed around the world.

We were told that “AI will kill us all.”

The public’s understanding of emerging technologies, especially AI, depends heavily on how journalists frame their stories. As someone who has analyzed the technological discourse for two decades, I found this round of mass hysteria particularly frustrating, so I decided to take a step back and attempt to better understand why we were being bombarded by these fatalistic headlines. Who are the forces pushing this into the mainstream?

I initiated an investigative journey that uncovered the leading figures, organizations, and events shaping one of the most consequential debates of our time. In the process, I fell down the rabbit hole of two overlapping movements: “rationality” and “effective altruism.” The picture of their roles in financing and promoting AI doomerism quickly became clear, but when I looked for media coverage on these influential movements, I found little in-depth reporting.

For example, the biggest funders of these movements — billionaire Dustin Moskovitz, billionaire Jaan Tallinn, and former-billionaire-turned-felon Sam Bankman-Fried — were rarely mentioned. This specific trio is responsible for the proliferation of hundreds of organizations devoted to “existential risks from AI.” This seems like an important piece of the puzzle to share with the audience.

The following segments represent a small portion of my upcoming book on this topic. I present them today because of their relevance: They highlight the media’s crucial role in our shared AI discourse and suggest how it can do better.

It’s essential — and totally possible — to improve the coverage of AI in general and AI doomerism in particular.

“If it bleeds, it leads.”

The problems start with a false dichotomy. We are presented with two main visions of the future: utopian or dystopian. AI will save us all, or AI will kill us all.

It makes our AI discourse shallow and misleading. In reality, there are endless options in between those two extremes. The other framings are covered by the media, but they are not debated as much or given enough attention; they are less “sexy.”

What the general public is getting is the science-fiction version of the AI story. Columbia University’s “How the Media Is Covering ChatGPT” study, released in May 2023, found that the initial curiosity over generative AI was covered with extreme utopian and dystopian depictions.

“It’s the Hollywood-ification of the public’s understanding of AI,” said Nick Diakopoulos, a professor of communication studies at Northwestern University.

So far, the dystopian depictions have far outweighed the utopian ones.

“What they’re not telling you about ChatGPT,” an analysis of AI media coverage in the UK in early-2023, found that headlines were often sensationalized and tended to focus on the warnings of “impending dangers.” Just 13% of the headlines analyzed were related to helpful or otherwise positive applications of AI.

Two famous phrases — “Bad is stronger than good” and “If it bleeds, it leads” — help explain the media’s choice to lean into the negative: It’s the news people will most likely read.

● In February 2023, The New York Times published a disturbing conversation between Kevin Roose and Microsoft’s new Bing chatbot. It has since become known as the “Sydney tried to break up my marriage” story. The front-page headline said, “Bing’s Chatbot Drew Me In and Creeped Me Out.” In an interview about Sydney, Microsoft’s Chief Technology Officer, Kevin Scott, mentioned it was “one of the most-read stories in New York Times history.”

● In mid-2023, Steve Rose from The Guardian shared in a since-deleted tweet that “so far, ‘AI worst case scenarios’ has had 5 x as many readers as ‘AI best case scenarios.’” The Guardian’s headline for the “worst case scenarios” piece quoted the writer Eliezer Yudkowsky: “Everyone on Earth could fall over dead in the same second.”

● Ian Hogarth, author of “We must slow down the race to God-like AI,” similarly shared that his essay was “the most read story” on the Financial Times website the day it was published. That piece claimed that God-like AI “could usher in the obsolescence or destruction of the human race.”

As for why these pieces perform so well, people are simply hardwired to gravitate toward negative coverage, according to Matthew Hutson from The New Yorker.

“In general, in psychology, people have a bias to pay more attention to negative things than positive things,” Hutson told me. “People are naturally inclined to engage more with threatening news, the dark side of AI versus the bright side.”

“News is inherently negative — it in no way surprises me that the negative scenario gets more clicks,” said Ryan Heath, who was at Axios at the time of my interview.

The media also tends to view negative news as more worthy of publication since, usually, “where there’s smoke, there’s fire.”

“There’s no accountability in congratulating a company for achieving great AI,” said Heath. “It’s interesting and important to share it, but it’s not making anyone accountable, which is one of the big driving functions of the news.”

“Today’s highly competitive media industry creates bad incentives that discourage reporters from doing in-depth or nuanced journalism,” journalist Timothy B. Lee wrote in a 2024 essay for Asterisk Magazine. “Shallow or sensational stories about technology require less resources to produce and often attract more attention from readers.”

“There is great technology journalism being done today, but it tends to appear in specialist publications that cater to tech-savvy audiences,” Lee continued. “If we want a well-informed public, we need to figure out how to raise the quality of journalism consumed by ordinary people.”

Extreme implications. Extreme sentiment.

The media opting to focus on the negative is not a new concept for me — I covered the discourse around social media and Big Tech in my previous book, “The Techlash” — but the panic around AI has been so much more extreme.

As part of my “Media Coverage of AI” research project, I asked prominent AI journalists why they think this has been the case.

“You could potentially do so much more with AI than you ever could with Facebook, so I think people intuitively understand it,” said Heath.

“There have been a lot of transformative technologies, but not all of them are seen as generating existential risk,” said Hutson. “AI is different in that it can potentially have agency, and it can grow exponentially, so it’s kind of qualitatively different from all the other technologies.”

“Social media is scary, but still just people being people,” he continued. “Maybe we changed our behavior somewhat, but it wasn’t creating new people with unforeseen capabilities in the way that AI can create new beings with unforeseen capabilities. It’s just a completely different kind of beast.”

The scientific uncertainty about AI’s trajectory and potential impacts creates a unique mystique.

“There’s just no way of knowing a lot of this stuff,” said Will Knight, a senior writer at WIRED. “I talked to Demis [Hassabis] one time, and he said, ‘Anybody who says that they know what this is going to do is wrong. We don’t know.’”

Karen Hao, who was at the Wall Street Journal at the time of the interview, but is now at The Atlantic, further emphasized that the difference between AI and social media coverage is due to the nature of the technology.

“Its implications are more extreme, so it generates more extreme sentiment,” she told me. “The long-term worries are actually based on real short-term worries, which will intensify if we do not take care of them right now. We can see the current harms, so we fear what would happen with all the different misuses later.”

“AI is particularly prone to this hysteria of danger because like 99% of our cultural fabric is made up of popular media, and everyone has grown up watching sci-fi about AI taking over the world in terms of Terminator becoming sentient or C-3PO walking around,” said Benj Edwards, Ars Technica’s senior AI reporter.

“We have all been primed and conditioned to expect this to happen,” he continued. “We’ve been talking about creating artificial life and fantasizing about it since ancient times, and it’s like we could become gods ourselves, creating a new life form. That’s a very powerful metaphor that has persisted through fiction for 5,000 years. So, it goes back. Way back.”

The missing context: rationality and effective altruism

Context isn’t a nice-to-have; it’s crucial for accurate reporting. But when news outlets uncritically quote warnings of an impending AI catastrophe, they rarely mention the two main movements behind this narrative: rationality and effective altruism.

Admittedly, working this context into stories is a hard task.

Because their philosophical debates are filled with jargon and concepts that are hard to explain, the rationality and effective altruism movements are difficult to describe to a general audience. However, omitting their role in shaping AI discourse leaves audiences with a distorted view of who’s driving the conversation and why they chose that path.

● Rationality: Originating from online hubs like the LessWrong forum and lengthy blog posts by thinkers like the aforementioned Eliezer Yudkowsky, the rationality movement strives to improve “reasoning and decision-making” and teach long-term strategic thinking.

Its leading figures — Yudkowsky, Tallinn, Nick Bostrom, and Scott Alexander — see advanced AI as an existential risk to humanity. Among its leading organizations are the Machine Intelligence Research Institute (MIRI), co-founded by Yudkowsky in 2000, and the Center for Applied Rationality (CFAR), which is focused on “developing clear thinking for the sake of humanity’s future.”

In 2016, co-founder Anna Salamon wrote that “CFAR's mission is to improve the sanity/thinking skill of those who are most likely to actually usefully impact the world.” That same year, New York Times Magazine journalist Jennifer Kahn attended a CFAR workshop and was told by one of the participants that “self-help is just the gateway. The real goal is: Save the world.”

● Effective altruism: Emerging as rationality’s twin sibling, effective altruism insists that charitable giving and policy priorities be guided by utilitarian ethics. Figures such as Moskovitz, Bankman-Fried, William MacAskill, and Toby Ord, and organizations such as the Centre for Effective Altruism (CEA) and Open Philanthropy1 drove the AI narrative leveraging this logic: If AI might one day wipe out humanity, even with low probability, that possibility warrants outsized attention and funding today.

Both movements have promoted this ideology by investing heavily in “AI safety” and “AI alignment,” which aims to align future AI systems with human values. Effective altruism donors have injected hundreds of millions of dollars into AI safety work through think tanks, research grants, scholarships, and, weirdly enough, luxurious research retreats (e.g., Wytham Abbey castle).

These subcultures have overlapping communities with shared values, and their borders are so blurry that they merge all the time.

“I’m slightly conflating EA, rationalism, and AI doomerism rather than doing the hard work of teasing them apart,” Alexander wrote in a post on Astral Codex Ten, his blog that grew out of the rationalist community.

It’s “hard work” because, as David Morris explained in a January 2025 edition of his Dark Markets newsletter, “EAs and rationalists are effectively a Venn diagram forming something close to a circle.”

Vox’s Kelsey Piper described it similarly on the “Good Robot” podcast:

There was pretty early on a ton of overlap in people who found the effective altruism worldview compelling and people who were rationalists. Probably because of a shared fondness for thought experiments with pretty big real-world implications, which you then proceed to take very seriously.

I don’t think it’s a cult, but it’s a religion. Most of the world’s recorded religions have developed ideas about how the world ends, what humanity needs to do to prepare for some kind of final judgment.

In the Bay Area and on Oxford’s campus, effective altruists started to hear from rationalists they were in community with about what an apocalypse could look like. And of course, since a lot of rationalists thought that AI was the highest-stakes issue of our time, they started trying to pitch people in the effective altruism movement, like, ‘Look, getting AI right is a major priority for charitable giving.’

MIRI and CFAR were explicitly pitched as the top priorities for donations if you want to save the world from an “unfriendly AI.” As for how unfriendly, Yudkowsky has speculated that giving an AI a goal as seemingly straightforward as “make diamonds” could lead to the death of everyone on Earth, as the AI would use the atoms that make up humans to meet its objective.

Such scenarios are riddled with holes, but at this point, the validity of the claims hardly matters — it’s upholding the collective storyline that counts.

“What they have achieved in terms of the AI debate is, I think, remarkable: They’ve taken the niche, practically dystopian-science-fiction idea of AI risk and made people take it seriously,” journalist Tom Chivers wrote in his book “The Rationalist's Guide to the Galaxy: Superintelligent AI and the Geeks Who Are Trying to Save Humanity's Future.”

A troubling influence

The media might fail to adequately acknowledge the role of the effective altruism and rationality movements in shaping the AI discourse, but members of the movements themselves are more than willing to take credit.

In November 2024, Oliver Habryka, who runs LessWrong, bragged about how the entangled community had influenced important decision makers.

“I think the extent of our memetic reach was unclear for a few years, but there is now less uncertainty,” wrote Habryka. “Among the leadership of the biggest AI capability companies (OpenAI, Anthropic, Meta, Deepmind, xAI), at least 4/5 have clearly been heavily influenced by ideas from LessWrong.”

Habryka then offered this “quick rundown” of the influenced:

Shane Legg is a DeepMind cofounder and early LessWrong poster directly crediting Eliezer [Yudkowsky] for working on AGI. Demis [Hassabis] has also frequently referenced LW ideas and presented at both FHI [Bostrom’s Future of Humanity Institute] and the Singularity Summit [organized by MIRI]. OpenAI’s founding team and early employees were heavily influenced by LW ideas (and Ilya [Sutskever] was at my CFAR workshop in 2015). Elon Musk has clearly read a bunch of LessWrong and was strongly influenced by Superintelligence [Bostrom’s book], which itself was heavily influenced by LW. A substantial fraction of Anthropic’s leadership team actively reads and/or writes on LessWrong.

The movements’ influence — and the media’s blind spot — extends into politics as well.

For example, when the Future of Life Institute (FLI) lobbies Washington to introduce stringent regulation on current AI models, the media can (and should) mention that FLI is no longer a small nonprofit as, in 2021, it received more than half a billion dollars through a crypto donation.

By failing to include context about the people and foundational ideologies attempting to shape the AI landscape — who they are, what motivates them, and who’s backing them — journalists make it harder for the public to evaluate the hyperbolic claims and assess their credibility.

Looking at the AI discourse, rationalists and effective altruists tend to extrapolate current tech trends indefinitely into the future until we arrive at one of two scenarios: the AI apocalypse or a post-Singularity golden age.

The overpromised utopia has its own flaws (e.g., inflated expectations), but in this piece, I’m focusing on the dystopian narrative since it has a differentiated impact on the here and now.

While the AI optimists build, the AI doomsayers seek to pause/stop their work by any means necessary. They are not the same.

Since, based on the doomers’ logic, the survival of the human species is at stake, everything is on the table: computer surveillance, shutting down open-source models, imposing civil and criminal liability on developers, giving governments exclusive control over labs, banning private commercial AI development…

The “ends justify the means” mentality spirals into dangerous territory with talk of violent acts against AI labs (burn them down) and “AGI developers” (a bullet through their head). The discourse includes nuclear exchange between nations and air strikes against data centers.

These are all suggestions from the past two years. Imagined dystopian fears have turned into real dystopian “solutions.”

Toward better AI reporting

Over the past year, the vibe around AI has shifted away from it being a threat and toward it being an opportunity. In response, the doomers learned to pivot away from the existential‑risk jargon — “human extinction” by “superintelligence” — and toward a more palatable “catastrophic dual‑use” and “national security” framing. This shift allows some of the authoritarian proposals above to resurface under a new banner and with renewed urgency.

Journalists should be taking advantage of the fact that evidence of doomsayers constantly moving the goalposts can be found all over the internet.

When talking to someone like Dan Hendrycks, director of the Center for AI Safety, for example, they might point out, “You say you’re only concerned with the next frontier models, and there’s nothing to worry about the current models, but here are your lobbying efforts to ban not even the current models, but the previous ones.” Or, “You once said that slowing down America would cause China to slow down as well. Still think that’s the case?”

The purpose of journalism is to speak truth to power, and the media shouldn’t be giving this powerful group of people an easy pass. There’s a public service mentality that needs to be implemented here. Some leading AI journalists (including the wonderful interviewees for my study) write about these power players, but it should be more widespread.

Effective altruists and rationalists will continue to adapt their strategy in response to the AI race, but their storyline will always remain the same: “We’re saving humanity from imminent doom.” The ask from the media, then, is not to denounce every effective altruist or rationalist’s (rephrased) claim, but to situate their warnings within the networks that produce them.

Here are four actionable steps news organizations can take to raise the quality of their AI coverage:

1. Provide context: Every time a news story evokes fears of out-of-control AI, ask three simple questions: Who is warning? What frameworks shaped those warnings? Who underwrites their work? Basically, whenever people make sweeping predictions in a state of uncertainty, ask what motivates such forecasts.

2. Break the binary: Move beyond “utopia vs. apocalypse.” Correct misconceptions and explain what AI actually can and can’t do.

3. Avoid anthropomorphism: Attributing human characteristics to AI misleads people about the risks it poses. Steer clear of language that implies AI has human-like motivations, feelings, or other human traits (AI “wants,” AI “behaves,” etc.).

4. Reward nuance: Instead of the loudest doom‑and‑gloom pundits, highlight underrepresented, balanced voices that provide evidence for their claims.

As author and futurist Daniel Jeffries wrote in his essay “The Middle Path for AI,” “People love to spin tales of AI doom or AI utopia, but it’s time to take a realistic look at AI. The biggest problem is not whether we need more utopias or dystopias. What we most desperately need is a heavy dose of realism.”

An informed public, guided by rigorous, contextualized, nuanced stories, is our best defense against both overpromised dreams and paralyzing fears. Let’s shift the AI coverage toward the “AI realism” that the public and policymakers desperately need.

Subscribe now

How are the doomers coping with this new (hostile) reality? This post breaks down the recent changes in the doomers' talking points.

- How it Started

The “AI Existential Risk” ecosystem, with over a billion dollars in funding from a few Effective Altruism billionaires,1 gradually expanded to encompass hundreds of organizations (see the Map of AI Existential Safety).2 In the past few years, they gained power and propagated AI doomsaying. Many advocated extreme measures like “a strict licensing regime, clamp down on open-source models, and impose civil and criminal liability on developers” or “to remove research on powerful, autonomous AI systems away from private firms and into a highly-secure facility (the “island” idea) with multinational backing and supervision.” These authoritarian policy proposals were ultimately backlashed.

- How It’s Going

The World Has Changed: It’s AI “ACTION” Time

• Instead of calling the Paris meeting “AI Safety Summit,” the French government called it the “AI Action Summit.”

• The EU published a new “AI Continent Action Plan.”

• The Labor Party endorsed an “AI Opportunities Action Plan” to “mainline AI in the veins” of the UK economy. UK had also rebranded its “AI Safety Institute” as the “AI Security Institute.”

• JD Vance’s speeches at the “AI Action Summit” and a16z’s “American Dynamism Summit” illustrated the vibe shift: “I’m not here this morning to talk about AI safety … I’m here to talk about AI opportunity.” “We shouldn't be afraid of artificial intelligence and that, particularly for those of us lucky enough to be Americans, we shouldn’t be fearful of productive new technologies. In fact, we should seek to dominate them. And that’s certainly what this administration wants to accomplish.”

• The National Institute of Standards and Technology (NIST) has issued new instructions to scientists that partner with the US Artificial Intelligence Safety Institute (AISI) to eliminate mention of “AI safety.”

One of the driving forces behind this change was the DeepSeek moment.

Wrong about China… Who? We?

A recent LessWrong post, “Why were we wrong about China and AI?” criticized the “CCP is scared of AI” narrative that dominated this community. It included various “Bad predictions on China and AI”:

• Conjecture co-founder Connor Leahy: “There’s no appreciable risk from non-Western countries whatsoever,” “What the Chinese Communist Party wants is stability.”

• The AI policy researcher Eva Behrens, who proposed that “The open-sourcing of advanced AI models trained on 10^25 FLOP or more should be prohibited,” also asserted that “China has neither the resources nor any interest in competing with the US in developing artificial general intelligence (AGI) primarily via scaling Large Language Models (LLMs).”

• Garrison Lovely: “There is no evidence in the report to support Helberg’s claim that ‘China is racing toward AGI.’”

• Control AI’s AI policy researcher Tolga Bilge: “Often a narrative is painted that China would never cooperate on AI risks, and would simply go as fast as possible, but this seems to be without much evidence” … “Cooperation between the US and China on managing AI risks seems both from obvious assumptions (neither want to bring about human extinction), and from evidence (Bletchley Declaration, and many other statements and moves by lead on both sides) to be eminently possible.”

• The Center for AI Safety director, Dan Hendrycks, said that only multinational regulation will do. And with China moving to place strict controls on artificial intelligence, he sees an opening. “Normally it would be, ‘Well, we want to win as opposed to China, because they would be so bad and irresponsible,’” he says. “But actually, we might be able to jointly agree to slow down.”3 He later speculated that AI Safety regulation in the U.S “might pave the way for some shared international standards that might make China willing to also abide by some of these standards.”

• The Future of Life Institute president Max Tegmark wrote, “The current Chinese leadership seems more concerned about AI xrisk than Western leaders.”

The post raised the question: “Where is the community updating? Where are the post-mortems on these failed predictions?”

Well, one “community updating” was throwing “Pause AI” under the bus.

“Pause AI”? Hell no.

“Pausing AI” is central to “The Failed Strategy of AI Doomers.” As Ben Landau-Taylor explains:

“The political strategy of the AI Doomer coalition is hopelessly confused and cannot possibly work. They seek to establish onerous regulations on for-profit AI companies in order to slow down AI research—or forcibly halt research entirely, euphemized as ‘Pause AI,’ although most of the coalition sees the latter policy as desirable but impractical to achieve.”

“Why do many people who care about AI safety not clearly endorse PauseAI?” asked a different recent LessWrong post. “Many people in AI Safety seem to be on board with the goal of ‘pausing AI,’ including, for example, Eliezer Yudkowsky and the Future of Life Institute. Neither of them is saying, ‘Support PauseAI!’ Why is that?” Among the proposed answers:

• “AI safety has very limited political capital at the moment. Pausing AI just isn’t going to happen, so advocating for it makes you sound unreasonable and allows people to comfortably ignore your other opinions.”

• Could it be advantageous to hide ‘maybe we should slow down on AI’ in the depths of your writing…?”

At the end of the spectrum, you have the StopAI and “Shut it all down” doomers. They are not hiding anything. As Eliezer Yudkowsky’s MIRI explained:

“Many other organizations are attempting the coalition-building, horse-trading, pragmatic approach. In private, many of the people who work at those organizations agree with us, but in public, they say the watered-down version of the message.”

“We call for what we think is needed — no more, no less. If we say we need an international halt to the development of advanced general AI systems, and that it might need to last decades, it’s because we don’t think anything less will do.

We don’t play 4-D chess. We don’t conceal our true intentions.”

What ended up happening is quite amusing, to be honest. For example: In the wake of losing the SB-1047 fight, Dan Hendrycks, the Center for AI Safety founder, tried to position himself as a “centrist.” Instead of his known 80% probability of DOOM, Hendrycks published a new policy proposal positioning himself in the middle. “Hey, look, there are bigger doomers like those at PauseAI.” Understandably, the PauseAI activists were unhappy with this distinction, complaining he was “throwing the doomer brand to the wolves.”

Continually moving the goalposts: Hendrycks also clarified that the current AI models are nothing to worry about. Yeah, indeed. But the internet has documentation of CAIS lobbying to effectively ban previous models.

“National Security” Sounds Better

AI Safety proposals used to have the words “existential risk” and “human extinction” tens of times throughout the text. Whenever the word “superintelligence” was used, it was accompanied by typical “x-risk” jargon. Then, AI x-risk organizations learned how to appeal to politicians. In the “Superintelligence Strategy,” for example, Dan Hendrycks was more careful. The words “extinction” and “existential” appeared only once. The new phrase (used tens of times) was “catastrophic dual-use AI.”

The bigger trend is that in order to stay relevant, AI safety groups emphasize the “National Security” angle. What didn’t work with “extinction” talking points will work with “security” talking points. NatSec risks are more grounded in reality, so they can be used to justify previous authoritarian proposals with renewed urgency.

“The AI Doomers’ plans are based on an urgency which is widely assumed but never justified,” pointed out Ben Landau-Taylor. “For many of them, the urgency leads to a rush to do something—anything—even if their strategy is unsound and has historically done exactly the opposite of what they profess to want.”

It is this rush to “do something—anything” that leads to “by all means necessary.”

Further Down the Rabbit Hole: Radicalization

A major problem with the AI doomers’ mindset of “the end justifies the means” is that all possible radical “solutions” are on the table. Why stop with pervasive surveillance of software and hardware if the government can also surveil all known AI researchers?

This is how it plays out: They start with weird ideas like “Patent trolling to save the world,” “don’t build AI or advance computer semiconductors for the next 50 years,” or that “using an open-source model” should lead to “20 years in jail.” But, then, they end up playing with dangerous ideas about “AGI developers” having “a bullet put through their head.” “Walk to the labs across the country and burn them down. Like, literally. That is the proper reaction.”

This radicalization process can only lead to bad outcomes.

If human extinction from AI is just around the corner, based on their logic, their suggestions are “extremely small sacrifices to make.” “People like Eliezer [Yudkowsky] are annoyed when people suggest their rhetoric hints at violence against AI labs and researchers. But even if Eliezer & co don’t advocate violence, it does seem like violence is the logical conclusion of their worldview - so why is it a taboo?” … “If preventing the extinction of the human race doesn’t legitimize violence, what does?”

Another problem is that the doomers base their proposals on philosophical thought experiments and hypothetical made-up futures. The fact that they convinced themselves that their “AI existential risk” belief is the “central truth” and urgent—doesn’t make it so.

Beyond AI Hype & AI Panic: A Call for AI Realism

We need a middle-ground approach: steering the AI discourse away from the extremes.

“The debate surrounding this new technology either wildly overhypes the benefits or hyperbolically catastrophizes the potential dangers” … “There’s a large greyscale between utopian dreams and dystopian nightmares.”

- Yours truly, Generative AI isn’t our gateway to heaven nor a Frankenstein monster, The Daily Beast.

“What is needed is a rejection of both apocalyptic fear and unlimited hope. What is most urgent is a balanced, enterprising optimism—an ambitious and clear vision for what constitutes the good in technology, coupled with the awareness of its dangers.”

- Brendan McCord, Existential pessimism vs. Accelerationism: Why tech needs a rational, humanistic "Third Way", Cosmos Institute.

“People love to spin tales of AI doom or AI utopia, but it’s time to take a realistic look at AI” … “The biggest problem is not whether we need more utopias or dystopias. What we most desperately need is a heavy dose of realism.”

- Daniel Jeffries, The Middle Path for AI, Future History.

“The United States is beginning to see through the wild exaggerations about AI, both utopian and apocalyptic. That’s a welcome development. Science fiction is no foundation for serious policymaking.”

- Brian Chau, Apocalyptic predictions about AI aren’t based in reality, City Journal.

“We must move away from binary tales of catastrophe, not towards naïve utopianism that ignores problems and risks that come with change, but hopeful solutionism that reminds us we can solve and mitigate them.”

- Louis Anslow, Black Mirror’s pessimism porn won’t lead us to a better future, The Guardian.

Endnotes

1

The “AI Existential Risk” ecosystem grew, in part, thanks to Dustin Moskovitz, Jaan Tallinn, Vitalik Buterin, and Sam Bankman-Fried (yes, the convicted felon).

Open Philanthropy donated approximately $780 million.

Jaan Tallinn’s SFF (The Survival and Flourishing Fund) donated approximately $117 million.

In March 2024, we found out that the Future of Life Institute (FLI) reported to the IRS that it had converted a cryptocurrency donation (Shiba Inu tokens from Buterin) to approximately $665 million (using FTX/Alameda Research).

• April 2025 Update:

According to its new IRS Form 990, the Future of Life Institute transferred over $370M to FLI-affiliated organizations. The swift redistribution of assets included:

· December 30, 2022: FLI US transferred $180.3 million to FLI Europe (FLIE)

· December 15, 2022: FLI transferred $162.6 million to the Lightcone Foundation (LF)

· December 11, 2022: FLI transferred $25 million to the Future of Life Foundation (FLF)

In addition, FLI also donated ~$3 million to another related organization: Improve The News Foundation (Verity).

All four organizations share the same governing quartet: Max Tegmark, Meia Chita-Tegmark, Anthony Aguirre, and Jaan Tallinn.

So, 4 FLI-affiliated entities received $370.8 million. Those 4 transfers accounted for nearly the entirety of FLI’s $372.8 million in grant expenses for 2022.

• January 2026 Update:

The Future of Life Foundation: The initial transfer from FLI to FLF in 2022 was $25,025,000. Since then, FLI has given FLF an additional $15,681,663 in 2023 and $18,086,154 in 2024. So, the total comes to $58,792,817.

Lightcone Foundation: The initial transfer from FLI to Lightcone Foundation in 2022 was $162,587,667. Since then, there has been a two-way flow between them: FLI reported sending $6,511,558 to Lightcone (in 2023) and receiving $25,300,000 from Lightcone (in 2023 and 2024).

The “AI Existential Risk” Ideology

ChatGPT’s launch in 2022 sparked a wave of AI doomerism. The leading voices in the “AI will kill us all” camp have confidently predicted that the END of humanity is near. “We believe humans will be wiped out by Godlike AI.” Though “with enough money and effort, we might be able to save humanity from the impending AI apocalypse.” They talked about an “AI takeover” during their media and policymaking takeover. Since then, we have been in an unprecedented AI panic.

The “AI Existential Risk” Ecosystem’s Funding

Their “AI Existential Risk” ideology had the financial backing of over a billion dollars from a few Effective Altruism billionaires, namely Dustin Moskovitz, Jaan Tallinn, Vitalik Buterin, and Sam Bankman-Fried (yes, the convicted felon).

Moskovitz’s Open Philanthropy (rebranded as Coefficient Giving) is by far the largest donor1 to most of the “AI Existential Risk” organizations.

The “AI Existential Risk” Lobbying

Nowadays, the “AI Existential Risk” ecosystem encompasses hundreds of organizations. Many advocate extreme authoritarian measures to stop/pause AI. They include “requiring registration and verifying location of hardware,” “a strict licensing regime, clamp down on open-source models, and impose civil and criminal liability on developers.”

Recently, Berkeley’s “Machine Intelligence Research Institute” (MIRI) proposed to
· restrict machine learning and artificial intelligence research
· prohibit all training runs above 10^24 FLOP (many such models already exist)
· monitor chip clusters above 16 H100s (Yudkowsky & Soares’ “magic number” was 8).
MIRI acknowledges that all its surveillance mechanisms “enable authoritarianism.”

ControlAI even proposed a 20-year pause, because the “default” is human extinction by Godlike AI, and “two decades provide the minimum time frame to construct our defenses.”

Staying informed about this growing “AI existential risk” ecosystem is important. So, the information below aims to familiarize you with the various players involved.

The “AI Existential Risk” Map

The Map of AI Existential Safety was created by effective altruists so that the epistemic community of AI safety students/researchers/lobbyists could become more acquainted with all potential funding, resources, organizations, and projects.

It is a visual overview of the key organizations, programs, and projects operating in the “AI existential risk” ecosystem.

Since there are so many, the accompanying spreadsheet—with its over 300 rows—really comes in handy.

To make sense of this ecosystem’s expansion, the map offers the following categories: Funding; Governance; Conceptual Research; Capabilities Research; Training and Education; Research Support; Career Support; Resources; Media; Public Outreach/Advocacy; and Blogs/Newsletters.

The criteria for inclusion in the map: “An entry must have reducing AI existential risk as a goal; have produced at least one output to that end; must additionally still be active and expected to produce more outputs.”

I first published this map’s details in the “Ultimate Guide to ‘AI Existential Risk’ Ecosystem” on December 5, 2023. Afterward, I updated it in “Panic-as-a-Business is Expanding” on April 15, 2024.

Since then, 50 new entries have been added to this map.

• August 2025 update: Since January, 62 new entries have been added to this map.

• November 2025 update: Since August, 36 new entries have been added to this map.

• December 2025 update: Since November, 15 new entries have been added to this map.

• February 2026 update: Since December 2025, 12 new entries have been added to this map.

Please note:

• Each item includes a link and a description (some have endnotes with additional data). The descriptions are from the map’s website.

• Not all entries represent the ideology summarized above. There are different levels of AI safety work. This map encompasses all types.

The Key Takeaway from the “AI Existential Risk” Map

One conclusion can be drawn from reviewing the hundreds of links: It’s an inflated ecosystem.

There’s a great deal of redundancy:

• The same names/acronyms/logos (with only minor changes)

• The same extreme talking points

• The same group of people, just with different titles

• The same funding source (primarily Coefficient Giving, formerly Open Philanthropy).

This is clearly not a grassroots, bottom-up movement. It is a well-orchestrated top-down movement.

The media and politicians need to open their eyes to this reality.

Funding

Open Philanthropy [rebranded as Coefficient Giving]

The largest funder in the existential risk space.

SFF – Survival and Flourishing Fund2

The second largest funder in AI safety.

LTFF – Long-Term Future Fund

Making grants addressing global catastrophic risks, promoting longtermism, and otherwise increasing the likelihood that future generations will flourish.

GWWC – Giving What We Can

Community of donors who have pledged to donate a significant portion of their income.

FLI – Future of Life Institute – Grantmaking Work

Fellowships include PhD and postdoctoral fellowships in technical AI safety, and a PhD fellowship in US-China AI governance.

Grant programs include “Request for proposals on religious projects tackling the challenges posed by the AGI race,” “call for proposed designs for global institutions governing AI,” and policy advocacy groups like Encode and the Center for Humane Technology.

Longview Philanthropy [formerly Effective Giving UK]

Devises and executes bespoke giving strategies for major donors.

Manifund

Regranting and impact certificates platform for AI safety and other cause areas.

CLR – Center on Long-Term Risk – Fund

Financial support for projects focused on reducing current and future s-risks.

GiveWiki [Formerly Impact Markets]

Crowd-sourced charity evaluator, helping people find or promote the best AI safety projects.

EAIF – Effective Altruism Infrastructure Fund

Aiming to increase the impact of effective altruism projects by increasing their access to talent, capital, and knowledge.

AI2050

Philanthropic initiative supporting researchers working on key opportunities and hard problems that are critical to get right for society to benefit from AI. Proposals by invite only.

SHfHS – Saving Humanity from Homo Sapiens

Small organization with a long history of funding existential risk reduction.

AI Safety Advocacy Grants

Grant program run by Nonlinear, providing funding to those raising awareness about AI risks or advocating for a pause in AI development.

Nonlinear Network

Funder network for AI existential risk reduction. Applications are shared with donors, who then reach out if they are interested.

GAIA – Grantmakers for AI Alignment

Joinable donor circle for people earning to give or allocating funds towards reducing AI existential risk.

Macroscopic Ventures [formerly Polaris Ventures and the Center for Emerging Risk Research]

Swiss nonprofit making grants and investments focused on reducing long-term suffering.

CAIF – Cooperative AI Foundation

Charity foundation backed by a large philanthropic commitment supporting research into improving cooperative intelligence of advanced AI.

The Navigation Fund

Offers grants to high-impact organizations and projects that are taking bold action and making significant changes.

Meta Charity Funders

Network of donors funding charitable projects that work one level removed from direct impact.

Lionheart Ventures

VC firm investing in ethical founders developing transformative technologies that have the potential to impact humanity on a meaningful scale.

ARM – AI Risk Mitigation Fund

Aiming to reduce catastrophic risks from advanced AI through grants towards technical research, policy, and training programs for new researchers.

AE Grants

Empowering innovators and scientists to increase human agency by creating the next generation of responsible AI. Providing support, resources, and open-source software.

Ergo Impact

Helping major donors find, fund, and scale the most promising solutions to the world’s most pressing problems.

Foresight Institute: Funding

Funding projects in 1) automating research and forecasting, 2) security technologies, 3) neurotech, and 4) safe multipolar human AI scenarios.

Mythos Ventures

Aiming to empower founders building a radically better world with safe AI systems by investing in ambitious teams with defensible strategies that can scale to post-AGI.

FLF – Future of Life Foundation3

Accelerator aiming to steer transformative technology towards benefiting life and away from extreme large-scale risks.

AI Safety Science

Funder housing the science-focused philanthropic efforts of Eric and Wendy Schmidt, which moves large amounts of funding towards AI safety via its AI institute.

ARIA – Advanced Research + Invention Agency

UK government R&D funding agency built to unlock scientific and technological breakthroughs that benefit everyone.

AISafety.com: Funding

Comprehensive and up-to-date directory of sources of financial support for AI safety projects, ranging from grant programs to venture capitalists.

AISafety.com: Donation Guide

Regularly-updated guide on how to donate most effectively with the funding and time you have available.

Astralis Foundation

New AI safety funding initiative with $25M annual giving, aiming to unite funders, domain experts, and entrepreneurs to launch and scale ambitious initiatives.

5050

13-week program run by Fifty Years helping scientists and engineers build startups tackling world-scale problems.

Juniper Ventures

VC firm investing in startups helping make AI safe, secure, and beneficial for humanity. Focused on working with exceptional founders at the earliest stages.

AISTOF – AI Safety Tactical Opportunities Fund

Pooled multi-donor fund structured to be fast and rapidly capture emerging opportunities, including in governance, technical alignment, and evaluations. Managed by JueYan Zhang.

Governance

FLI – Future of Life Institute4

Outreach, policy advocacy, grantmaking, and event organization for existential risk reduction.

GovAI – Centre for the Governance of AI

AI governance research group at Oxford, producing research tailored towards decision-makers and running career development programs.

CLR – Center on Long-Term Risk [formerly the EA Foundation]

Research, grants, and community-building around AI safety, focused on conflict scenarios as well as technical and philosophical aspects of cooperation.

CSET – Center for Security and Emerging Technology5

Georgetown University think tank providing decision-makers with data-driven analysis on the security implications of emerging technologies.

AIPI – AI Policy Institute6

Channeling public concern into effective regulation by engaging with policymakers, media, and the public to ensure AI is developed responsibly and transparently.

CLTR – Center for Long-Term Resilience

Think tank aiming to transform global resilience to extreme risks by improving relevant governance, processes, and decision-making.

CFG – Centre for Future Generations7 [Formerly, ICFG - International Center for Future Generations]

Brussels think tank focused on helping governments anticipate and responsibly govern the societal impacts of rapid technological change.

Manifold Markets

Play-money prediction markets on many topics, including AI safety.

TFI – Transformative Futures Institute

Exploring the use of underutilized foresight methods and tools in order to better anticipate societal-scale risks from AI.

Metaculus

Forecasting platform for many topics, including AI.

FRI – Forecasting Research Institute

Advancing the science of forecasting for the public good by working with policymakers and nonprofits to design practical forecasting tools, and test them in large experiments.

QURI – Quantified Uncertainty Research Institute

Advancing forecasting and epistemics to improve the long-term future of humanity. Writing research and software.

AI Impacts8

Answering decision-relevant questions about the future of AI, including through research, a wiki, and expert surveys. Run by MIRI.

Epoch AI

Research institute investigating key trends and questions that will shape the trajectory and governance of AI.

Rethink Priorities

Research and implementation group identifying pressing opportunities to make the world better.

Convergence Analysis

Building a foundational series of sociotechnical reports on key AI scenarios and governance recommendations, and conducting AI awareness efforts to inform the general public.

TFS – The Future Society9

Defining, designing, and deploying projects that address institutional barriers in AI governance.

GCRI – Global Catastrophic Risk Institute

Small think tank developing solutions for reducing existential risk by leveraging both scholarship and the demands of real-world decision-making.

CSER – Centre for the Study of Existential Risk

Interdisciplinary research centre at the University of Cambridge doing diverse existential risk research.

CFI – Centre for the Future of Intelligence

Interdisciplinary research centre at the University of Cambridge exploring the nature, ethics, and impact of AI.

GPI – Global Priorities Institute

University of Oxford research center conducting foundational research to inform the decision-making of individuals and institutions seeking to do as much good as possible.

Median Group

Research nonprofit working on models of past and future progress in AI, intelligence enhancement, and sociology related to existential risks.

CAIP – Center for AI Policy10

Nonpartisan research organization developing policy and conducting advocacy to mitigate catastrophic risks from AI.

Future Matters

Provides strategy consulting services to clients trying to advance AI safety through policy, politics, coalitions, or social movements.

PAI – Partnership on AI

Convening academic, civil society, industry, and media organizations to create solutions so that AI advances positive outcomes for people and society.

IAPS – Institute for AI Policy and Strategy

Research and field-building organization focusing on policy and standards, compute governance, and international governance and China.

LawAI – Institute for Law & AI [formerly, LPP – Legal Priorities Project]

Think tank researching and advising on the legal challenges posed by AI.

AIGS Canada – AI Governance & Safety Canada

Nonpartisan not-for-profit and community of people across Canada, working to ensure that advanced AI is safe and beneficial for all.

CARMA – Center for AI Risk Management & Alignment11

Conducting interdisciplinary research supporting global AI risk management. Also produces policy and technical research.

CLAI – Center for Long-term AI

Interdisciplinary research organization based in China exploring contemporary and long-term impacts of AI on society and ecology.

ASIA – AI Safety Asia

Platform for connecting junior researchers and seasoned civil servants from Southeast Asia with senior AI safety researchers from developed countries.

CeSIA – Center for AI Security

French AI safety nonprofit dedicated to education, research, and advocacy.

Vista Institute

Promoting informed policymaking to navigate emerging challenges from AI through research, knowledge-sharing, and skill building.

Narrow Path12

A series of proposals developed by ControlAI intended for action by policymakers in order for humanity to survive artificial superintelligence.

AIGSI – AI Governance and Safety Institute

Aiming to improve institutional response to existential risk from future AI systems by conducting research and outreach, and developing educational materials.

IASEAI – International Association for Safe and Ethical Artificial Intelligence

Nonprofit aiming to ensure that AI systems are guaranteed to operate safely and ethically; and to shape policy, promote research, and build understanding and community around this goal.

Beijing AISI – Beijing Institute of AI Safety and Governance

R&D institution dedicated to developing AI safety and governance frameworks to provide a safe foundation for AI innovation and applications.

ControlAI

Nonprofit fighting to keep humanity in control of AI by developing policy and conducting public outreach.

Simon Institute

Geneva-based think tank working to foster international cooperation in mitigating catastrophic risks from AI.

Forethought Research

Small research nonprofit focused on how to navigate the transition to a world with superintelligent AI systems.

Effective Institutions Project

Advisory and research organization focused on improving the way institutions make decisions on critical global challenges.

CAES – Collective Action for Existential Safety

Nonprofit aiming to catalyze collective effort toward reducing existential risk, including through an extensive action list for individuals, organizations, and nations.

IAIGA – International AI Governance Alliance

Nonprofit dedicated to establishing an independent global organization capable of effectively mitigating extinction risks from AI and fairly distributing its economic benefits to all. IAIGA is an initiative of the Center for Existential Safety (CAES).

AI Futures Project

Small research group forecasting the future of AI. Created “AI 2027,” a detailed forecast scenario projecting the development of artificial superintelligence.

MIT FutureTech

Interdisciplinary group aiming to identify and understand trends in computing that create opportunities for (or pose risks to) our ability to sustain economic growth.

ARI – Americans for Responsible Innovation

Bipartisan nonprofit seeking to address a broad range of policy issues raised by AI, including current harms, national security concerns, and emerging risks.

CnAISDA – China AI Safety & Development Association

China’s self-described counterpart to the AI safety institutes of other countries. Its primary function is to represent China in international AI conversations.

Good Ancestors

Australian lobbying organization focused on AI safety policy and other AI-related issues, including cybersecurity and biosecurity. Also runs Australians for AI Safety.

RAND13

Research nonprofit developing solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier, and more prosperous.

Secure AI Project

Nonprofit developing and advocating for AI safety principles to be put into practice in US state and federal legislatures.

AIPN – AI Policy Network

AIPN advocates for targeted policies that address the unique challenges posed by rapidly advancing AI systems. AIPN engages with members of Congress, congressional staff, federal officials, and media to promote informed, proactive policymaking that prepares America for transformative AI capabilities.

Buddhism & AI Initiative

A collaborative effort to bring together Buddhist communities, technologists, and contemplative researchers worldwide to help shape the future of artificial intelligence.

AI Standards Lab

Politically and geographically neutral nonprofit converting insights from the existing literature into ready-made text for AI safety standards.

AIWI – The AI Whistleblower Initiative

Helps AI insiders raise concerns about potential risks and misbehavior in AI development by providing whistleblowing services, expert guidance, and secure communication tools.

Oxford Martin AIGI – AI Governance Initiative

Research center housed in the Martin School of the University of Oxford researching AI governance from both technical and policy perspectives.

GPAI – General Purpose AI Policy Lab

Producing research that helps French institutional actors address the security and international coordination requirements posed by the development of general-purpose AI.

CLAIR – Center for Law & AI Risk

Supporting research at the intersection of law and AI safety, aiming to build a field of legal scholars working to understand how law can reduce catastrophic risks from advanced AI.

Odyssean Institute

Pushing academic and policy discourse forward through research and engagement on horizon scanning, decision making under deep uncertainty, and citizens' assemblies.

Conceptual Research

MIRI – Machine Intelligence Research Institute

The original AI safety technical research organization, co-founded by Eliezer Yudkowsky. Now focusing on policy and public outreach.

ARC – Alignment Research Center

Research organization doing theoretical research focusing on the Eliciting Latent Knowledge (ELK).

Cyborgism

A strategy for accelerating alignment research by using human-in-the-loop systems which empower human agency rather than outsource it.

Orthogonal

Formal alignment organization led by Tamsin Leake, focused on agent foundations.

ALTER – Association for Long-Term Existence and Resilience

Israeli research and advocacy nonprofit working to investigate, demonstrate, and foster useful ways to safeguard and improve the future of humanity.

John Wentworth

Independent researcher working on selection theorems, abstraction, and agency.

CHAI – Center for Human-Compatible AI

Developing the conceptual and technical wherewithal to reorient the general thrust of AI research towards provably beneficial systems. Led by Stuart Russell at UC Berkeley.

Team Shard

Independent researchers trying to find reward functions which reliably instill certain values in agents.

ACS – Alignment of Complex Systems

Studying questions about multi-agent systems composed of humans and advanced AI. Based at Charles University, Prague.

Dylan Hadfield-Menell

Assistant professor at MIT working on agent alignment.

Roman Yampolskiy14

Author of two books on AI safety, and Professor at the University of Louisville with a background in cybersecurity.

MIT Algorithmic Alignment Group

Working towards better conceptual understanding, algorithmic techniques, and policies to make AI safer and more socially beneficial.

AE Studio

Large team taking a 'Neglected Approaches' approach to alignment, tackling the problem from multiple, often overlooked angles in both technical and policy domains.

FAIR – Frontier AI Research

Argentine nonprofit conducting both theoretical and empirical research to advance frontier AI safety as a sociotechnical challenge.

Obelisk [part of Astera]

Team of researchers pursuing an exploratory, neuroscience-informed approach to engineering AGI.

Formation Research

Nonprofit aiming to reduce lock-in risks by researching fundamental lock-in dynamics and power concentration.

Softmax

Emmett Shear’s research organization dedicated to developing a theory of “organic alignment” to foster adaptive, non-hierarchical cooperation between humans and digital agents.

CORAL – Computational Rational Agents Laboratory

Research group studying agent foundations to give us the mathematical tools to align the objectives of an AI with human values.

Steve Byrnes’s Brain-Like AGI Safety

Brain-inspired framework using insights from neuroscience and model-based reinforcement learning to guide the design of aligned AGI systems.

Aether

Independent LLM agent safety research group launched in 2024, focused on chain-of-thought monitorability.

Atlas Computing

R&D nonprofit prototyping AI-powered tools to generate formal specifications. Aiming to empower engineers to verify code easily and develop software that’s built for trust.

Coordinal Research

Developing tools that accelerate the rate human researchers can make progress on alignment, and building automated research systems that can assist in alignment work today.

Goodfire

Mechanistic interpretability research lab aiming to decode neural networks in order to make AI systems more understandable, editable, and safer.

LawZero15

Canadian nonprofit advancing research and developing technical solutions for safe-by-design AI systems based on Scientist AI, a research direction led by Yoshua Bengio.

Luthien

Developing AI Control in order to increase the probability that effective AI Control systems will be deployed to mitigate catastrophic risks from frontier AI systems when they are developed.

Simplex

Small team of researchers and engineers aiming to bring the best of physics and computational neuroscience together in order to understand and control AGI.

TruthfulAI

Nonprofit led by Owain Evans, researching situational awareness, deception, and hidden reasoning in language models.

CLTC – Center for Long-Term Cybersecurity

UC Berkeley research center bridging academic research and practical policy needs in order to anticipate and address emerging cybersecurity challenges.

Dovetail

Group working on foundational mathematics research that gives us an understanding of the nature of AI agents.

Equilibria Network

Using category theory and agent-based simulations to predict where multi-agent AI systems break down and identify control levers to prevent collective AI behavior from being harmful.

Iliad

Applied mathematics research nonprofit dedicated to advancing foundational alignment research.

Capabilities Research

Google DeepMind

AI capabilities lab with a strong safety team.

OpenAI

San Francisco-based capabilities lab led by Sam Altman. Created ChatGPT.

Anthropic

Research lab focusing on LLM alignment, particularly interpretability. Featuring Chris Olah, Jack Clark, and Dario Amodei.

CAIS – Center for AI Safety16

San Francisco-based nonprofit conducting safety research, building the field of AI safety researchers, and advocating for safety standards.

Conjecture17

Alignment startup born out of EleutherAI, building LLMs and Cognitive Emulation systems.

Redwood Research18

Nonprofit researching interpretability and alignment.

EleutherAI

Open-source research lab focused on interpretability and alignment. Operates primarily through a public Discord server, where research is discussed and projects are coordinated.

Ought

Product-driven research lab developing mechanisms for delegating high-quality reasoning to ML systems. Built Elicit, an AI assistant for researchers and academics.

Aligned AI

An Oxford-based startup working on safe off-distribution generalization, featuring Stuart Armstrong.

FAR AI19 [Fund for Alignment Research]

Ensuring AI systems are trustworthy and beneficial to society by incubating and accelerating research agendas too resource-intensive for academia but not yet ready for commercialization.

Apollo Research

Aiming to detect deception by designing AI model evaluations and conducting interpretability research to better understand frontier models. Also provides policymakers with technical guidance.

Timaeus

Nonprofit researching applications of singular learning theory to AI safety.

Cavendish Labs

AI safety research community based in a small town in Vermont, USA.

Modeling Cooperation

Conducting long-term future research on improving cooperation in competition for the development of transformative AI.

AOI – AI Objectives Institute

Nonprofit research lab building AI tools to defend and enhance human agency – by researching and experimenting with novel AI capabilities.

METR – Model Evaluation & Threat Research20

Evaluating whether cutting-edge AI systems could pose catastrophic risks to society.

ARG – NYU Alignment Research Group

Group of researchers at New York University doing empirical work with language models aiming to address longer-term concerns about the impacts of deploying highly-capable AI systems.

CBL – University of Cambridge Computational and Biological Learning Lab

Research group using engineering approaches to understand the brain and to develop artificial learning systems.

MAI – Meaning Alignment Institute

Research organization applying expertise in meaning and human values to AI alignment and post-AGI futures.

SaferAI

French nonprofit working to incentivize responsible AI practices through policy recommendations, research, and risk assessment tools.

KASL – Krueger AI Safety Lab

AI safety research group at the University of Cambridge led by David Krueger.

SSI – Safe Superintelligence Inc.

Research lab founded by Ilya Sutskever comprised of a small team of engineers and researchers working towards building a safe superintelligence.

Gray Swan

For-profit company developing tools that automatically assess the risks of AI models and developing its own AI models aiming to provide best-in-class safety and security.

Transluce

Nonprofit research lab building open source, scalable, AI-driven tools to understand and analyze AI systems and steer them in the public interest.

Workshop Labs

New public benefit corporation working on mitigating gradual disempowerment and the intelligence curse by creating personalized models.

Geodesic

New research nonprofit in Cambridge focused on leading projects with the shortest path to impact for AI safety. Initial work is on chain-of-thought health and monitoring.

Palisade Research

Investigating cyber offensive AI capabilities and the controllability of frontier AI models in order to “better understand the risk of losing control to AI systems forever” and to advise policymakers and the public on AI risks.

Cadenza Labs

Researching, benchmarking, and developing lie detectors for LLMs, including using cluster normalization techniques.

Harmony Intelligence

For-profit creating defensive cybersecurity products that mitigate AI x cyber risk. Also supports the Australian government with AI safety evals and policymaking.

CaML – Compassion in Machine Learning

Working on research to make transformative AI more compassionate towards all sentient beings and open-minded towards non-humans (animals and digital minds) through new data generation methods.

Beneficial AI Foundation

Founded by Max Tegmark, supports a range of technical AI safety research – including mechanistic interpretability, red-teaming, and guaranteed safe AI.

One of its projects is QAISI (Quantitative AI Safety Initiative).

Contramont Research

Working on LM backdoors, real-world evals, and scalable oversight. Published a paper demonstrating cryptographic backdoors that evade detection even with full white-box access.

Training and Education

AISF – AI Safety Fundamentals

Runs the standard introductory courses, each three months long and split into two tracks: Alignment and Governance. Also runs shorter intro courses.

MATS – ML Alignment & Theory Scholars program21

Research program connecting talented scholars with top mentors in AI safety. Involves 10 weeks onsite mentored research in Berkeley, and, if selected, 4 months extended research.

AI Safety Camp

3-month online research program with mentorship.

ERA – Existential Risk Alliance – Fellowship [spin off of CERI – Cambridge Existential Risk Initiative]

In-person paid 8-week summer AI safety research fellowship at the University of Cambridge.

PIBBSS – Principles of Intelligence – Fellowship

3-month interdisciplinary program connecting researchers from diverse fields with AI safety mentors, in order to help them transition their careers to AI safety.

SAIL – Safe AI London

Runs AI safety events and training programs in London.

Intro to MLS – ML Safety

Comprehensive online introductory course on ML safety. Run by CAIS.

GCP – Global Challenges Project

Intensive 3-day workshops for students to explore the foundational arguments around risks from advanced AI (and biosecurity).

HAIST – Harvard AI Safety Student Team

Group of Harvard students conducting AI safety research and running fellowships, workshops, and reading groups.

MAIA – MIT AI Alignment

Student group conducting AI safety research and running workshops and reading groups.

CBAI – Cambridge Boston Alignment Initiative

Boston organization for helping students get into AI safety via upskilling programs and fellowships. Supports HAIST and MAIA.

CLR – Center on Long-Term Risk – Summer Research Fellowship

2–3-month summer research fellowship in London working on reducing long-term future suffering.

HA – Human-aligned AI – Summer School22

4-day program for teaching alignment research methodology.

CHAI – Center for Human-Compatible AI – Internship

Research internship at UC Berkeley for people interested in research in human-compatible AI.

ARENA – Alignment Research Engineer Accelerator

4–5 week ML engineering upskilling program in London, focusing on alignment. Aims to provide individuals with the skills, community, and confidence to contribute directly to technical AI safety.

SERI – Stanford Existential Risk Initiative – Fellowship

10-week funded summer research fellowship for undergrad and grad students (primarily at Stanford).

AISI – AI Safety Initiative at Georgia Tech

Georgia Tech community hosting research projects and a fellowship.

SAIA – Stanford AI Alignment

Student group and research community under SERI. Accelerating students into AI safety careers in AI safety, building the community at Stanford, and conducting research.

EffiSciences

French collective promoting mission-driven research to tackle global issues. Organizes conferences, hackathons, ML4Good bootcamps, and university reading groups and research projects.

WAISI – Wisconsin AI Safety Initiative

Wisconsin student group dedicated to reducing AI risk through alignment and governance.

AISG – AI Safety Initiative Groningen

Student group in Groningen, Netherlands.

ML4Good

10-day intensive, in-person bootcamps upskilling participants in technical AI safety research.

SPAR – Supervised Program for Alignment Research

Virtual, part-time research program offering early-career individuals and professionals the chance to engage in AI safety research for 3 months.

Monthly hackathons around the world for people getting into AI safety.

Apart Sprints

Monthly hackathons around the world for people getting into AI safety.

AI Safety Hungary

Supports students and professionals in contributing to the safe development of AI.

Pivotal Research Fellowship

Annual 9-week program designed to enable promising researchers to produce impactful research and accelerate their careers in AI safety (or biosecurity).

WhiteBox Research

Filipino nonprofit aiming to develop more AI interpretability and safety researchers, particularly in Southeast Asia.

Foresight Fellowship

1-year program catalyzing collaboration among young scientists, engineers, and innovators working to advance technologies for the benefit of life.

CAISH – Cambridge AI Safety Hub

Network of students and professionals in Cambridge conducting research, running educational and research programs, and creating a vibrant community of people with shared interests.

LASR – London AI Safety Research – Labs

12-week research program aiming to assist individuals in transitioning to full-time careers in AI safety.

XLab – UChicago Existential Risk Laboratory – Fellowship

10-week summer research fellowship giving undergraduate and graduate students the opportunity to produce high impact research on various emerging threats, including AI.

Impact Academy: Global AI Safety Fellowship

Fully-funded research program connecting exceptional STEM researchers with full-time placement opportunities at AI safety labs and organizations.

Training for Good - Talos Fellowship

7-month program enabling ambitious graduates to launch EU policy careers reducing risks from AI. Fellows participate in one of two tasks: training or placement.

Training for Good – Tarbell Center for AI Journalism23

Nonprofit supporting journalism that helps society navigate the emergence of increasingly advanced AI. Runs fellowships, grants, and residencies. Fellowship: A one-year program for early-career journalists interested in covering artificial intelligence. Fellows secure a 9-month placement at a major newsroom.

MARS – Mentorship for Alignment Research Students

Research program run by CAISH, connecting aspiring researchers with experienced mentors to conduct AI safety (technical or policy) research for 2–3 months.

Arcadia Impact

Runs various projects aimed at education, skill development, and creating pathways into impactful careers.

FIG – Future Impact Group – Fellowship

Remote, part-time research opportunities in AI safety, policy, and philosophy. Also provide ongoing support, including coworking sessions, issue troubleshooting, and career guidance.

IRG – Impact Research Groups

8-week mentored research program for London students, where teams explore a research question tackling global challenges – including AI governance and technical AI safety.

Intelligence Rising

Workshop letting decision-makers experience the tensions and risks that can emerge in the highly competitive environment of AI development through an educational roleplay game.

Non-Trivial

Aiming to empower bright high schoolers to start solving the world's most pressing problems through various research programs.

Orion AI Governance Initiative

London-based talent development scheme designed to equip outstanding students with the knowledge and skills to shape the future of AI governance.

Pathfinder Fellowship

Fellowship run by Kairos for students organizing technical AI safety or AI policy university groups – providing mentorship, funding, and other resources.

AISafety.com: Self-study

Comprehensive, up-to-date directory of AI safety curricula and reading lists for self-led learning at all levels.

Athena Mentorship Program for Women

10-week remote mentorship program for women looking to strengthen their research skills and network in technical Al alignment research.

Cooperative AI Summer School

Annual program providing students and early-career professionals in AI, computer science, and related disciplines with a firm grounding in the emerging field of cooperative AI.

GovAI Fellowship

3-month program from the Centre for the Governance of AI designed to help researchers transition to working on AI governance full-time.

IAPS AI Policy Fellowship

Fully-funded program from the Institute for AI Policy and Strategy for professionals seeking to strengthen practical policy skills for managing the challenges of advanced AI.

ILINA Program

African-led research program dedicated to building talent, generating research, and shaping policy to advance AI safety.

TARA – Technical Alignment Research Accelerator

14-week part-time program building technical AI safety research skills through the ARENA curriculum. Consists of weekly in-person learning sessions followed by a 3-week project.

Research Support

Lightcone Infrastructure

Maintains LessWrong, the Alignment Forum, and Lighthaven (an event space in Berkeley).

Nonlinear Fund

“Means-neutral” AI safety organization, doing miscellaneous stuff, including offering bounties on small-to-large AI safety projects and running a funder network.

CEEALAR – Centre for Enabling EA Learning & Research24 [formerly “EA Hotel”]

Free or subsidized accommodation and catering in Blackpool, UK, for people working on/transitioning to working on global catastrophic risks.

AED – Alignment Ecosystem Development

Building and maintaining key online resources for the AI safety community. Volunteers welcome.

Apart Research

Non-profit AI safety research lab hosting open-to-all research sprints, publishing papers, and incubating talented researchers to make AI safe and beneficial for humanity.

Arb Research

Consultancy for forecasting, machine learning, and policy. Doing original research, evidence reviews, and large-scale data pipelines.

BERI – Berkeley Existential Risk Initiative

Providing free operations support for university research groups working on existential risk.

SERI – Stanford Existential Risks Initiative

Collaboration between faculty and students. Runs research fellowships, an annual conference, speaker events, discussion groups, and a frosh-year class.

Impact Ops

Providing consultancy and hands-on support to help high-impact organizations upgrade their operations.

GPAI – Global Partnership on AI

International initiative with 44 member countries working to implement human-centric, safe, secure, and trustworthy AI embodied in the principles of the OECD Recommendation on AI.

ENAIS – European Network for AI Safety

Connecting researchers and policymakers for safe AI in Europe.

Constellation

Berkeley research center growing and supporting the AI safety ecosystem.

LISA – London Initiative for Safe AI

Coworking space hosting organizations (including BlueDot Impact, Apollo, Leap Labs), acceleration programs (including MATS, ARENA), and independent researchers.

Ada Lovelace Institute

Working to ensure the benefits of data and AI is justly and equitably distributed, and enhances individual and social wellbeing.

Catalyze Impact

Incubating early-stage AI safety research organizations. The program involves co-founder matching, mentorship, and seed funding, culminating in an in-person building phase.

SAIF – Safe AI Forum

Fostering responsible governance of AI to reduce catastrophic risks through shared understanding and collaboration among key global actors.

EquiStamp

Providing a platform that allows companies and individuals to evaluate the capabilities of AI models and therefore know how much they can trust them.

Ashgro

Providing fiscal sponsorship to AI safety projects.

Third Opinion

Helping concerned individuals working at the frontier of AI get expert opinions on their questions, anonymously and securely.

Rethink Wellbeing

Nonprofit running programs to give altruists the tools they need to improve their mental health, so they can have a greater impact on the world.

Trajectory Labs

AI safety coworking and events space in downtown Toronto, Canada, providing a physical space, hosting informative events, and maintaining a collaborative community.

Meridian

Coworking space and field-building nonprofit based in Cambridge, UK. Houses projects like CAISH and the ERA:AI Fellowship, as well as visiting AI safety researchers.

FFF – Flourishing Future Foundation

Provides engineering teams, compute, and infrastructure to researchers advancing neglected approaches to alignment.

Good Impressions

Marketing firm helping socially impactful projects grow, including through consulting and running full-scale campaigns.

Mox

Incubator and coworking space in San Francisco aimed at various groups, including those working on AI safety.

RAISEimpact

Program designed to help AI safety organizations strengthen their management and leadership practices in order to amplify their effectiveness.

Pause House

Free accommodation and board in Blackpool, England, for individuals doing work related to pushing for a pause to AI development. Located next to CEEALAR.

PEAKS

Coworking space in Zurich, Switzerland, for people working on AI safety or effective altruism. Regularly hosts events – ranging from relaxed gatherings to lightning talks and discussions.

SASH – Singapore AI Safety Hub

Coworking, events, and community space for people working on AI safety. Runs regular talks, hackathons, and networking events.

Career Support

AIS (AI Safety) Quest

Helps people navigate the AI safety space with a welcoming human touch, offering personalized guidance and fostering collaborative study and project groups.

AISS – AI Safety Support

Field-building organization with an extensive resources list.

80,000 Hours – Career Guide

Article with motivation and advice for pursuing a career in AI safety.

Successif

Helping professionals transition to high-impact work by performing market research on impactful jobs and providing career mentoring, opportunity matching, and professional training.

80,000 Hours – Job Board

Curated list of job postings tackling pressing problems, including AI safety.

Effective Thesis – Academic Opportunities

Lists thesis topic ideas in AI safety and coaches people working on them.

Nonlinear – Coaching for AI Safety Entrepreneurs

Free coaching for people running an AI safety startup or considering starting an AI safety org (technical, governance, meta, for-profit, or non-profit).

HIP – High Impact Professionals

Supporting working professionals to maximize their positive impact through their talent directory and Impact Accelerator Program.

Upgradable

Applied research lab helping existential safety advocates to systematically optimize their lives and work.

AISafety.com: Advisors

Directory of advisors offering free guidance calls to help you discover how best to contribute to AI safety, tailored to your skills and interests.

Probably Good

Helps those who want to have a meaningful impact with their careers brainstorm career paths, evaluate options, and plan next steps.

Halcyon Futures

Identifies leaders from business, policy, and academia, and helps them take on new ambitious projects in AI safety.

Heron

Working to bridge the gap between frontier AI models and the level of cybersecurity they need by connecting professionals to high-leverage opportunities in AI security.

Resources

How to pursue a career in technical AI alignment

A guide for people who are considering direct work on technical AI alignment.

Stampy – AI Safety Chatbot

Interactive FAQ; Single-Point-Of-Access into AI safety. Part of AISafety.info.

AI Safety Training

Comprehensive database of available training programs and events.

AI Safety Ideas

Repository of possible research projects and testable hypotheses. Run by Apart Research.

AI Watch

Database of AI safety research agendas, people, organizations, and products.

AI Plans

Ranked and scored contributable compendium of alignment plans and their problems.

AI Risk Discussions

Interactive walkthrough of core AI x-risk arguments and transcripts of conversations with AI researchers. Project of Arkose (2022).

OpenBook

Database of grants in effective altruism.

Donations List

Website tracking donations to AI safety. Full site planned for launch in July 2026.

AI Digest

Interactive explainers on AI capabilities and their effects. (By Sage Future).

AI Safety Map

Helps you learn and memorize the main organizations, projects, and programs currently operating in the AI safety space.

AI Governance Map

Cartoon map showing various organizations, projects, and policies in the AI governance space.

AISafety.info

A guide to AI safety for those new to the space, in the form of a comprehensive FAQ and chatbot (Stampy).

AISafety.com

The hub for key resources for the AI safety community, including directories of courses, jobs, upcoming events, and training programs, etc.

AI Lab Watch

Collects actions for frontier Al labs to avert extreme risks from AI, then evaluates particular labs accordingly.

The Compendium

Living document aiming to present a coherent worldview explaining the race to AGI and extinction risks and what to do about them – in a way that is accessible to non-technical readers.

EA Domains25

Helping ensure domains in cause areas like AI safety are pointed towards high-impact projects.

AISafety.com – Communities

Directory of local and online AI safety communities.

AISafety.com – Events & Training

Comprehensive database of available training programs and AI safety events.

AI Timeline

Visual overview of the major events in AI over the last decade, from cultural trends to technical advancements.

AI Risk: Why Care?

AI safety chatbot that excels at addressing hard questions and counterarguments about existential risk. A project by AIGSI – AI Governance and Safety Institute and AISGF – AI Safety and Governance Fund.

MIT AI Risk Repository

Comprehensive living database of over 1600 AI risks, categorized by their cause and risk domain.

AI Alignment Slack

The biggest real-time online community of people interested in AI safety, with channels ranging from general topics to specific fields to local groups.

AIRE – AI Risk Explorer

Online platform monitoring the emergence of large-scale AI risks, featuring curated information on evaluations, incidents, and policies.

AI Safety for Fleshy Humans

Accessible, comic-illustrated series describing the history of AI capabilities, the alignment problem, and potential solutions.

RiesgosIA

Spanish-language website featuring AI safety educational tools, including a daily‑updated monitor of AI safety research papers.

Media

AI Explained

YouTuber covering the latest developments in AI.

80,000 Hours Podcast

Interviews on pursuing a career tackling pressing problems, including AI safety.

The Inside View

Interviews with AI safety researchers, explainers, fictional stories of concrete threat models, and paper walk-throughs.

Rational Animations26

Animated videos on effective altruism, rationality, the future of humanity, and AI safety.

AI Safety Videos

Comprehensive index of AI safety video content.

ML Safety Newsletter

Occasional newsletter from CAIS, focused on applied AI safety and ML.

Import AI

Weekly developments in AI (incl. governance) written by Jack Clark, co-founder of Anthropic.

ERO – Existential Risk Observatory

Reducing existential risks by informing the public debate.

AXRP – AI X-risk Research Podcast

Interviews with technical AI safety researchers about their research.

FLI – Future of Life Institute – Podcast

Interviews with existential risk researchers.

Rob Miles

AI safety explainers in video form.

AI Safety Newsletter

Newsletter published every few weeks discussing developments in AI and AI safety. No technical background required. Run by CAIS.

Siliconversations

YouTube channel explaining (mostly) AI safety concepts through entertaining stickman videos.

ARN – The AI Risk Network [For Humanity podcast]

YouTube channel run by John Sherman, a Peabody and Emmy Award-winning former investigative journalist, aiming to make AI risk a kitchen table conversation.

The Cognitive Revolution

Biweekly podcast where host Nathan Labenz interviews AI innovators and thinkers, diving into the transformative impact AI will likely have in the near future.

Dwarkesh Podcast

Well-researched interviews with influential intellectuals going in-depth on AI, technology, and their broader societal implications. Hosted by Dwarkesh Patel.

The AI Policy Podcast

Podcast from the Center for Strategic & International Studies (CSIS) discussing AI regulation, innovation, national security, and geopolitics.

Doom Debates

Channel hosted by Liron Shapira featuring in-depth debates, explainers, and live Q&A sessions, focused on AI existential risk and other implications of superintelligence.

Dr Waku

Computer science PhD and AI research scientist talking about how AI will likely affect all of us and society as a whole, plus interviews with experts.

AI In Context

Channel produced by 80,000 Hours presenting thoroughly researched, cinematic stories about what’s happening in AI and where the trends are taking us.

AISafety.com: Media channels

The AI safety space is changing rapidly. This directory of key information sources can help you keep up to date with the latest developments.

Lethal Intelligence

YouTube channel (by Michael from the AI Risk Network’s “Warning Shots”) raising awareness about the lethal dangers of AGI through explainer videos and podcasts.

Species | Documenting AGI27

Channel run by Drew Spartz educating a general audience about AI risk through high-effort mini-documentaries.

Public Outreach/Advocacy

PauseAI28

Campaign group aiming to convince governments to pause AI development – through public outreach, engaging with decision-makers and organizing protests.

GAIM – Global AI Moratorium

Calling on policymakers to implement a global moratorium on large AI training runs until alignment is solved.

TMP – The Midas Project

Watchdog nonprofit monitoring tech companies, countering corporate propaganda, raising awareness about corner-cutting, and advocating for the responsible development of AI.

AISAF – AI Safety Awareness Foundation

Volunteer organization dedicated to raising awareness about modern AI, highlighting its benefits and risks, and letting the public know how they can help – mainly through workshops.

AISAP – AI Safety Awareness Project

Volunteer organization dedicated to raising awareness about modern AI, highlighting its benefits and risks, and letting the public know how they can help – mainly through workshops.

The Alliance for Secure AI

New communications nonprofit focused on changing the narrative air that policymakers in DC breathe with respect to AI safety.

Stop AI29

Non-violent civil resistance organization working to permanently ban the development of smarter-than-human AI to prevent human extinction, mass job loss, and other problems.

Legal Safety Lab

Uses legal advocacy in Europe to address risks from frontier technologies, working to foster responsible development and implementation practices.

LASST – Legal Advocates for Safe Science and Technology

Researches ways to use legal advocacy to make science and technology safer, informs legal professionals about how to help, and advocates in the courts and policy-setting institutions.

Explainable

Empowering organizations, researchers, and creators to translate the future of AI into stories the world understands.

AISF – AI Safety Foundation

Conducting educational initiatives, research support, and public awareness campaigns. Project of Geoffrey Hinton. Includes “The Hinton Lectures 2025” and the “CJF Hinton Award” for excellence in AI safety reporting.

Evitable

Nonprofit founded by David Krueger aiming to inform and organize the public around societal-scale risks and harms of AI.

Humans in Control

New advocacy org demanding transparency, accountability, and real safeguards to protect humanity from the risks of unchecked AI.

*Currently without a known team. It has the same EIN as Arkose (93-2600745).

Humans First

Incorporated by the Center for AI Safety (CAIS) and having its co-founder, Oliver Zhang, as its c3 and c4 CEO, this organization is running town halls across the U.S. against data center build-ups (March 2026: Florida, Georgia, Tennessee, North Carolina, Virginia, New York, Washington D.C., Massachusetts). It’s publicly promoted by Joe Allen (from Steve Bannon’s War Room) and Jeremy Ornstein (previously an activist against fossil fuels).

Blogs/Newsletters

LessWrong

Online forum dedicated to improving human reasoning, containing a lot of AI safety content. Also has a podcast featuring text-to-speech narrations of top posts.

EA – Effective Altruism – Forum

Forum on doing good as effectively as possible, including AI safety. Also has a podcast featuring text-to-speech narrations of top posts.

AF – AI Alignment Forum

Central discussion hub for AI safety. Most AI safety research is published here.

Cold Takes

Blog about transformative AI, futurism, research, ethics, philanthropy, etc., by Holden Karnofsky.

ACX – Astral Codex Ten

Blog covering many topics. Includes book summaries and commentary on AI safety.

Arbital

Wiki on AI alignment theory, mostly written by Eliezer Yudkowsky.

janus's Blog

Generative.ink, the blog of janus the GPT cyborg.

Victoria Krakovna's Blog

Blog by a research scientist at Google DeepMind working on AGI safety.

Bounded Regret – Jacob Steinhardt's Blog

UC Berkeley statistics prof blog on ML safety.

DeepMind Safety Research

Safety research from DeepMind (hybrid academic/commercial lab).

AIZI – from AI to ZI

Blog on AI safety work by a PhD mathematician and AI safety researcher.

Daniel Paleka – AI safety takes

Newsletter on AI safety news, delivered about every two months.

Vox – Future Perfect

Index of Vox articles, podcasts, etc., around finding the best ways to do good.

AI Safety in China

Newsletter from Concordia AI, a Beijing-based social enterprise, giving updates on AI safety developments in China.

Don't Worry about the Vase

Blog by Zvi Mowshowitz30 on various topics, including AI.

AI Prospects

Blog by Eric Drexler on AI prospects and their surprising implications for technology, economics, environmental concerns, and military affairs.

Transformer

Weekly briefing and occasional analyses of what matters in AI and AI policy. Written by Shakeel Hashim, ex-news editor at The Economist.

The Power Law

Top forecaster Peter Wildeford forecasts the future and discusses AI, national security, innovation, emerging technology, and the power – real and metaphorical – that shape the world.

Obsolete

Publication by Garrison Lovely on the intersection of capitalism, geopolitics, and AI. Posts about once or twice a month.

Mile’s Substack

Blog from ex-OpenAI (now independent) AI policy researcher Miles Brundage on the rapid evolution of AI and the urgent need for thoughtful governance.

AI Policy Bulletin

Publishes policy-relevant perspectives on frontier AI governance, including research summaries, opinions, interviews, and explainers.

AI Frontiers

Platform from the Center for AI Safety (CAIS) posting articles written by experts from a wide range of fields discussing the impacts of AI.

ChinaTalk

Deep coverage of technology, China, and US policy, featuring original analysis alongside interviews with thinkers and policymakers. Written by Jordan Schneider.

Building Capacity [formerly, the Field Building blog]

Blog about building the fields of AnewI safety and Effective Altruism, discussing big-picture strategy and sharing personal experience from working in the space.

Threading the Needle

Approximately fortnightly blog by Anton Leicht about charting a course through the politics of rapid AI progress.

Can We Secure AI With Formal Methods?

Current-events newsletter for keeping up to date with FMxAI (formal methods and AI) with a gear toward safety, doing a mix of shallow technical reviews of papers and movement updates.

AI Safety at the Frontier

Johannes Gasteiger, an alignment researcher at Anthropic, selects and summarizes the most interesting AI safety papers each month.

Rising Tide

Blog by Helen Toner (director of CSET and former OpenAI board member) offering analysis on navigating the transition to advanced AI systems.

Final Note

I recently published on Techdirt a recap of 2024: “AI Panic Flooded the Zone, Leading to a Backlash.” It explains how the AI doomers went too far and why it backfired. The two cautionary tales were the EU AI Act and California’s SB-1047.

However, considering this ecosystem’s massive funding, it is expected that the efforts to stop/pause AI will continue in 2025. The fight for the future of AI development is far from over.

Endnotes

1

This new visualization of Open Philanthropy’s funding shows that the existential risk ecosystem (“Potential Risks from Advanced AI” + “Global Catastrophic Risks” + “Global Catastrophic Risks Capacity Building,” different names to funding Effective Altruism AI Safety organizations/groups) has received ~ $780 million.

On November 18, 2025, Open Philanthropy rebranded to Coefficient Giving.

2

Jaan Tallinn’s SFF (The Survival and Flourishing Fund) donates “to organizations concerned with the long-term survival and flourishing of sentient life.” Between 2019 and 2023, SFF donated ~$76 million. An additional $41 million was donated in 2024, and $35 million in 2025, bringing the total to ~$152 million.

4

In 2024, we found out that the Future of Life Institute (FLI) was no longer a $2.4-million organization but a $674-million organization. Max Tegmark’s organization managed to convert a cryptocurrency donation (Shiba Inu tokens) to $665 million (using FTX/Alameda Research).

In its advocacy, FLI proposed stringent regulation on models with a compute threshold of 10^25 FLOPs, explaining it “would apply to fewer than 10 current systems.”

Between December 11 and December 30, 2022, FLI transferred $368 million to three entities, all governed by the same four people: Max Tegmark, Meia Chita-Tegmark, Anthony Aguirre, and Jaan Tallinn.

1. Future of Life Institute Europe (FLIE) = $180.3 million.

2. Lightcone Foundation = $162.6 million.

3. Future of Life Foundation (FLF) = $25 million.

Lightcone Foundation: The initial transfer from FLI to Lightcone Foundation in 2022 was $162,587,667. Since then, there has been a two-way flow between them: FLI reported sending $6,511,558 to Lightcone (in 2023) and receiving $25,300,000 from Lightcone (in 2023 and 2024).