The “AI Existential Risk” Industrial Complex
This is your guide to the growing “AI Existential Risk” ecosystem.
The “AI Existential Risk” Ideology
ChatGPT’s launch in 2022 sparked a wave of AI doomerism. The leading voices in the “AI will kill us all” camp have confidently predicted the END of humanity is near. “We believe humans will be wiped out by Godlike AI.” Though “with enough money and effort, we might be able to save humanity from the impending AI apocalypse.” They talked about an “AI takeover” during their takeover of media and policymaking. Since then, we have been in an unprecedented AI panic.
The “AI Existential Risk” Ecosystem’s Funding
Their “AI Existential Risk” ideology had the financial backing of $1.6 billion from a few Effective Altruism billionaires, namely Dustin Moskovitz, Jaan Tallinn, Vitalik Buterin, and Sam Bankman-Fried (yes, the convicted felon).
Moskovitz’s Open Philanthropy is by far the largest donor1 to most of the “AI Existential Risk” organizations.
The “AI Existential Risk” Lobbying
Nowadays, the “AI Existential Risk” ecosystem encompasses hundreds of organizations. Many advocate extreme authoritarian measures to stop/pause AI. They include “requiring registration and verifying location of hardware,” “a strict licensing regime, clamp down on open-source models, and impose civil and criminal liability on developers.”
ControlAI even proposed a 20-year pause, because the “default” is human extinction by Godlike AI, and “two decades provide the minimum time frame to construct our defenses.”
Staying informed about this growing “AI existential risk” ecosystem is important. So, the information below aims to familiarize you with the various players involved.
The “AI Existential Risk” Map
The Map of AI Existential Safety was created by effective altruists so that the epistemic community of AI safety students/researchers/lobbyists could become more acquainted with all potential funding, resources, organizations, and projects.
Since there are so many, the accompanying spreadsheet (with its 260 rows) really comes in handy.
To make sense of this ecosystem’s expansion, the map offers the following categories: Funding; Strategy and Governance Research; Conceptual Research; Applied Research; Training and Education; Research Support; Career Support; Resources; Media; Public Outreach; and Blogs.2
The criteria for inclusion in the map: “An entry must have reducing AI existential risk as a goal; have produced at least one output to that end; must additionally still be active and expected to produce more outputs.”
I first published this map’s details in the “Ultimate Guide to ‘AI Existential Risk’ Ecosystem” on December 5, 2023. Afterward, I updated it in “Panic-as-a-Business is Expanding” on April 15, 2024.
Since then, 50 new entries have been added to this map.
Each item appears with a link and a description (some have endnotes with more data). The descriptions are identical to those on the map without edits. I let their words stand on their own.
The Key Takeaway from the “AI Existential Risk” Map
One conclusion can be drawn from reviewing the hundreds of links: It’s an inflated ecosystem.
There’s a great deal of redundancy:
The same names/acronyms/logos (with only minor changes)
The same extreme talking points
The same group of people, just with different titles
The same funding source (primarily Open Philanthropy).
This is clearly not a grassroots, bottom-up movement. It is a well-orchestrated top-down movement.
The media and politicians need to open their eyes to this reality.
Funding
The largest funder in the existential risk space.
SFF – Survival and Flourishing Fund3
The second largest funder in AI safety, using an algorithm and meeting procedure called “The S-process” to allocate grants.
Making grants addressing global catastrophic risks, promoting longtermism, and otherwise increasing the likelihood that future generations will flourish.
Community of donors who have pledged to donate a significant portion of their income.
FLI – Future of Life Institute – Fellowships
PhD and postdoc funding for work improving the future.
Longview Philanthropy [formerly Effective Giving UK]
Devises and executes bespoke giving strategies for major donors.
Regranting and impact certificates platform for AI safety and other cause areas.
CLR – Center on Long-Term Risk – Fund
Financial support for projects focused on reducing current and future s-risks.
AI Safety GiveWiki [Formerly Impact Markets]
Crowd-sourced charity evaluator, helping people find or promote the best AI safety projects.
EAIF – Effective Altruism Infrastructure Fund
Aiming to increase the impact of effective altruism projects by increasing their access to talent, capital, and knowledge.
Philanthropic initiative supporting researchers working on key opportunities and hard problems that are critical to get right for society to benefit from AI. Proposals by invite only.
SHfHS – Saving Humanity from Homo Sapiens
Small organization with a long history of funding existential risk reduction.
Decentralized bounty platform for existential risk reduction and other effective altruist cause areas.
Funder network for AI existential risk reduction.
GAIA – Grantmakers for AI Alignment
Joinable donor circle for people earning to give or allocating funds towards reducing AI existential risk.
Polaris Ventures [formerly the Center for Emerging Risk Research]
Swiss nonprofit making grants and investments focused on reducing long-term suffering.
NSF – National Science Foundation – Safe Learning-Enabled Systems
Funds research into the design and implementation of safe learning-enabled systems in which safety is ensured with high levels of confidence.
CAIF – Cooperative AI Foundation
Charity foundation backed by a large philanthropic commitment supporting research into improving cooperative intelligence of advanced AI.
Offers grants to high-impact organizations and projects that are taking bold action and making significant changes.
Network of donors funding charitable projects that work one level removed from direct impact.
VC firm investing in ethical founders developing transformative technologies that have the potential to impact humanity on a meaningful scale.
Aiming to reduce catastrophic risks from advanced AI through grants towards technical research, policy, and training programs for new researchers.
Empowering innovators and scientists to increase human agency by creating the next generation of responsible AI. Providing support, resources, and open-source software.
Helping major donors find, fund, and scale the most promising solutions to the world’s most pressing problems.
Funding projects in 1) automating research and forecasting, 2) security technologies, 3) neurotech, and 4) safe multipolar human AI scenarios.
Aiming to empower founders building a radically better world with safe AI systems by investing in ambitious teams with defensible strategies that can scale to post-AGI.
FLF – Future of Life Foundation4
Accelerator aiming to steer transformative technology towards benefiting life and away from extreme large-scale risks.
Strategy and Governance Research
FLI – Future of Life Institute5
Outreach, policy advocacy, grantmaking, and event organization for existential risk reduction.
GovAI – Centre for the Governance of AI
AI governance research group at Oxford, producing research tailored towards decision-makers and running career development programs.
CLR – Center for Long-Term Risk [formerly the EA Foundation]
Research, grants, and community-building around AI safety, focused on conflict scenarios as well as technical and philosophical aspects of cooperation.
CSET – Center for Security and Emerging Technology6
Georgetown University think tank providing decision-makers with data-driven analysis on the security implications of emerging technologies.
Channeling public concern into effective regulation by engaging with policymakers, media, and the public to ensure AI is developed responsibly and transparently.
CLTR – Center for Long-Term Resilience
Think tank aiming to transform global resilience to extreme risks by improving relevant governance, processes, and decision-making.
CFG – Centre for Future Generations8 [Formerly, ICFG - International Center for Future Generations]
Brussels think tank focused on helping governments anticipate and responsibly govern the societal impacts of rapid technological change.
Play-money prediction markets on many topics, including AI safety.
TFI – Transformative Futures Institute
Exploring the use of underutilized foresight methods and tools in order to better anticipate societal-scale risks from AI.
Forecasting platform for many topics, including AI.
FRI – Forecasting Research Institute
Advancing the science of forecasting for the public good by working with policymakers and nonprofits to design practical forecasting tools, and test them in large experiments.
QURI – Quantified Uncertainty Research Institute
Advancing forecasting and epistemics to improve the long-term future of humanity. Writing research and software.
Answering decision-relevant questions about the future of AI, including through research, a wiki, and expert surveys. Run by MIRI.
Research institute investigating key trends and questions that will shape the trajectory and governance of AI.
Research and implementation group identifying pressing opportunities to make the world better.
Building a foundational series of sociotechnical reports on key AI scenarios and governance recommendations, and conducting AI awareness efforts to inform the general public.
Defining, designing, and deploying projects that address institutional barriers in AI governance.
GCRI – Global Catastrophic Risk Institute
Small think tank developing solutions for reducing existential risk by leveraging both scholarship and the demands of real-world decision-making.
CSER – Centre for the Study of Existential Risk
Interdisciplinary research centre at the University of Cambridge doing diverse existential risk research.
CFI – Centre for the Future of Intelligence
Interdisciplinary research centre at the University of Cambridge exploring the nature, ethics, and impact of AI.
GPI – Global Priorities Institute
University of Oxford research center conducting foundational research to inform the decision-making of individuals and institutions seeking to do as much good as possible.
Research nonprofit working on models of past and future progress in AI, intelligence enhancement, and sociology related to existential risks.
Nonpartisan research organization developing policy and conducting advocacy to mitigate catastrophic risks from AI.
Provides strategy consulting services to clients trying to advance AI safety through policy, politics, coalitions, or social movements.
Convening academic, civil society, industry, and media organizations to create solutions so that AI advances positive outcomes for people and society.
IAPS – Institute for AI Policy and Strategy
Research and field-building organization focusing on policy and standards, compute governance, and international governance and China.
LawAI – Institute for Law & AI [formerly, LPP – Legal Priorities Project]
Think tank researching and advising on the legal challenges posed by AI.
UK AISI – UK AI Safety Institute
UK government organisation conducting research and building infrastructure to test the safety of advanced AI and measure its impacts. Also working to shape global policy.
USAISI – U.S. AI Safety Institute
US government organization working to advance the science, practice, and adoption of AI safety across the spectrum of risks.
AIGS Canada – AI Governance & Safety Canada
Nonpartisan not-for-profit and community of people across Canada, working to ensure that advanced AI is safe and beneficial for all.
CARMA – Center for AI Risk Management & Alignment12
Conducting interdisciplinary research supporting global AI risk management. Also produces policy and technical research.
CLAI – Center for Long-term AI
Interdisciplinary research organization based in China exploring contemporary and long-term impacts of AI on society and ecology.
Platform for connecting junior researchers and seasoned civil servants from Southeast Asia with senior AI safety researchers from developed countries.
CeSIA – Center for AI Security
French AI safety nonprofit dedicated to education, research, and advocacy.
Promoting informed policymaking to navigate emerging challenges from AI through research, knowledge-sharing, and skill building.
A series of proposals developed by ControlAI intended for action by policymakers in order for humanity to survive artificial superintelligence.
AIGSI – AI Governance and Safety Institute
Aiming to improve institutional response to existential risk from future AI systems by conducting research and outreach, and developing educational materials.
IASEAI – International Association for Safe and Ethical Artificial Intelligence
Nonprofit aiming to ensure that AI systems are guaranteed to operate safely and ethically; and to shape policy, promote research, and build understanding and community around this goal.
Beijing AISI – Beijing Institute of AI Safety and Governance
R&D institution dedicated to developing AI safety and governance frameworks to provide a safe foundation for AI innovation and applications.
Established within the European Commission as the centre of AI expertise, playing a key role in implementing the AI Act.
Nonprofit fighting to keep humanity in control of AI by developing policy and conducting public outreach.
Geneva-based think tank working to foster international cooperation in mitigating catastrophic risks from AI.
Conceptual Research
MIRI – Machine Intelligence Research Institute
The original AI safety technical research organization, co-founded by Eliezer Yudkowsky. Now focusing on policy and public outreach.
ARC – Alignment Research Center
Research organization doing theoretical research focusing on the Eliciting Latent Knowledge (ELK).
Accelerating alignment progress by extending human cognition with AI.
Formal alignment organization led by Tamsin Leake, focused on agent foundations.
ALTER – Association for Long-Term Existence and Resilience
Israeli research and advocacy nonprofit working to investigate, demonstrate, and foster useful ways to safeguard and improve the future of humanity.
Independent researcher working on selection theorems, abstraction, and agency.
CHAI – Center for Human-Compatible AI
Developing the conceptual and technical wherewithal to reorient the general thrust of AI research towards provably beneficial systems. Led by Stuart Russell at UC Berkeley.
Independent researchers trying to find reward functions which reliably instill certain values in agents.
ACS – Alignment of Complex Systems
Studying questions about multi-agent systems composed of humans and advanced AI. Based at Charles University, Prague.
Assistant professor at MIT working on agent alignment.
Author of two books on AI safety, and Professor at the University of Louisville with a background in cybersecurity.
MIT Algorithmic Alignment Group
Working towards better conceptual understanding, algorithmic techniques, and policies to make AI safer and more socially beneficial.
Large team taking a 'Neglected Approaches' approach to alignment, tackling the problem from multiple, often overlooked angles in both technical and policy domains.
Argentine nonprofit conducting both theoretical and empirical research to advance frontier AI safety as a sociotechnical challenge.
Applied Research
Google DeepMind
AI capabilities lab with a strong safety team.
San Francisco-based capabilities lab led by Sam Altman. Created ChatGPT.
Research lab focusing on LLM alignment, particularly interpretability. Featuring Chris Olah, Jack Clark, and Dario Amodei.
San Francisco-based nonprofit conducting safety research, building the field of AI safety researchers, and advocating for safety standards.
Alignment startup born out of EleutherAI, building LLMs and Cognitive Emulation systems.
Nonprofit researching interpretability and alignment.
Open-source research lab focused on interpretability and alignment. Operates primarily through a public Discord server, where research is discussed and projects are coordinated.
Product-driven research lab developing mechanisms for delegating high-quality reasoning to ML systems. Built Elicit, an AI assistant for researchers and academics.
An Oxford-based startup working on safe off-distribution generalization, featuring Stuart Armstrong.
FAR AI18 [Fund for Alignment Research]
Ensuring AI systems are trustworthy and beneficial to society by incubating and accelerating research agendas too resource-intensive for academia but not yet ready for commercialization.
Aiming to detect deception by designing AI model evaluations and conducting interpretability research to better understand frontier models. Also provides policymakers with technical guidance.
Nonprofit researching applications of singular learning theory to AI safety.
AI safety research community based in a small town in Vermont, USA.
Researching AI competition dynamics and building research software tools.
Nonprofit research lab building AI tools to defend and enhance human agency – by researching and experimenting with novel AI capabilities.
METR – Model Evaluation & Threat Research19
Evaluating whether cutting-edge AI systems could pose catastrophic risks to society.
ARG – NYU Alignment Research Group
Group of researchers at New York University doing empirical work with language models aiming to address longer-term concerns about the impacts of deploying highly-capable AI systems.
CBL – University of Cambridge Computational and Biological Learning Lab
Research group using engineering approaches to understand the brain and to develop artificial learning systems.
MAI – Meaning Alignment Institute
Research organization applying expertise in meaning and human values to AI alignment and post-AGI futures.
French nonprofit working to incentivize responsible AI practices through policy recommendations, research, and risk assessment tools.
AI safety research group at the University of Cambridge led by David Krueger.
SSI – Safe Superintelligence Inc.
Research lab founded by Ilya Sutskever comprised of a small team of engineers and researchers working towards building a safe superintelligence.
China AISI – AI Development and Safety Network
Network of industry, academia, and research institutions within China, representing China in conducting exchanges and cooperation with AI research institutions worldwide.
For-profit company developing tools that automatically assess the risks of AI models and developing its own AI models aiming to provide best-in-class safety and security.
Nonprofit research lab building open source, scalable, AI-driven tools to understand and analyze AI systems and steer them in the public interest.
Training and Education
Runs the standard introductory courses, each three months long and split into two tracks: Alignment and Governance. Also runs shorter intro courses.
MATS – ML Alignment & Theory Scholars program20
Research program connecting talented scholars with top mentors in AI safety. Involves 10 weeks onsite mentored research in Berkeley, and, if selected, 4 months extended research.
3-month online research program with mentorship.
ERA – Existential Risk Alliance – Fellowship [spin off of CERI – Cambridge Existential Risk Initiative]
In-person paid 8-week summer AI safety research fellowship at the University of Cambridge.
Pairs fellows from disciplines studying complex and intelligent behaviour in natural and social systems with mentors from AI alignment.
Runs AI safety events and training programs in London.
Comprehensive online introductory course on ML safety. Run by CAIS.
GCP – Global Challenges Project
Intensive 3-day workshops for students to explore AI safety (and biosecurity).
HAIST – Harvard AI Safety Student Team
Group of Harvard students conducting AI safety research and running fellowships, workshops, and reading groups.
Student group conducting AI safety research and running workshops and reading groups.
CBAI – Cambridge Boston Alignment Initiative
Boston organization for helping students get into AI safety via upskilling programs and fellowships. Supports HAIST and MAIA.
CLR – Center on Long-Term Risk – Summer Research Fellowship
2–3-month summer research fellowship in London working on reducing long-term future suffering.
HA – Human-aligned AI – Summer School21
4-day program for teaching alignment research methodology.
CHAI – Center for Human-Compatible AI – Internship
Research internship at UC Berkeley for people interested in research in human-compatible AI.
ARENA – Alignment Research Engineer Accelerator
4–5 week ML engineering upskilling program in London, focusing on alignment. Draws heavily from MLAB.
SERI – Stanford Existential Risk Initiative – Fellowship
10-week funded summer research fellowship for undergrad and grad students (primarily at Stanford).
AISI – AI Safety Initiative at Georgia Tech
Georgia Tech community hosting research projects and a fellowship.
Student group and research community under SERI. Accelerating students into AI safety careers in AI safety, building the community at Stanford, and conducting research.
French collective promoting mission-driven research to tackle global issues. Organizes conferences, hackathons, ML4Good bootcamps, and university reading groups and research projects.
WAISI – Wisconsin AI Safety Initiative
Wisconsin student group dedicated to reducing AI risk through alignment and governance.
AISG – AI Safety Initiative Groningen
Student group in Groningen, Netherlands.
10-day intensive, in-person bootcamps upskilling participants in technical AI safety research.
SPAR – Supervised Program for Alignment Research
Virtual, part-time research program offering early-career individuals and professionals the chance to engage in AI safety research for 3 months.
Monthly hackathons around the world for people getting into AI safety.
MARS – Mentorship for Alignment Research Students
Research program connecting aspiring researchers with experienced mentors to conduct AI safety (technical or policy) research for 2–3 months.
Monthly hackathons around the world for people getting into AI safety.
Supports students and professionals in contributing to the safe development of AI.
Annual 9-week program designed to enable promising researchers to produce impactful research and accelerate their careers in AI safety (or biosecurity).
Filipino nonprofit aiming to develop more AI interpretability and safety researchers, particularly in Southeast Asia.
1-year program catalyzing collaboration among young scientists, engineers, and innovators working to advance technologies for the benefit of life.
CAISH – Cambridge AI Safety Hub
Network of students and professionals in Cambridge conducting research, running educational and research programs, and creating a vibrant community of people with shared interests.
LASR – London AI Safety Research – Labs
12-week research program aiming to assist individuals in transitioning to full-time careers in AI safety.
XLab – UChicago Existential Risk Laboratory – Fellowship
10-week summer research fellowship giving undergraduate and graduate students the opportunity to produce high impact research on various emerging threats, including AI.
Impact Academy: Global AI Safety Fellowship
Fully-funded research program connecting exceptional STEM researchers with full-time placement opportunities at AI safety labs and organizations.
Training for Good - Talos Fellowship
7-month program enabling ambitious graduates to launch EU policy careers reducing risks from AI.
Training for Good – Tarbell Fellowship
A one-year programme for early-career journalists interested in covering artificial intelligence. Fellows secure a 9-month placement at a major newsroom.
Research Support
Maintains LessWrong, the Alignment Forum, and Lighthaven (an event space in Berkeley).
“Means-neutral” AI safety organization, doing miscellaneous stuff, including offering bounties on small-to-large AI safety projects and running a funder network.
CEEALAR – Centre for Enabling EA Learning & Research22 [formerly “EA Hotel”]
Free or subsidized accommodation and catering in Blackpool, UK, for people working on/transitioning to working on global catastrophic risks.
AED – Alignment Ecosystem Development
Building and maintaining key online resources for the AI safety community. Volunteers welcome.
Non-profit AI safety research lab hosting open-to-all research sprints, publishing papers, and incubating talented researchers to make AI safe and beneficial for humanity.
Consultancy for forecasting, machine learning, and policy. Doing original research, evidence reviews, and large-scale data pipelines.
BERI – Berkeley Existential Risk Initiative
Providing free operations support for university research groups working on existential risk.
SERI – Stanford Existential Risks Initiative
Collaboration between faculty and students. Runs research fellowships, an annual conference, speaker events, discussion groups, and a frosh-year class.
Providing consultancy and hands-on support to help high-impact organizations upgrade their operations.
GPAI – Global Partnership on AI
International initiative with 44 member countries working to implement human-centric, safe, secure, and trustworthy AI embodied in the principles of the OECD Recommendation on AI.
ENAIS – European Network for AI Safety
Connecting researchers and policymakers for safe AI in Europe.
Berkeley research center growing and supporting the AI safety ecosystem.
LISA – London Initiative for Safe AI
Coworking space hosting organizations (including BlueDot Impact, Apollo, Leap Labs), acceleration programs (including MATS, ARENA), and independent researchers.
AI safety field-building nonprofit. Runs support programs facilitating technical research, does outreach, and curates educational resources.
Working to ensure the benefits of data and AI is justly and equitably distributed, and enhances individual and social wellbeing.
Incubating and supporting early-stage AI safety research organizations.
Fostering responsible governance of AI to reduce catastrophic risks through shared understanding and collaboration among key global actors.
Providing a platform that allows companies and individuals to evaluate the capabilities of AI models and therefore know how much they can trust them.
Providing fiscal sponsorship to AI safety projects.
Helping concerned individuals working at the frontier of AI get expert opinions on their questions, anonymously and securely.
Career Support
Helps people navigate the AI safety space with a welcoming human touch, offering personalized guidance and fostering collaborative study and project groups.
Field-building organization with an extensive resources list.
Article with motivation and advice for pursuing a career in AI safety.
Helping professionals transition to high-impact work by performing market research on impactful jobs and providing career mentoring, opportunity matching, and professional training.
AI Safety Google Group [formerly 80,000 Hours AI Safety Group]
Information about how to get into technical research, including updates on academic posts, grad school, and training programs.
Curated list of job postings tackling pressing problems, including AI safety.
Effective Thesis – Academic Opportunities
Lists thesis topic ideas in AI safety and coaches people working on them.
Nonlinear – Coaching for AI Safety Entrepreneurs
Free coaching for people running an AI safety startup or considering starting an AI safety org (technical, governance, meta, for-profit, or non-profit).
HIP – High Impact Professionals
Supporting working professionals to maximize their positive impact through their talent directory and Impact Accelerator Program.
Resources
How to pursue a career in technical AI alignment
A guide for people who are considering direct work on technical AI alignment.
Interactive FAQ; Single-Point-Of-Access into AI safety. Part of AISafety.info.
Comprehensive database of available training programs and events.
Repository of possible research projects and testable hypotheses. Run by Apart Research.
Database of AI safety research agendas, people, organizations, and products.
Ranked and scored contributable compendium of alignment plans and their problems.
Interactive walkthrough of core AI x-risk arguments and transcripts of conversations with AI researchers. Project of Arkose (2022).
Database of grants in effective altruism.
Website tracking donations to AI safety. Full site planned for launch in 2025.
Interactive explainers on AI capabilities and their effects. (By Sage Future).
Helps you learn and memorize the main organizations, projects, and programs currently operating in the AI safety space.
Cartoon map showing various organizations, projects, and policies in the AI governance space.
A guide to AI safety for those new to the space, in the form of a comprehensive FAQ and chatbot (Stampy).
The hub for key resources for the AI safety community, including directories of courses, jobs, upcoming events, and training programs, etc.
Collects actions for frontier Al labs to avert extreme risks from AI, then evaluates particular labs accordingly.
Living document aiming to present a coherent worldview explaining the race to AGI and extinction risks and what to do about them – in a way that is accessible to non-technical readers.
Helping ensure domains in cause areas like AI safety are pointed towards high-impact projects.
Media
YouTuber covering the latest developments in AI.
Interviews on pursuing a career tackling pressing problems, including AI safety.
Interviews with AI safety researchers, explainers, fictional stories of concrete threat models, and paper walk-throughs.
Animated videos on effective altruism, rationality, the future of humanity, and AI safety.
Comprehensive index of AI safety video content.
Occasional newsletter from CAIS, focused on applied AI safety and ML.
Weekly developments in AI (incl. governance) written by Jack Clark, co-founder of Anthropic.
ERO – Existential Risk Observatory
Reducing existential risks by informing the public debate.
AXRP – AI X-risk Research Podcast
Interviews with technical AI safety researchers about their research.
FLI – Future of Life Institute – Podcast
Interviews with existential risk researchers.
AI safety explainers in video form.
Newsletter published every few weeks discussing developments in AI and AI safety. No technical background required. Run by CAIS.
Public Outreach
Campaign group aiming to convince governments to pause AI development – through public outreach, engaging with decision-makers and organizing protests.
Calling on policymakers to implement a global moratorium on large AI training runs until alignment is solved.
Watchdog nonprofit monitoring tech companies, countering corporate propaganda, raising awareness about corner-cutting, and advocating for the responsible development of AI.
AISAF – AI Safety Awareness Foundation
Volunteer organization dedicated to raising awareness about modern AI, highlighting its benefits and risks, and letting the public know how they can help – mainly through workshops.
Blogs
Online forum dedicated to improving human reasoning, containing a lot of AI safety content. Also has a podcast featuring text-to-speech narrations of top posts.
EA – Effective Altruism – Forum
Forum on doing good as effectively as possible, including AI safety. Also has a podcast featuring text-to-speech narrations of top posts.
Central discussion hub for AI safety. Most AI safety research is published here.
Blog about transformative AI, futurism, research, ethics, philanthropy, etc., by Holden Karnofsky.
Blog covering many topics. Includes book summaries and commentary on AI safety.
Wiki on AI alignment theory, mostly written by Eliezer Yudkowsky.
Generative.ink, the blog of janus the GPT cyborg.
Blog by a research scientist at Google DeepMind working on AGI safety.
Bounded Regret – Jacob Steinhardt's Blog
UC Berkeley statistics prof blog on ML safety.
Safety research from DeepMind (hybrid academic/commercial lab).
Blog on AI safety work by a PhD mathematician and AI safety researcher.
Daniel Paleka – AI safety takes
Newsletter on AI safety news, delivered about every two months.
Index of Vox articles, podcasts, etc., around finding the best ways to do good.
Newsletter from Concordia AI, a Beijing-based social enterprise, giving updates on AI safety developments in China.
Blog by Zvi Mowshowitz26 on various topics, including AI.
Blog by Eric Drexler on AI prospects and their surprising implications for technology, economics, environmental concerns, and military affairs.
No Longer Active
ML & AI Safety Updates (Apart Research)
Weekly podcast, YouTube, and newsletter with updates on AI safety. Dropped due to organizational reprioritization.
AISS – AI Safety Support – Newsletter
Lists opportunities in alignment. Dropped due to lack of a maintainer.
Fast funding for projects that help humanity flourish among the stars. Last funding round was 2023.
Funding organization aiming to minimize the risk of AI systems, created by PreambleAI. Died for reasons unknown to us.
OpenAI – Superalignment Fast Grants
10 million USD in grants to support technical research toward aligning superintelligent AI.
Runs research paper/essay-writing contests to advance alignment. Last round was 2023.
FHI – Future of Humanity Institute
Longtermist/x-risk research organization led by Nick Bostrom at the University of Oxford. Killed by university politics disallowing them from fundraising, now transitioning to the Macrostrategy Research Initiative.
Stop AGI [formerly, Stop AI]
Website communicating the risks of god-like AI to the public and offering proposals on preventing its development.
PAISRI – Phenomenological AI Safety Research Institute
Performs and encourages AI safety research using phenomenological methods.
A video game company focused on enabling the safe introduction of AI technologies into gaming.
Blog on aligning prosaic AI by one of the leading AI safety researchers.
Supported individuals pursuing a career in AI safety by running AI Safety Hub Labs, where participants complete their first research project – now run by LASR Labs.
AISHED – AI Safety Hub Edinburgh
Community of people interested in ensuring that AI benefits humanity’s long-term future.
MLAB – Machine Learning for Alignment Bootcamp
Bootcamp aimed at teaching ML relevant to doing alignment research. Run by Redwood Research for two iterations in 2022.
Effective Thesis – Early Career Research Opportunities
Lists academic career opportunities for early-stage researchers (jobs, bootcamps, internships).
AI Safety Fundamentals – Opportunities Board
Curated list of opportunities to directly work on technical AI safety.
AI Safety Info Distillation Fellowship
3-month paid fellowship to write content for Stampy's AI Safety Info.