THE RISK OF EXTINCTION
In AI circles, estimates of catastrophic or existential risk are often referred to as p(doom)—the probability that advanced AI could lead to catastrophe, including human extinction.
The current debate erupted after researcher Jacob Coxon resigned from Anthropic, warning that people building advanced AI genuinely believe it could kill humanity. Evan Hubinger, who leads alignment research at Anthropic, publicly agreed—and put his own estimate at greater than 10% within the next decade.
No one knows whether 10% is remotely correct. There is no validated way to calculate such a probability, and experts disagree sharply about the risk.
But when the possible consequence is human extinction, even profound uncertainty deserves attention.
POSSIBLE SOLUTIONS
1. Establish red lines. - AI pioneer Stuart Russell recently proposed a practical response: establish red lines. For example, an AI should not be able to replicate itself without authorization or break into other computer systems. If a system crosses a red line and developers cannot demonstrate that the risk has been addressed, development stops.
2. Establish accountability. - AI companies should also be held accountable when the systems they develop and deploy cause harm. Congress should make clear that Section 230—the law that protects internet platforms from liability for much third-party content—should not shield companies from responsibility for harms caused by their own AI systems.
3. Other ideas. - Other proposals include emergency “kill switches,” greater government review of frontier AI systems, and new government bodies focused specifically on AI risk.
We may never know the true p(doom).
But we don't need to know it to agree that some lines should not be crossed.
JeffreyLCooper.com
Technology Analyst & Author
My novel: When Machines Begin to Dream amazon.com/dp/B0H85WBVC8
“Profound and moving.” The U.S. Review of Books | “Gripping.” Midwest Book Review | “Crisp and cinematic.” The BookLife Prize
#ArtificialIntelligence #AISafety #AI