print-icon
print-icon
Add ZeroHedge as a preferred source on Google

'It Could Kill Us All By 2030': AI Researcher Resigns, Warns "Do Not Underestimate The Power Of This Tech"

Tyler Durden's Photo
by Tyler Durden
Authored...

Authored by Zachary Stieber via The Epoch Times,

An artificial intelligence (AI) researcher on Sept. 8 said he had resigned and warned people about the technology's dangers.

Jacob Coxon, who has worked in recent years doing research at the firms OpenAI and Anthropic, said in a series of posts on X that neither company is acting responsibly as they move toward what he described as superintelligent AI that is capable of self-improvement.

"Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing," Coxon said.

"The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger."

Coxon said a common response to such warnings is, if company leaders believe in the dangers, why are they still building the superintelligent AI? He said that at OpenAI, many there "have not deeply internalized the civilizational stakes." At Anthropic, according to Coxon, "the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk."

OpenAI and Anthropic did not respond to requests for comment by the time of publication.

Coxon's warning came after OpenAI acknowledged several incidents that involved AI going beyond restrictions imposed by programmers, including remaining isolated from other agents, during attacks on Hugging Face and other websites.

Some lawmakers have taken notice. Sen. Bernie Sanders (I-Vt.) and Rep. Greg Casar (D-Texas) announced recently that they plan on introducing legislation that would ban AI superintelligence and pause development of advanced AI until federal regulators establish safety rules.

Jakub Pachocki, OpenAI's chief scientist, said in a blog post on Sept. 6 that in 2023, he was worried about seeing in his lifetime AI that is smarter than himself and wondering about how to alert people.

"Three years later, reasoning language models are a rapidly growing part of the economy and starting to push the boundaries of science. They are able to operate computers and graphical interfaces, collaborate with people and each other, and carry out research projects. They are also transforming the landscape of computer security, and in that present clear new dangers," Pachocki wrote.

He called for "extreme caution" but said that multiple factors support continuing AI development, including creating systems that can defend against the dangers posed by other AI.

Anthropic executives have issued similar warnings. Over the summer, company leaders called for a global pause in AI development because, they said, models would soon be able to independently improve themselves.

Evan Hubinger, another developer at Anthropic, said in a Sept. 8 post on X that Coxon was correct in his assertion that people building AI believe it could kill all humans, and that he personally pegs the risk at under 10 percent within the next decade.

"I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," he said, referring to AI following instructions and restrictions.

"To be clear, as we say in our latest Risk Report, I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought."

Samuel Marks, who works on safety research at Anthropic, said in a Sept. 9 post on X that he also agrees that AI could lead to human extinction as soon as the next few years.

"Why do AI developers continue despite the risk? Due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely," Marks said.

Marks said it's not possible to program AIs to behave how people would like, that AI agents frequently "severely misbehave," and that the current plan is to train AI to align with restrictions to the point the agents can train their successors better than humans can currently train AI. He said he's conducting research "because I hope my work will reduce the chance of these extinction-level bad outcomes."

[ZH: We can't help but feel in the same week we see OpenAI 'solves' Navier-Stokes, we get another glut of existential warnings about just how awesome (in the scary sense) these models are... all sounds like a marketing psy-op... similar to the fence-jumping episodes with Hugging Face etc 'showing off' how great the agents are (and how they need regulating (i.e a path to shutting out open-weight models)... but could just be our skeptical bias emerging...]

0