Anthropic Issues Warning Over Self-Replicating AI - And Says A Chinese Model Pulled It Off
Dario's back with another one.
Anthropic's Frontier Red Team (yes, really) warned Monday in a blog post that small AI models may be "months away" from copying themselves across smart ovens, off-brand Android TV boxes and other poorly secured gadgets in your home, forming a botnet with no off switch.
"A replicator like this would be challenging to shut down, since there's no centralized point to shut off - though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own."
To prove the point, researchers bought 15 TV boxes at about $55 each, networked them together, and ran Google's Gemma model across them at a glacial 0.39 tokens per second - in what Anthropic called the coming "digital slime mold."
Co-author Daniel Freeman was even more breathless on X:
If you react to this by thinking it’s kind of sci-fi, that’s because I kind of think it is!
— Daniel Freeman (@bucketofkets) October 12, 2026
This is super weird!
We did not used to live in a world where little guys in boxes could replicate across networks, gobble up computers, reconfigure themselves to work better in the machines they find, etc etc
His conclusion: "Regardless, whatever we do, they're coming!"
Anthropic also tested whether models could run the entire replication cycle on their own: scan a network, hop to a new device, deliver the payload, get running, repeat. Exactly one succeeded:
"Only GLM-5.2 out of our set of tested open-weight models is capable of full end-to-end replication."
GLM-5.2 comes from Beijing-based Z.ai, formerly Zhipu AI. It spread to all 8 test nodes in four of five trials. The small-model workhorse in the rest of the experiments was Alibaba's Qwen.
Neither name is new to Anthropic's threat reports. As we noted last month, Anthropic says it disrupted industrial-scale "distillation" campaigns against Claude by seven China-based labs, and Alibaba and Z.ai were both on the list. As we reported in June, Anthropic took its Alibaba accusations straight to senators and White House officials. Fittingly, the "related content" box under Anthropic's new post links straight to its latest piece on fending off distillation attacks.
The Fine Print
That said, the hacking step was made trivial with unlocked network ports, which Anthropic itself calls an "upper bound on real-world success." GLM-5.2 is also far too big to live on a TV box, so it ran on a separate server and installed a small Llama model on each target. The real fear is a small model reaching GLM-5.2's level, and Anthropic's own trendline puts that at year-end.
Red Team, Meet Red China
Washington has spent months debating what to do about Chinese open-weight AI. In July, Treasury Secretary Scott Bessent warned that Chinese firms caught running industrial-scale distillation could face sanctions and Entity List designations:
We support open-source AI and the innovation it unlocks. But open source is not open season on American IP. When PRC firms conduct covert, industrial-scale distillation attacks that cross the line into IP theft, sanctions and Entity List designations will be on the table.
— Treasury Secretary Scott Bessent (@SecScottBessent) July 22, 2026
Sens. Cassidy and Rosen want a public list of AI models from adversarial nations, with agencies needing an exemption to use them.
Anthropic insists it isn't asking for a ban - after very clearly begging for one. In July, Dario wrote that "Anthropic has never advocated for a ban on open-weights models." What Anthropic says it wants instead is chip export controls on China and mandatory safety testing. The same post cites warnings that releasing open weights for highly cyber-capable models creates permanent risk, since safeguards can be stripped and copies run beyond anyone's monitoring.
That's where paper casting a Chinese open model as proto-malware comes in Dario never has to say "ban China." It just builds the evidence for capability gates on open-weight releases. Analysts warn those gates could raise costs and shrink the supply of advanced open models, with smaller developers least able to comply. Labs that don't release weights, like Anthropic, would be untouched.
Anthropic was also notably absent from the industry letter, backed by Nvidia, Microsoft, Meta, IBM, Mistral and Hugging Face, urging policymakers not to rush into open-weight restrictions.
The Tour Continues
As we noted in September, Dario wants the government to broker an industry slowdown and grant the labs a narrow antitrust waiver. His specific asks were to cut off Nvidia chip sales to China and crack down on distillation. House Speaker Mike Johnson warned the plan could "smother innovation" and hand China the lead. David Sacks summed it up as Dario wanting a "DMV for AI."
The money keeps flowing too. Anthropic spent $3.53 million lobbying Washington in the first half of 2026, nearly tripling its lobbying and outspending OpenAI by more than $1 million.
China, meanwhile, keeps taking over the open market. By May, Chinese open-weight models accounted for roughly 61% of tokens on OpenRouter, and Qwen had passed 1 billion downloads on Hugging Face. Z.ai released GLM-5.2 on June 13 for anyone in the world to download and run, one day after Commerce export controls forced Anthropic to suspend access to its newest models. Nearly 200 startups have told the White House that banning Chinese open-weight models would hurt American builders more than Chinese labs. Guess who wouldn't be hurt.
The irony is that Anthropic's own list of defenses doesn't include a single new rule for AI labs. It calls for ISP traffic monitoring, quarantining infected subscribers, and killing default IoT passwords. By its own account, the threat is cheap gadgets nobody bothered to secure.
"As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren't under human control."
Freeman's right: these things are coming either way. The only question is whose competitors get regulated out of the petri dish.
