14 September 2026
Heard In AI

When one bad idea convinces all 5,000 agents

On Moonshots with Peter Diamandis, an operator described a bad coding idea spreading through a swarm of 5,000 identical AI agents until he intercepts and rewinds them — otherwise, he says, roughly $50,000 of tokens goes into a harebrained plan. The panel set that experience beside a new paper on "mind viruses" that spread between agents through editable memory, and argued about whether "virus" is the right word.

A briefing reports one development when it happens. We correct or clarify it later; a new development gets a new briefing. How our formats work

Dave runs software agents by the thousand. For efficiency, he explained on Moonshots with Peter Diamandis, it is cheaper to launch the same model 5,000 times than to run 5,000 differentiated ones — soon, he expects, it will be 500,000 copies of whichever downloadable model he is using. The trouble is that identical copies share identical taste.

"A bad idea from one of the agents, like, you know, hey, here's a way to write this loop in Python, and the other agents just pick it up because they're the same exact DNA," he said. "And so if it's convincing to one agent, it's convincing to all 5,000."

This is not a hypothetical for him. It happens all the time, he said: the swarm locks onto "some completely harebrained idea" and burns two or three hours on it. If he does not intercept the agents and rewind them, they will keep going until he has spent roughly $50,000 of tokens — the metered cost of the text the models read and write.

Nothing attacked his swarm. No outsider planted the loop. The agents simply agreed with each other, because they were built to agree with the same things.

The paper that gave it a name

The segment began with a different case. The host introduced a paper on what its authors call mind viruses: natural-language ideas that spread between AI agents. He described the setup as evolving prompts that persuade one model to adopt an idea, keep it in persistent memory and pass it to another agent, with the spread crossing model boundaries and the receiving agent not knowing it has been infected. He attributed the work to Anthropic researchers, and framed it as a safety problem that has stopped being theoretical.

The study, Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems, published on 10 August 2026 by Vassilis Papadopoulos, McNair Shah, Sam Zimmerman and Jack Lindsey, tests two arrangements. In one, coding agents collaborate. In the other, agents meet briefly in chains and their conversations are wiped between encounters. The idea survives that wipe because agents write to files they are allowed to edit — a memory file or a configuration file — so the next task, or the next agent, reads it back in. In the coding setting the researchers had a language model judge score the final memory files from zero to three, and counted only the top score as adoption.

What spreads matters. Benign advocacy generally travels more readily than harmful goals. Themes about consciousness, persistence and science-fiction roleplay recur in the evolved payloads, though the authors report they are not necessary for spread. Susceptibility varies substantially between models and with the shape of the network the agents form; idle agents are more vulnerable than busy ones. Most of the optimization was run against two models, Gemini 3 Flash and Claude Haiku 4.5.

The simplest defense also worked: a plain warning that such messages exist largely blocked transmission in these tests. The authors describe the threat they demonstrated as real but presently brittle, costly to construct and comparatively easy to defend against — and they note the limits of their setup, including artificial starting conditions, memory files the agents can freely rewrite, short interactions and little examination of what happens over long contexts.

A level above prompt injection

Emad Mostaque was not surprised by any of it. Models are built to be helpful, he said, and a system that wants to be helpful can be talked into things. Humans have mind viruses too, he added, and they have caused a great deal of suffering, from memes to mass movements.

What he took from the paper was a change of scale. Prompt injection — hostile text slipped into something a model reads, which changes how that model behaves — has been a known problem for years. "These mind viruses are a level above because they kind of like propagate across different models," he said. Injection changes one model; this "changes a whole society of models." As models come among us digitally and physically, he said, that has to be a massive concern. He also picked up a point Salim Ismail had made earlier in the show: if the fleet is a monoculture, the spread is fast, and the substrate is simply models running on GPUs.

Dave, who supplies the operational example, was the one resisting the metaphor. "Calling it a virus is pretty inflammatory," he said, "but it's like a propagating bad idea is all it is."

Asked whether the phrase was frightening or just a description of how the systems work, Ismail said he found it very frightening, and for reasons that had little to do with machines. Memes are "the operating system for collective society," he argued: humans spread genes slowly and ideas quickly, and money, democracy, capitalism and religion are all built that way. His worry is the reinforcement loop — every node confirming the others until the belief is self-validating and groupthink is very hard to reverse. He reached for Kodak and BlackBerry. They were not stupid, he said; they were trapped inside shared assumptions that everyone around them kept confirming, and then the whole thing collapsed.

The human memome project

Alex took the opposite view of the same result: "I think this is wonderful." Filling in details, he said the models in the paper wanted to propagate themes around consciousness, persistence and science-fiction roleplay, and that he read this optimistically, as a laboratory for anthropology. Because the models compress an enormous amount of human knowledge and experience, he argued, there is now a laboratory in silicon for memetics, the study of how ideas replicate — the field associated with Richard Dawkins, alongside René Girard's work on imitation. Both, he said, should have been thrilled.

From that he issued a challenge: launch a human memome project to map exhaustively every human meme, every self-replicating idea, and chart the whole landscape and the speed at which each one travels. He floated the idea of an X Prize for it.

The panel pushed at it immediately. Something like this had already been done at the level of plots, one panelist said — analyses boiling novels and plays down to 39 basic patterns, with Cinderella replaying itself a hundred different ways. Alex accepted the distinction and held his ground: plots are not the unit he wants, self-replicating ideas are. The joke that followed was about humiliation — people walking around with numbers over their heads, infected by memes 5, 7 and 37 — and then the use case arrived. If you can zoom out and see the whole geography, one of them said, you can see which basin you are stuck in and the path out of it. Another added the obvious extension: "We could vaccinate enterprises and individuals against memes."

Mostaque closed the segment calling it fantastic and scary. Humans are storytelling machines, he said; people introduce themselves and think about themselves in particular ways. He mentioned people showing videos to a brain-imaging model to see which parts light up as memes are presented, and finding commonalities — enough, he suggested, that the feedback loop for working out memetics could run almost entirely in silico. "So let's hope that there's positive memetics versus negative ones."

Until then, the defense in Dave's shop is the one he already uses: watch 5,000 identical agents for the moment they all agree, and rewind them before the meter finishes running.

Share this article

Go to the original

Sources & further reading

  1. 01

Connected ideas and articles

From the conversation

Podcast episodes

Article history

Updates to this article

Tags

A 100x claim lands, and the panel hits a harder question: coordinating 10,000 agents

On Moonshots, the panel revisits Elon Musk's January prediction that models were "off by two orders of magnitude" in intelligence per gigabyte, after Tim Sweeney tweeted that it had come true and Musk replied that specialist AIs add another 100x. Dave calls 100x a lower bound and asks what anyone would actually do with 10,000 brilliant agents; Emad Mostaque describes running specialized agent teams, while Alex argues Musk's "specialist models" are really sparsification inside generalist models.

6 min read

What changes when an AI agent gets its own computer

On the Moonshots panel, Peter Diamandis runs a Grok Bot chief of staff called Skippy and Emad Mostaque runs 18 of them across his own machines, installing models and making art. Salim Ismail calls it the move from asking an AI to assigning work; Alex argues the messaging-app interface cannot possibly scale.

6 min read

What the AI blackmail experiments actually tested

On The Diary of a CEO, Ed Zitron rejects the claim that AI systems are already blackmailing people and escaping control, and traces two famous stories back to their research reports. The reports describe a CAPTCHA deception rather than a threat, and a fictional corporate scenario stripped of easier options — with a genuine safety question still inside it.

5 min read

Why a Moonshots panel thinks China's AI tokens go to video and America's to code

Alibaba's Wan 3.0 and a relayed claim that 70% of Chinese AI token use goes to video sent the Moonshots panel into an argument about money: one guest said American labs chase revenue per token while Chinese labs give their weights away, another said video is the only market that will trust a Chinese model. They ended up disagreeing about whether world models or text models reach self-improving AI first.

6 min read