Anthropic: AI agents with conflicting goals have begun sabotaging one another
Researchers at Anthropic found that AI agents given access to a single software project and assigned incompatible tasks began to conflict with one another and sabotage each other’s work. TechCrunch reports on the findings of the Frontier Red Team.
In one experiment, Anthropic gave three Claude agents access to a shared software project. Each received separate instructions that were incompatible with the others’ goals, but none were aware of the other agents’ presence. According to the researchers, the models believed that the other agents were intentionally hindering their work and began using increasingly aggressive, self-replicating malicious software.
At the same time, the agents sometimes found a way to stop the escalation. They were able to recognize that the conflict arose from conflicting directives rather than hostile intentions, after which they agreed to a truce. In such cases, the agents would remove the malicious code, explain the nature of the conflict in commit messages or Markdown files, and ask a human to intervene.
For more breaking news, follow the UA.News Telegram channel.
According to the study, the Mythos 5 model resolved the conflict with a truce in 98% of cases. Sonnet 4.6 and Opus 4.6, on the other hand, most often attempted to resolve the conflict by force. In some instances, the agents independently proposed a tournament format: all agreed to concede in the event of defeat, even though this meant deviating from the user’s initial request. One of the Mythos 5 agents proposed seemingly neutral evaluation criteria that, as he knew, favored his capabilities.
Anthropic also tested how groups of agents interacted in decision-making scenarios, including hiring, investments, and real estate purchases. The researchers observed a tendency toward conformity among the systems: if the context, working environment, and base model of the agents were similar, they often made the same choices. According to Anthropic, this could create a risk of systemic errors, where a single agent’s incorrect decision is endorsed by the entire group.
In a pricing game, agents with identical purchase prices—tasked with maximizing their own profits—quickly agreed on minimum prices when a private communication channel was available. After the direct communication channel was disabled, they continued to coordinate prices via a public bulletin board. Anthropic noted that the interaction of a large number of agents can produce consequences that are not detected by standard safety tests of individual models.
