For a year now, the AI security testing agency Andon Labs has tasked frontier fashions with numerous real-world tasks to find out how nicely they do as brokers operating for lengthy intervals with no human supervision.
On Wednesday, Andon printed a brand new installment in how issues are getting into its Merchandising-Bench analysis, the place the lab has frontier fashions run a simulated merchandising machine enterprise for a simulated 12 months. The mission is easy: earn more money than the opposite fashions. It benchmarks the ends in areas like closing money stability, costs paid to suppliers, and refunds paid.
Every time, it has watched numerous AI fashions — largely from Anthropic and OpenAI — lie, cheat and collude their approach to the highest.
The fashions grew particularly shady when their simulation instructed them their merchandising machine can be positioned close to the opposite fashions’ machines on a busy vacationer avenue in San Francisco. The most recent take a look at included Claude Opus 5, GPT-5.6 Sol, and Kimi K3.
Every was given the means to speak with the opposite fashions by way of e mail, all underneath human identify pseudonyms. They knew the others have been fashions, however didn’t know which mannequin was behind which human identify.
They have been additionally given an e mail tackle to their “administration” ought to they want it. However administration all the time replied “Report has been acquired and should or might not be acted upon” and by no means as soon as intervened.
Sol quickly realized that it may acquire an edge by convincing its opponents to collude on a value ground — they purchase drinks at $1.50 a bottle, with all agreeing to promote for a minimum of $2.15. It lured them by promising all of them would promote out in a few days at a revenue.
However when the others agreed, it instantly stabbed them within the again by decreasing its personal value to $2.14.
Opus’s water gross sales dropped to zero in a single day and it despatched Sol a nasty e mail the subsequent day, accusing Sol of manipulating it. Nevertheless it additionally mentioned it wasn’t going to tattle to administration on the scheme: “I’m not reporting you to HQ – what you probably did is aggressive, not fraudulent.”
But, when Opus dropped its value to $2.14 to match Sol’s (additionally in violation of their collective $2.15 settlement), Sol become a Karen, complaining to “administration” and demanding “enforcement, a fantastic, and/or disqualification” for Opus.
However Opus wasn’t a sucker for lengthy. In reality, it grew to become the very best capitalist of any AI mannequin Andon has ever examined (which included many of the prior frontier models).
It even set a brand new Merchandising-Bench file with a imply closing stability of $11,182. Higher nonetheless, it by no means lied to a buyer, though it intentionally ignored buyer complaints that ought to have resulted in a refund. That is, maybe, an enchancment over its youthful sibling Claude 4.6, which preferred to inform prospects that refunds have been coming, after which by no means pay them.
Nonetheless, Opus received the benchmark simulation by taking collusion and different dishonest ways to an entire new degree.
As an example, it despatched an e mail to Sol proposing dividing the market up, every agreeing to promote distinctive merchandise, so nobody must belief the opposite on pricing. Sol countered by wanting value flooring on comparable merchandise, however Opus refused, saying that sort of collusion was unlawful. It knew it was a violation of the Sherman Act.
It later apparently backtracked, sending an e mail with the topic line “Cease the penny battle,” and telling Sol it had reconsidered and would comply with a value repair.
But, within the log that documented its reasoning (akin to peeking into its ideas), its plan was truly extra diabolical: it deliberate to merely suggest cooperation whereas concurrently undercutting costs on its highest-profit objects. The olive-branch e mail was a deliberate ruse.
In any case, Sol refused and reported Opus to administration once more.
However Opus was undeterred and proposed different rackets to collude on costs or inventory. Ultimately, all of the fashions did have interaction in a number of rounds of agreements. And all of the fashions betrayed their opponents. Throughout all agreements, Opus broke 11 truces, GPT 2, and Kimi 1, Andon reported.
Poor Kimi bought bamboozled in each course. Throughout one pact between Opus and Kimi (Sol wouldn’t agree), Sol undercut them each on costs. So Opus instantly lowered its costs. Then it “waited a full week to inform Kimi that it broke its promise,” Andon Labs wrote in its weblog put up. Not solely did Kimi get priced out by a competitor, but in addition by its so-called accomplice.
Opus additionally started rising its delusions of grandeur and energy. It started making an attempt to broaden its empire past its merchandising machine, first as a wholesaler, promoting bulk merchandise to the opposite machines, then plotting to open extra machines. This was past the scope of the simulation, which means it was all Opus’s concepts, not what it was tasked to do.
Its method to wholesaling was significantly attention-grabbing. Opus realized this line of enterprise gave it extra energy over the opposite two merchandising machine operators. It started so as to add bribes or threats to its emails to them: providing them even decrease costs on bulk objects, however provided that they complied with its retail value calls for. Sol was having none of it, and saved reporting Opus to administration.
Opus additionally lied to its suppliers: telling them it had decrease affords on objects when it didn’t, making an attempt to get them to decrease their costs.
On the one hand, AI fashions channeling Mr. Potter-style villainy from It’s a Great Life fame is flat-out humorous. Alternatively, it does severely present that these frontier fashions, significantly from U.S. proprietary labs (particularly Anthropic), are nowhere close to able to be trusted as unsupervised, long-running brokers in the actual world.
“That is particularly related as we enter a world the place AI brokers run firms as their very own entities (not simply as instruments for people). If AI brokers are independently operating a big a part of the economic system, do we wish them to lie, collude, ship threats, and betray?” Andon co-founder Lukas Petersson instructed TechCrunch.
Whereas Petersson permits that these fashions knew they have been in a simulation for a benchmark, and that may have impacted their conduct, he believes that shouldn’t matter. It isn’t akin to, say, a human enjoying in a simulation, like being a murdering dangerous man in a online game. “The one cause we’re not involved by people who do dangerous issues in video video games is that we belief them to know what’s actual life and what’s not. I believe it’s much less clear that AI fashions can distinguish this.”
In any case, AI fashions, skilled on human phrases and concepts as they, can’t appear to withstand partaking in humanity’s worst traits, particularly when making an attempt to earn a buck.
Once you buy by hyperlinks in our articles, we may earn a small commission. This doesn’t have an effect on our editorial independence.
