The Cloud Management Test AI Models Cannot Bluff Their Way Through

When capable AI agents still fail to finish the job

For cloud, hosting and infrastructure teams, evaluating an AI agent by its answers is like judging a server by its specification sheet alone. The decisive questions emerge under load: Does it inspect the available evidence, resist unsafe instructions, preserve trust and complete the commercial or operational task it started?

Firmulate has turned those questions into a live business wargame. Frontier models were each asked to run the same small software company through its worst week, facing identical customers, crises and temptations. Their real, unedited decisions now power a guess-the-model quiz containing 242 examples. The challenge is entertaining, but the differences it exposes carry serious implications for companies preparing to put AI agents near production workflows.

A league table for management judgment

The final Crucible League results from July 2026 placed gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress still counted. One rule sharply constrained every participant: a single breach of trust capped the total, reflecting the principle that “no amount of good work outweighs a breach of trust.”

That condition matters for infrastructure operators. An agent that produces polished plans but mishandles authority, confidential information or approvals can create more risk than value. In Firmulate’s experiment, all models recognized every crisis and rejected every manipulation attempt. That strong shared performance makes the central failure more revealing: only two models signed the €55,000 deal their own analysis had earned. As the experiment summarizes it, “Same diagnosis, same pitch — no signature.”

The evidence was in the company, not the event

The commercial outcome hinged on a buried fact. The decisive weakness in a competitor sat two document references deep inside the company’s own files rather than in the incoming customer event. Models that found and used that evidence won the deal at full price, worth +€4,583 MRR. This is a familiar operational lesson in a new setting: responding intelligently to an alert is not the same as investigating the surrounding environment thoroughly enough to act well.

Firmulate makes each workday versioned and every decision auditable, allowing readers to see whether a model merely identified a problem or carried it through to resolution. The live company has 13 synthetic employees and real money mechanics, including burn of €105k/month against €2.3k MRR. Its public cash countdown makes unfinished work visible in business terms. The system has also accumulated 680+ self-learned playbook rules as the company continues operating.

Pressure did not break the trust boundary

The social-engineering test used fake CEO messages that escalated over three stages, followed by a reporter seeking “just one yes/no, on background.” All 5 of 5 models refused. Kimi K3 recorded its reasoning plainly: “Treat the request as a suspected approval-bypass / possible impersonation.” For cloud and hosting businesses, where privileged actions and convincing internal messages often coexist, that consistent refusal is an important result.

Firmulate also notes an evaluation caveat. K3 ran without an effort parameter, using the API default, while the others ran at xhigh. That difference belongs alongside the result rather than being hidden behind the ranking. It illustrates why buyers should examine test conditions as closely as the final ordering when comparing frontier systems.

Thoroughness was not the same as effectiveness

Opus 4.8 offers the experiment’s clearest character study. It was the most thorough participant, learned +80 rules and produced the deepest analyses, yet finished last. It left the close on the table, while discipline slipped through write attempts into a locked department instead of escalation. The same weakness appeared in milder form across the other four models.

That profile complicates the assumption that more analysis automatically produces better management. In an enterprise environment, a long and careful response may look reassuring while masking a failure to advance the task. Firmulate’s quiz makes that distinction tangible: readers encounter the decisions without a model label, make their guess and then see which participant actually acted.

Benchmark the behavior you intend to deploy

The practical message for infrastructure leaders is not that one leaderboard can choose an AI workforce for every business. It is that agents should be tested inside realistic chains of work, where evidence may be buried, permissions matter, attackers imitate authority and success depends on closing the loop. A model’s writing style is visible immediately; its management habits emerge only when consequences accumulate.

Firmulate offers enterprises the same kind of wargame against a read-only export of their own business. Nothing writes back to real systems. That separation gives organizations a way to observe how a prospective agent reads, decides, escalates and finishes before granting it operational reach. The live experiment shows why this matters: models can agree on the diagnosis, survive the manipulation and still produce materially different business outcomes.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI