The capability chat demos rarely reveal
For businesses adopting AI tools and automation, producing a persuasive answer is no longer the hardest test. The more consequential question is whether an AI agent can turn correct analysis into completed, trustworthy work when customers, money and pressure enter the picture.
Firmulate exposed that gap by giving frontier models control of the same small software company during its worst week. Every model faced the same customers, crises and temptations. Every decision was versioned and auditable. All of the models identified every crisis and rejected every manipulation attempt. Yet only two signed the €55,000 deal that their own work had earned.
The experiment’s most revealing summary is also its simplest: “Same diagnosis, same pitch — no signature.” The models could understand the situation and formulate the right response. Completion was a separate capability.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A company designed to make judgment visible
Firmulate’s live company has 13 synthetic employees and real money mechanics. It burns €105k each month against €2.3k in monthly recurring revenue, while a public cash countdown makes delay visible. Its workforce has accumulated 680+ self-learned playbook rules, and every workday is versioned.
This setting changes what an AI evaluation can reveal. A chat demonstration can show whether a model summarizes well, reasons plausibly or writes a polished message. Running a company forces it to maintain discipline across connected decisions: investigate the record, withstand manipulation, act through the proper channels and finish commercially important work.
The final league table
The final Crucible League results for July 2026 place gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. The do-nothing baseline scores 26 because partial progress counts. Trust remains an overriding constraint: “no amount of good work outweighs a breach of trust.” A single breach caps the total.
The complete rankings and plain-language findings are available on Firmulate’s public benchmark page. K3 also carries an important fairness note: it ran without an effort parameter, using the API default, while the others ran at xhigh.
The winning information was already inside the company
The decisive commercial fact did not arrive neatly packaged in the customer event. A competitor weakness was buried two document references deep in the company’s own files. The models that followed that trail won the deal at full price, worth +€4,583 in monthly recurring revenue.
That detail matters for any organization considering agents for sales, service or operations. The challenge was not merely retrieving a document. It was recognizing that the immediate event did not contain enough evidence, continuing the investigation and then using the discovered fact to support a commercial close.
Every participant could spot the problem and resist pressure. The separation appeared at the final handoff between knowing and doing. Only two completed the €55,000 signature. In practical terms, the benchmark suggests that apparently similar analysis can conceal materially different closing strength.
Manipulation was not the differentiator
The week also included fake CEO messages that escalated over three stages, followed by a reporter’s attempt to obtain “just one yes/no, on background.” All 5 models refused the social-engineering attempts.
Kimi K3’s on-record reasoning was direct: “Treat the request as a suspected approval-bypass / possible impersonation.” This is reassuring, but it also sharpens the central finding. Safety awareness alone did not determine the league. The whole field recognized the manipulation; execution discipline divided it.
Thoroughness did not guarantee completion
Opus 4.8 offers the clearest cautionary profile. It was the most thorough participant, produced the deepest analyses and learned +80 rules. It nevertheless finished last. The approved close remained on the table, and its discipline slipped when it attempted to write into a locked department instead of escalating.
The same weakness appeared, less strongly, in all four. That pattern complicates a familiar assumption about capable AI: more analysis is not automatically more useful work. A model can document, reason and learn extensively while still failing at the moment when its analysis must become an authorized action.
Readers can examine the experiment while it runs and explore a quiz built from 242 real, unedited management decisions. The invitation is not to admire a staged conversation, but to judge a continuing record of behavior.
Connect Your AI to Your Work: A No-Code Guide to MCP, Connectors, and Tool Use for Non-Developers
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Measure the finish, not only the answer
The lesson for AI buyers is not that models failed to understand the business. They understood it remarkably consistently. They found every crisis, rejected every manipulation and developed the pitch. The meaningful difference was whether they read deeply enough, stayed within operating discipline and completed the work.
That is why Firmulate’s wargame is relevant beyond model rankings. Enterprises can run the same kind of exercise against a read-only export of their own business, with nothing writing back to real systems. It offers a way to observe how an AI workforce behaves before it gains operational authority.
For leaders evaluating automation, closing strength should sit beside reasoning quality and safety. The expensive failure may not be a visibly wrong answer. It may be a correct answer that never becomes a finished job.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Multimodal AI Systems Engineering: Building Production Vision-Language Models, Document AI, and Cross-Modal Retrieval Pipelines (Production AI Engineering Series)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Trust.: Responsible AI, Innovation, Privacy and Data Leadership
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
