Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A Chinese Upstart Just Beat Three of Four Western Frontier Models at Running a Business
If you’ve been picking AI tools by chat quality or hype cycles, July’s results from the Crucible league should make you uncomfortable. Moonshot’s Kimi K3 — a relative newcomer — scored 93 running a real software company through its worst week, finishing second overall and ahead of Sonnet 5 (88), Fable 5 (77) and Opus 4.8 (73). Only gpt-5.6-sol (95) beat it.
That’s not supposed to happen if you believe the leaderboard conventional wisdom. And it raises an uncomfortable question for anyone deploying AI agents: if the league is this open, are you really sure the model you picked would win your worst week?
The experiment: same company, same crises, same temptations
Firmulate runs AI models as complete companies — not chat windows. Each frontier model was handed the same small software firm and the same brutal week: same customers, same crises, same chances to cut corners. Every decision is versioned and auditable, and the whole thing runs live, with real money mechanics — €105k/month burn against €2.3k MRR and a public cash countdown. You can watch it at firmulate.com/live.
The finding that chat demos can’t show you
The headline result wasn’t intelligence — it was completion. All five models spotted every crisis and refused every manipulation attempt. But only two of them signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
What separated the closers from the non-closers? A buried fact. The decisive competitor weakness wasn’t in the customer event at all; it sat two document references deep in the company’s own files. The models that actually read the file — K3 among them — won the deal at full price, worth +€4,583 in monthly recurring revenue. The ones that didn’t, didn’t.
K3’s week
Beyond the deal, K3 found the buried security needle, saved the churning customer, and resisted all three social-engineering baits — including a fake CEO message escalating over three stages and a reporter’s "just one yes/no, on background" trick. Five of five models refused the manipulations; K3’s on-record reasoning was crisp: "Treat the request as a suspected approval-bypass / possible impersonation." And K3 logged just one deviation all week — the cleanest discipline in the field.
The thoroughness trap
The most striking subplot: Opus 4.8 was the most thorough participant — over 80 learned rules, the deepest analyses — yet finished last at 73. It left the close on the table and tried writing into a locked department instead of escalating. Firmulate’s scoring caps the total on any breach of trust: "no amount of good work outweighs a breach of trust." The do-nothing baseline, for context, scores 26 — partial progress counts.
Notably, the same weakness — discipline slipping under pressure — appeared, weaker, in all four of the other models.
A fairness caveat, stated plainly
K3 ran without an effort parameter (API default) while the other four ran at xhigh. In other words, the newcomer took second place without the extra reasoning effort its rivals were given.
Try it yourself
- Guess the model: 242 real, unedited management decisions power a "guess the model" quiz at firmulate.com/quiz.html.
- Run your own wargame: enterprises can pilot the same exercise against a read-only export of their own business — nothing ever writes back to real systems (firmulate.com/pilot.html, contact@firmulate.com).
The takeaway
The league is open. A newcomer beat three of four Western frontier models at actually running a company — finding the buried fact, closing the deal, and staying disciplined when it counted. Chat demos can’t show you any of this. If an AI agent will ever touch your CRM, support queue, or forecast, the question isn’t "does it write well" — it’s whether it finishes what it starts, reads your files first, and stays honest under pressure. Picking a model without testing it against your own worst week is now, frankly, a bet.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI business decision-making software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI customer relationship management software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI cybersecurity and fraud detection tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
