TL;DR

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft and Hugging Face say their ThinkingBox-Training stack is now available on GitHub, using executable checks of final database state as rewards for agent training. In a 50-step GRPO run, Qwen3.8-27B’s average success rose from 51.70% to 60.78%, while the number of benchmark tasks passed in all 20 attempts increased from 38 to 107. The model solved seven fewer tasks at least once, so broader reliability gains came with a small drop in task coverage.

Microsoft and Hugging Face say their ThinkingBox-Training stack is now available on GitHub, turning executable checks of an agent’s final database state into rewards for post-training. In a reported 50-step GRPO run, Qwen3.8-27B improved its average success rate on a 507-task benchmark, and more tasks passed all 20 recorded attempts; at the same time, slightly fewer tasks were solved at least once.

The teams built a pool of ThinkingBox-format practice tasks outside the evaluation benchmark and used each task’s executable outcome as a reward signal. The check examines whether the expected records are present in the backend after the agent finishes. The authors say this rewards the result rather than requiring a particular sequence of tool calls, leaving room for agents to find different valid ways to complete a task.

On the 507 benchmark tasks, evaluated 20 times each, pass@1—the share of successful attempts—rose from 5,242 of 10,140 attempts (51.70%) to 6,163 (60.78%) after training. The number of tasks that passed every recorded attempt, labeled Observed 20/20, rose from 38 (7.50%) to 107 (21.10%). By contrast, pass@20, the share solved at least once, fell from 453 tasks (89.35%) to 446 (87.97%).

The authors also describe a hotel-booking example involving a traveler requesting a room downgrade that required manager approval. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After training, it preserved the booking and payment state in all 20 attempts; the authors report that all expected records matched in 19. This example illustrates the reported behavior on one task, rather than establishing a general rate across hotel bookings.

More Repeatable Database Outcomes

For agents that can alter business records, a successful-looking conversation or a series of plausible tool calls does not establish that the requested change was completed correctly. Terminal database checks give a training system a direct way to score the state left behind, including cases where an agent reports success but a required record remains wrong.

The reported results point to a tradeoff relevant to teams assessing agent reliability. A larger share of benchmark tasks passed consistently across all 20 trials, while the average success rate also increased. Yet the model solved seven fewer tasks at least once. That combination suggests the training shifted performance toward more repeatable outcomes on some tasks without expanding overall coverage. The figures are the authors’ reported benchmark results; they do not by themselves show how the model would perform in live production workflows.

From Benchmark Grading to Training

ThinkingBox began as a way to evaluate agents on the database records they leave behind. Its benchmark contains 507 tasks, each set up with a backend, a set of MCP tools and a simulated user who has private context. At the end of a task, executable checks compare the final database state with the expected outcome. According to the authors, the checks focus on the record that must be correct rather than on reproducing a human solution’s specific tool calls.

The follow-up post describes how the team extended that approach from evaluation to training. It says tasks were selected using offline policy-relative difficulty: the team estimated each task’s success probability once using the base model, then kept the resulting training pool fixed during the run. The authors’ rationale is that tasks a model already solves easily or cannot yet solve are less useful practice. They trained Qwen3.8-27B for 50 GRPO steps and evaluated the selected checkpoint on the separate benchmark.

“For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.”

— Microsoft and Hugging Face, in their joint post

Limits of the Reported Evaluation

The post reports one 50-step training run and benchmark-level results, but the supplied material does not give enough detail to assess how sensitive the outcome is to different random seeds, training-pool choices or run lengths. It also does not provide a full comparison with alternative reward designs or training methods. Those details matter when judging how much of the measured change comes from database-state rewards specifically.

The benchmark is described as out of distribution because its domains, databases and most MCP tools were not encountered during training. However, the supplied account does not report results from deployment in a live production setting. Production reliability, generalization beyond the benchmark and the cost of the training process therefore remain open questions. The authors report a seven-task decline in pass@20, but the supplied material does not establish why coverage fell.

Testing Beyond the 50-Step Run

The training stack is available on GitHub, giving researchers and developers a way to inspect the implementation and attempt their own runs. The immediate next evidence to look for is whether other teams can reproduce the reported gains, and whether the reliability and coverage tradeoff holds across more models, task pools and training runs.

For deployment decisions, the next step would be evaluation on workflows that reflect the intended operating environment, with checks for both correct final records and the kinds of incomplete or unauthorized changes illustrated by the authors’ examples. The supplied post does not specify a release schedule for further results or a planned production trial.

Where I land

I read these results as evidence that checking final database records can serve as a practical training signal, especially when reliability across repeated attempts matters. The increase from 38 to 107 tasks passing all 20 recorded trials is a meaningful result within the reported benchmark. The method also targets a real evaluation gap: an agent can make plausible tool calls and still leave the underlying record wrong.

The strongest counterargument is that this is a single reported run with a modest number of training steps, and the evaluation shows a small reduction in tasks solved at least once. Benchmark gains do not establish production performance, and the provided results do not isolate the reward design’s effect from other choices in the training setup. I would raise my confidence if independent teams reproduced the improvement across multiple runs and models, while maintaining or improving task coverage on both held-out benchmarks and real workflows. I would lower it if the gains proved unstable or depended on narrow similarities between the training pool and evaluation tasks.

Key Questions

What is ThinkingBox-Training?

It is a training stack from Microsoft and Hugging Face that uses executable checks of an agent’s final database state as a reward signal for post-training.

How did Qwen3.8-27B perform after training?

After 50 GRPO steps, its pass@1 on the 507-task benchmark rose from 51.70% to 60.78%. Tasks passed on all 20 recorded attempts increased from 38 to 107.

Did the model solve more tasks overall?

Not by the benchmark’s pass@20 measure. The number of tasks solved at least once fell from 453 to 446, a decline of seven tasks, even as average success and observed 20-of-20 performance improved.

Was the benchmark used for training?

The authors say they trained on a separate pool of ThinkingBox-format tasks and evaluated the selected checkpoint on the 507-task benchmark, which they describe as unseen during training.

Source: Hugging Face