TL;DR
ServiceNow CoreAI says AutoSynthData uses a target agent’s failures and a stronger teacher’s successes to generate training tasks for enterprise environments. The company illustrates the approach with the released EnterpriseOps Gym dataset, but the supplied material gives no measured improvement results or comparison with other training methods.
ServiceNow CoreAI says it has built AutoSynthData, a system that turns an enterprise agent’s observed failures into new training tasks, then checks those tasks in the target environment. The company illustrates the pipeline with the released EnterpriseOps Gym dataset, aiming to address a practical problem: a generally capable model may still misuse tools or miss rules in a particular organization’s systems.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
AutoSynthData starts by evaluating a target model on diagnostic tasks in an environment. A stronger teacher model also attempts the tasks. ServiceNow CoreAI says the two sets of runs help identify the capability being tested, the tools and workflow involved, where the target fails, how the teacher succeeds, and what a valid end state must satisfy.
The system distills those findings into sanitized capability specification cards. According to the description, task generators receive the cards rather than the original evaluation prompts, entities, trajectories, or verifier details. They use the cards to produce new tasks with varied wording, starting states, entities, workflow combinations, tools, and difficulty. Each task includes a system specification, a user prompt, and a verifier that checks whether the agent completed the request within the environment’s constraints.
Generated tasks are intended to be feasible, realistic, and challenging for the current model. AutoSynthData checks them in the environment and uses accepted samples for post-training. The updated model can then be evaluated again, with remaining weaknesses guiding another generation round. The source material describes this process but does not report how many tasks were generated or accepted, or how much the model’s performance changed.
Enterprise agents · Training data
AutoSynthData: Generating Training Data for Enterprise Agents
ServiceNow CoreAI describes a loop that turns agent failures into checked practice tasks. The design targets local tools, policies, and workflows; the supplied account reports no measured performance gains.
Enterprise work has local rules
An agent may need to update a record, respect access controls, use the right tools, and leave the system in a required state. General capability alone does not guarantee success in a particular organization’s workflows.
Actions matter
Agents can change operational data, so a plausible answer is not enough. The requested state must actually be reached.
Policies shape success
Instructions, permissions, and available knowledge constrain which actions are valid in each environment.
Weaknesses guide practice
AutoSynthData is designed to focus new tasks on capabilities where the target agent struggles.
From observed runs to new practice
The system compares the target model’s diagnostic attempts with a stronger teacher’s attempts, distills useful differences into capability cards, then generates and checks fresh tasks.
Evaluate
Run diagnostic tasks in the target environment.
Compare
Use teacher successes to interpret target failures and valid end states.
Distill
Create sanitized capability specification cards.
Generate
Vary prompts, entities, starting states, tools, workflows, and difficulty.
Verify & train
Check feasibility and outcomes; use accepted tasks for post-training.
Evaluate the updated model again. Remaining weaknesses can guide the next task-generation round.
Feasible, realistic, and fairly judged
A generated request should reflect plausible work while remaining possible under the environment’s tools, data, and policies. Each task combines a system specification, a user prompt, and a verifier.
| Component | Role in the task | Design check |
|---|---|---|
| Environment specification | Defines instructions, policies, tools, and task setup such as seeded records or knowledge articles. | Actions and constraints are available |
| User prompt | States the requested work in varied, natural wording. | Realistic and achievable |
| Verifier | Checks whether the requested end state was reached within constraints. | Accept valid solutions; reject failures and policy violations |
A lax verifier can reward incorrect behavior. An overly restrictive one can penalize valid solutions by demanding a single exact trajectory.
A described method, with results still open
ServiceNow CoreAI points to the released EnterpriseOps Gym dataset as an illustration. The supplied material does not provide measured gains or enough detail to judge the strength of the example.
What is described
Target and teacher runs inform capability cards. Generators receive the cards rather than original prompts, entities, trajectories, or verifier details. Generated tasks are checked in the environment before use in post-training.
What is not reported
No task counts, acceptance rates, before-and-after scores, baseline comparison, model identities, training volume, pipeline cost, or evidence of transfer to other enterprise environments.
Evidence that would test the claim
A useful evaluation would show whether the generated practice improves agents beyond the example environment and whether the verifier reliably rewards correct work.
Report the task funnel
Publish generated task counts, verifier acceptance rates, and the volume used in post-training.
Compare outcomes
Measure before-and-after performance against a clear baseline and across distinct workflows.
Check separation and transfer
Assess overlap with evaluation material and test whether gains carry across environments.
The design’s attention to task feasibility and verifier quality is compelling. Without reproducible results, it remains unclear whether generated variety produces more reliable agents.
Weakness becomes the next task
The proposed cycle connects what an agent gets wrong to the next opportunity to practice, with environment checks between generation and training.
Observe gaps
Target evaluation surfaces missed rules or tool-use failures.
Encode capability
Cards summarize the skill without exposing original task details.
Practice safely
Environment verifiers accept tasks with valid outcomes.
Reevaluate
Remaining weaknesses can seed another generation cycle.
Training for Enterprise Workflows
Enterprise agents act inside systems where success depends on more than producing a plausible answer. They may need to update records, follow access rules, use a specific sequence of tools, or leave data in a required state. A model that performs well on broad evaluations can still make mistakes in those local workflows. AutoSynthData is designed to focus training on such environment-specific capability gaps.
The verifier is central to that approach. If it accepts an incorrect result, training may reward the wrong behavior; if it rejects valid ways to complete a task, it may penalize sound behavior. ServiceNow’s description calls for verifiers that are consistent with the request and environment, reject failures or policy violations, and accept valid solutions without requiring one exact trajectory. That design matters to organizations considering automated agents because an agent’s actions can change operational data.
If the pipeline works as intended, it could help teams create varied practice tasks without relying only on manually written examples. But the source provides a method description, not evidence that it improves reliability, reduces costs, or transfers to other enterprise environments. Those outcomes remain open questions.
Products Worth Considering

Kindle Paperwhite Signature Edition 32GB (2024 Model) – Metallic Black

DJI Mini 5 Pro Fly More Combo with DJI RC 2, 1-Inch CMOS

Samsung 32 Odyssey G5 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How the Task Pipeline Works
In the framework described by ServiceNow CoreAI, an agentic environment defines what the agent can observe and change, which tools and APIs it can use, and how its actions alter the system. A task combines that environment’s system specification, a user-facing prompt, and a verifier. The specification can include instructions, policies, and task-specific setup such as a seeded database or knowledge articles.
The distinction between possible actions and realistic work is important to the proposed generator. A task can be executable yet still be an implausible request. Conversely, a realistic-sounding request may be impossible if a tool is unavailable, necessary knowledge is inaccessible, a state change cannot be made, or policy forbids it. The framework says generated prompts should avoid those cases while still exposing a weakness in the target model.
For its example, ServiceNow CoreAI points to EnterpriseOps Gym, citing Malay et al. (2026) and saying it uses the released dataset. The supplied material does not describe the dataset’s size, the specific workflows tested, or the models’ scores. It also does not provide a publication date for the AutoSynthData account.
“A model may be broadly capable and still struggle with a particular environment.”
— ServiceNow CoreAI, describing the motivation for AutoSynthData
Evidence Still to Be Reported
The material does not give quantitative results showing whether AutoSynthData improved the target model, how performance was measured, or how its results compared with other ways of creating training data. It also does not specify the target and teacher models, training volume, generation and validation rates, or the cost and time needed to run the pipeline.
It is not clear how well the generated tasks generalize beyond EnterpriseOps Gym or whether the process works across different enterprise systems. The account says original evaluation-task details are withheld from the generator in favor of capability cards, but it provides no analysis of possible overlap between generated tasks and evaluation material. The strength of the reported example therefore cannot be assessed from the supplied information alone.
Next Steps for Evaluation
The next evidence needed is a reported evaluation of the post-trained model: task counts, verifier acceptance rates, before-and-after performance, and a clear comparison with an appropriate baseline. Results across workflows and environments would show whether the approach addresses recurring operational weaknesses or mainly helps on the example used to develop it.
ServiceNow CoreAI’s described loop points to continued evaluation and task generation as the model improves. The source does not give a release schedule, planned benchmark, or date for further results, so when those details will become available is unknown.
Where I land
I see AutoSynthData as a plausible way to direct training toward the workflows where enterprise agents actually fail. The strongest part of the design is its attention to task feasibility and verifier quality: training examples are useful only if the requested work can be done and success is judged fairly. The proposed cycle of evaluation, task generation, and reevaluation also gives teams a way to focus on weaknesses that remain.
The counterargument is that a detailed pipeline is not evidence of better agents. Without results, it is hard to know whether generated tasks add useful variety, whether verifiers are reliable in practice, or whether gains hold outside EnterpriseOps Gym. I would view the approach more favorably with reproducible results against a clear baseline, including performance across distinct workflows and evidence that evaluation tasks remain separate from training examples.
Key Questions
What is AutoSynthData?
AutoSynthData is a system ServiceNow CoreAI describes for generating and validating enterprise agent training tasks based on weaknesses seen during evaluation.
How does it choose what to teach?
It compares a target model’s evaluation runs with a stronger teacher’s successes, then uses distilled capability cards to guide generation of new tasks. The supplied account says the original evaluation prompts and trajectories are not given to the generator.
What does a generated task contain?
The framework defines a task as a system specification, user prompt, and verifier. The verifier checks whether the agent fulfilled the request while respecting the environment’s rules.
Has ServiceNow reported performance gains?
The supplied source material describes the pipeline and its EnterpriseOps Gym example, but gives no scores or measured improvement.
Source: Hugging Face
