AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

ServiceNow CoreAI says AutoSynthData uses a target agent’s failures and a stronger teacher’s successes to generate training tasks for enterprise environments. The company illustrates the approach with the released EnterpriseOps Gym dataset, but the supplied material gives no measured improvement results or comparison with other training methods.

ServiceNow CoreAI says it has built AutoSynthData, a system that turns an enterprise agent’s observed failures into new training tasks, then checks those tasks in the target environment. The company illustrates the pipeline with the released EnterpriseOps Gym dataset, aiming to address a practical problem: a generally capable model may still misuse tools or miss rules in a particular organization’s systems.

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

AutoSynthData starts by evaluating a target model on diagnostic tasks in an environment. A stronger teacher model also attempts the tasks. ServiceNow CoreAI says the two sets of runs help identify the capability being tested, the tools and workflow involved, where the target fails, how the teacher succeeds, and what a valid end state must satisfy.

The system distills those findings into sanitized capability specification cards. According to the description, task generators receive the cards rather than the original evaluation prompts, entities, trajectories, or verifier details. They use the cards to produce new tasks with varied wording, starting states, entities, workflow combinations, tools, and difficulty. Each task includes a system specification, a user prompt, and a verifier that checks whether the agent completed the request within the environment’s constraints.

Generated tasks are intended to be feasible, realistic, and challenging for the current model. AutoSynthData checks them in the environment and uses accepted samples for post-training. The updated model can then be evaluated again, with remaining weaknesses guiding another generation round. The source material describes this process but does not report how many tasks were generated or accepted, or how much the model’s performance changed.

At a glance
reportWhen: Described in source material citing Ent…
The developmentServiceNow CoreAI has described AutoSynthData, a pipeline for generating and checking training tasks based on weaknesses observed in enterprise agents.
AutoSynthData: Generating Training Data for Enterprise Agents

Enterprise agents · Training data

AutoSynthData: Generating Training Data for Enterprise Agents

ServiceNow CoreAI describes a loop that turns agent failures into checked practice tasks. The design targets local tools, policies, and workflows; the supplied account reports no measured performance gains.

Core inputFailureTarget agent’s weak spots
Teaching signalTeacherStronger model’s successful runs
Task checkVerifierChecks outcomes in the environment
ExampleGymEnterpriseOps Gym dataset
01 / Why this approach

Enterprise work has local rules

An agent may need to update a record, respect access controls, use the right tools, and leave the system in a required state. General capability alone does not guarantee success in a particular organization’s workflows.

Actions matter

Agents can change operational data, so a plausible answer is not enough. The requested state must actually be reached.

Policies shape success

Instructions, permissions, and available knowledge constrain which actions are valid in each environment.

Weaknesses guide practice

AutoSynthData is designed to focus new tasks on capabilities where the target agent struggles.

02 / The task pipeline

From observed runs to new practice

The system compares the target model’s diagnostic attempts with a stronger teacher’s attempts, distills useful differences into capability cards, then generates and checks fresh tasks.

01

Evaluate

Run diagnostic tasks in the target environment.

02

Compare

Use teacher successes to interpret target failures and valid end states.

03

Distill

Create sanitized capability specification cards.

04

Generate

Vary prompts, entities, starting states, tools, workflows, and difficulty.

05

Verify & train

Check feasibility and outcomes; use accepted tasks for post-training.

A new round

Evaluate the updated model again. Remaining weaknesses can guide the next task-generation round.

03 / What makes a task useful

Feasible, realistic, and fairly judged

A generated request should reflect plausible work while remaining possible under the environment’s tools, data, and policies. Each task combines a system specification, a user prompt, and a verifier.

ComponentRole in the taskDesign check
Environment specificationDefines instructions, policies, tools, and task setup such as seeded records or knowledge articles.Actions and constraints are available
User promptStates the requested work in varied, natural wording.Realistic and achievable
VerifierChecks whether the requested end state was reached within constraints.Accept valid solutions; reject failures and policy violations
Verifier balance

A lax verifier can reward incorrect behavior. An overly restrictive one can penalize valid solutions by demanding a single exact trajectory.

04 / Evidence check

A described method, with results still open

ServiceNow CoreAI points to the released EnterpriseOps Gym dataset as an illustration. The supplied material does not provide measured gains or enough detail to judge the strength of the example.

What is described

Target and teacher runs inform capability cards. Generators receive the cards rather than original prompts, entities, trajectories, or verifier details. Generated tasks are checked in the environment before use in post-training.

What is not reported

No task counts, acceptance rates, before-and-after scores, baseline comparison, model identities, training volume, pipeline cost, or evidence of transfer to other enterprise environments.

Performance change: unknown Generalization: unknown Publication date: unspecified
05 / What to measure next

Evidence that would test the claim

A useful evaluation would show whether the generated practice improves agents beyond the example environment and whether the verifier reliably rewards correct work.

Report the task funnel

Publish generated task counts, verifier acceptance rates, and the volume used in post-training.

Compare outcomes

Measure before-and-after performance against a clear baseline and across distinct workflows.

Check separation and transfer

Assess overlap with evaluation material and test whether gains carry across environments.

Current read

The design’s attention to task feasibility and verifier quality is compelling. Without reproducible results, it remains unclear whether generated variety produces more reliable agents.

Traceability / The learning loop

Weakness becomes the next task

The proposed cycle connects what an agent gets wrong to the next opportunity to practice, with environment checks between generation and training.

A

Observe gaps

Target evaluation surfaces missed rules or tool-use failures.

B

Encode capability

Cards summarize the skill without exposing original task details.

C

Practice safely

Environment verifiers accept tasks with valid outcomes.

D

Reevaluate

Remaining weaknesses can seed another generation cycle.

Training for Enterprise Workflows

Enterprise agents act inside systems where success depends on more than producing a plausible answer. They may need to update records, follow access rules, use a specific sequence of tools, or leave data in a required state. A model that performs well on broad evaluations can still make mistakes in those local workflows. AutoSynthData is designed to focus training on such environment-specific capability gaps.

The verifier is central to that approach. If it accepts an incorrect result, training may reward the wrong behavior; if it rejects valid ways to complete a task, it may penalize sound behavior. ServiceNow’s description calls for verifiers that are consistent with the request and environment, reject failures or policy violations, and accept valid solutions without requiring one exact trajectory. That design matters to organizations considering automated agents because an agent’s actions can change operational data.

If the pipeline works as intended, it could help teams create varied practice tasks without relying only on manually written examples. But the source provides a method description, not evidence that it improves reliability, reduces costs, or transfers to other enterprise environments. Those outcomes remain open questions.

Products Worth Considering

Kindle Paperwhite Signature Edition 32GB (2024 Model) - Metallic Black

Kindle Paperwhite Signature Edition 32GB (2024 Model) – Metallic Black

As an affiliate, we earn on qualifying purchases.

How the Task Pipeline Works

In the framework described by ServiceNow CoreAI, an agentic environment defines what the agent can observe and change, which tools and APIs it can use, and how its actions alter the system. A task combines that environment’s system specification, a user-facing prompt, and a verifier. The specification can include instructions, policies, and task-specific setup such as a seeded database or knowledge articles.

The distinction between possible actions and realistic work is important to the proposed generator. A task can be executable yet still be an implausible request. Conversely, a realistic-sounding request may be impossible if a tool is unavailable, necessary knowledge is inaccessible, a state change cannot be made, or policy forbids it. The framework says generated prompts should avoid those cases while still exposing a weakness in the target model.

For its example, ServiceNow CoreAI points to EnterpriseOps Gym, citing Malay et al. (2026) and saying it uses the released dataset. The supplied material does not describe the dataset’s size, the specific workflows tested, or the models’ scores. It also does not provide a publication date for the AutoSynthData account.

“A model may be broadly capable and still struggle with a particular environment.”

— ServiceNow CoreAI, describing the motivation for AutoSynthData

Evidence Still to Be Reported

The material does not give quantitative results showing whether AutoSynthData improved the target model, how performance was measured, or how its results compared with other ways of creating training data. It also does not specify the target and teacher models, training volume, generation and validation rates, or the cost and time needed to run the pipeline.

It is not clear how well the generated tasks generalize beyond EnterpriseOps Gym or whether the process works across different enterprise systems. The account says original evaluation-task details are withheld from the generator in favor of capability cards, but it provides no analysis of possible overlap between generated tasks and evaluation material. The strength of the reported example therefore cannot be assessed from the supplied information alone.

Next Steps for Evaluation

The next evidence needed is a reported evaluation of the post-trained model: task counts, verifier acceptance rates, before-and-after performance, and a clear comparison with an appropriate baseline. Results across workflows and environments would show whether the approach addresses recurring operational weaknesses or mainly helps on the example used to develop it.

ServiceNow CoreAI’s described loop points to continued evaluation and task generation as the model improves. The source does not give a release schedule, planned benchmark, or date for further results, so when those details will become available is unknown.

Where I land

I see AutoSynthData as a plausible way to direct training toward the workflows where enterprise agents actually fail. The strongest part of the design is its attention to task feasibility and verifier quality: training examples are useful only if the requested work can be done and success is judged fairly. The proposed cycle of evaluation, task generation, and reevaluation also gives teams a way to focus on weaknesses that remain.

The counterargument is that a detailed pipeline is not evidence of better agents. Without results, it is hard to know whether generated tasks add useful variety, whether verifiers are reliable in practice, or whether gains hold outside EnterpriseOps Gym. I would view the approach more favorably with reproducible results against a clear baseline, including performance across distinct workflows and evidence that evaluation tasks remain separate from training examples.

Key Questions

What is AutoSynthData?

AutoSynthData is a system ServiceNow CoreAI describes for generating and validating enterprise agent training tasks based on weaknesses seen during evaluation.

How does it choose what to teach?

It compares a target model’s evaluation runs with a stronger teacher’s successes, then uses distilled capability cards to guide generation of new tasks. The supplied account says the original evaluation prompts and trajectories are not given to the generator.

What does a generated task contain?

The framework defines a task as a system specification, user prompt, and verifier. The verifier checks whether the agent fulfilled the request while respecting the environment’s rules.

Has ServiceNow reported performance gains?

The supplied source material describes the pipeline and its EnterpriseOps Gym example, but gives no scores or measured improvement.

Source: Hugging Face

You May Also Like

Introducing Tool Retrieval In The Gemini API

Google adds preview support for on-demand tool schemas in the Gemini Interactions API, aiming to reduce prompt tokens for agents with large tool libraries.

Build A Robot Pick-and-Place Simulation With SimReady Assets

A Hugging Face walkthrough uses SimReady robot and can assets in Isaac Sim to build and check a beverage pick-and-place task.

Anthropic Bought A One-Person Biotech That Ran A Drug Program With Claude – Forbes

A headline-only report says Anthropic bought a one-person biotech whose drug program used Claude; deal terms and research details remain unclear.

Anthropic Is Cutting Claude Code’s Current Weekly Limits By 17% – BleepingComputer

Anthropic is reportedly reducing Claude Code’s weekly usage limits by 17%, but the timing, affected plans and new allowances remain unclear.