TL;DR
ALTK-Evolve’s developers report that their agent-memory system matched or exceeded ACE on AppWorld while using far fewer inference tokens. The results come from the team’s own evaluation and have not been independently verified in the supplied material.
The developers of ALTK-Evolve reported that their agent-memory method matched or beat ACE on the AppWorld benchmark while using 59% to 85% fewer inference tokens per task. The comparison suggests that retrieving only the lessons suited to a task can cut the cost of memory-assisted agents, although the supplied results come from an in-house evaluation and have not been independently verified.
Both systems let an agent learn from its own completed trajectories without changing model weights or relying on human labels. They retain detailed lessons rather than reducing experience to a short summary. ACE places lessons in one evolving playbook with helpful and harmful counters, while ALTK-Evolve stores separately retrievable guidelines backed by support counts and links to source trajectories.
The main difference is how those lessons reach the model. According to the article, ACE supplies its full playbook at every agent step. ALTK-Evolve can provide a small fixed core, add task-specific guidelines chosen through similarity or model-guided selection, or send the full consolidated set when the underlying model can use a larger prompt effectively.
Using the same base ReAct agent on AppWorld, the authors reported 89.3 TGC and 80.4 SGC for ALTK-Evolve with DeepSeek-V3.2, compared with 80.4 and 73.2 for ACE. Token use was 263,000 per task for ALTK-Evolve and 634,000 for ACE. With gpt-oss-120b, the reported scores were 56.0 and 37.5 versus ACE’s 54.8 and 35.7, while token use fell from 777,000 to 116,000 per task.
Thinking Of ACE? We Can Do It With Fewer Tokens
ALTK-Evolve’s developers report that selective delivery of stored lessons matched or exceeded ACE on AppWorld while cutting inference-token use by 59% to 85% per task.
Memory delivery becomes a dial
Both systems learn detailed lessons from completed trajectories without changing model weights or requiring human labels. The distinction is how much memory reaches the model—and when.
One evolving playbook
A Generator, Reflector and Curator loop updates a shared playbook, tracks helpful and harmful lessons, and removes embedding-based duplicates.
Retrievable guidelines
Lessons are clustered and merged while preserving support counts, types, causal links and provenance back to source trajectories.
Selective delivery
A fixed core can be supplemented with task-specific guidelines—or replaced by the full consolidated store when a capable model can use it effectively.
Higher scores, smaller prompts
The authors used the same base ReAct agent on AppWorld. TGC and SGC are named in the supplied excerpt, but their full definitions were not provided.
| Base model | System | TGC | SGC | Tokens per task | Reading |
|---|---|---|---|---|---|
| DeepSeek-V3.2 | ALTK-Evolve | 89.3 | 80.4 | 263,000 | ✓ Best reported scores |
| DeepSeek-V3.2 | ACE | 80.4 | 73.2 | 634,000 | ✗ Higher token use |
| gpt-oss-120b | ALTK-Evolve | 56.0 | 37.5 | 116,000 | ~ Accuracy treated as tie |
| gpt-oss-120b | ACE | 54.8 | 35.7 | 777,000 | ✗ Largest token total |
The serving-cost gap
Selective retrieval reduces the amount of stored guidance repeatedly inserted into the context. The figures below are reported inference tokens per task.
DeepSeek-V3.2
Full consolidated ALTK store versus ACE playbook
gpt-oss-120b
Selective task-specific retrieval versus ACE playbook
From failure to reusable guidance
Detailed lessons can address tool-use failures such as broken pagination, selecting the wrong person or returning a value when no output was requested.
Observe a trajectory
Capture the agent’s successful and failed multi-step actions.
Extract a lesson
Convert the failure, recovery or optimization into reusable guidance.
Merge with evidence
Cluster related guidelines while retaining support counts and provenance.
Deliver what fits
Retrieve relevant guidance or provide the full store according to the model.
“Count them, don’t collapse them.”
ALTK-Evolve authors / Hugging Face“We treat delivery as a dial, not a constant.”
ALTK-Evolve authors“Close enough that we call it a tie on accuracy.”
On the gpt-oss-120b resultPromising—not independently confirmed
The reported efficiency is meaningful if it generalizes, but the supplied material leaves important experimental details unresolved.
What is missing
- Independent replication of the AppWorld results
- Statistical confidence intervals and complete run counts
- Matched hyperparameter-search details for both systems
- Full accounting of learning and inference model calls
- Complete definitions for the named TGC and SGC metrics
What replication should test
- Matched agents, budgets and evaluation settings
- Variance across repeated runs and additional task suites
- Memory creation, consolidation and update costs
- Retrieval latency as the lesson store grows
- When selective retrieval beats full-memory delivery
Read the numbers as developer-reported evidence from two models on one benchmark—not as proof that ALTK-Evolve will preserve the same advantage across production workloads or longer-lived memory stores.
The practical takeaway
Memory-assisted agents do not necessarily need to carry their entire history into every step. Retrieval strategy can be tuned to the task and the underlying model.
What are ACE and ALTK-Evolve?
Agent-memory methods that extract lessons from past trajectories and reuse them on later tasks without weight updates or human labels.
Why can ALTK-Evolve use fewer tokens?
It can retrieve a compact set of relevant guidelines instead of repeatedly sending the complete memory playbook at every agent step.
Did it clearly win on accuracy?
The reported aggregate scores were higher, but the authors treated the gpt-oss-120b comparison as an accuracy tie after a repeat matched ACE.
Is selective retrieval always best?
No. The authors used retrieval for gpt-oss-120b but supplied the full consolidated store to DeepSeek-V3.2, making delivery model-dependent.
Selective Memory Lowers Serving Costs
Agent-memory systems can improve reliability on multi-step work, but repeatedly placing a large memory store in the prompt can raise inference costs and crowd the model’s working context. If the reported reductions hold across other tests, task-specific retrieval could make learning agents cheaper to operate without requiring model retraining.
The model-level split also matters. The authors used selective retrieval for gpt-oss-120b but supplied the full consolidated store to DeepSeek-V3.2. Their interpretation is that weaker models may be distracted by excess guidance, while stronger models can make use of more stored lessons. That makes memory delivery a model-dependent configuration choice, not a single fixed recipe.
AI inference token optimization tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two Methods Preserve Detailed Lessons
ACE and ALTK-Evolve address failures that arise even when an agent already knows the available tools. The source cites examples such as incorrect API pagination, selecting the wrong person and returning a value when none was requested. Both methods turn such failures into reusable instructions for later runs.
ACE builds its playbook through a Generator, Reflector and Curator loop, with incremental updates and embedding-based duplicate removal. ALTK-Evolve clusters similar lessons and merges them while retaining their combined support count. It also labels guidelines by type, including strategy, recovery and optimization, and records their causal links and provenance. The authors argue that subtask-level lessons can transfer between applications.
“Count them, don’t collapse them.”
— The ALTK-Evolve authors, writing on Hugging Face
As an affiliate, we earn on qualifying purchases.
Independent Benchmark Evidence Is Missing
The supplied material does not include independent replication, statistical confidence intervals, complete run counts or a full accounting of how configurations were selected. It is also unclear whether both systems received an equally broad hyperparameter search or whether token totals include every model call used during learning and inference.
The article names TGC and SGC but does not fully define both metrics in the supplied excerpt. Results are limited to AppWorld and two models, so they do not establish that ALTK-Evolve will retain the same cost advantage on other benchmarks, production workloads or longer-lived memory stores.
As an affiliate, we earn on qualifying purchases.
Replication Must Test Retrieval Tradeoffs
The next test is whether outside researchers can reproduce the reported accuracy and token-use figures using matched agents, budgets and evaluation settings. Tests across more models and task suites could show when selective retrieval helps and when a full memory store performs better.
Further reporting should also disclose variance across repeated runs, the cost of creating and updating each memory store, retrieval latency and performance as the number of lessons grows. Those measurements would clarify whether the lower serving bill offsets any extra work required to build and search the guidelines.
As an affiliate, we earn on qualifying purchases.
Key Questions
What are ACE and ALTK-Evolve?
They are agent-memory methods that extract lessons from an agent’s past trajectories and supply those lessons during later tasks. Neither method requires weight updates or human labels, according to the source.
How does ALTK-Evolve use fewer tokens?
It can retrieve a small set of relevant guidelines for each task instead of sending the entire memory playbook at every step. The authors identify this delivery choice as the main source of the reported savings.
Did ALTK-Evolve clearly beat ACE on accuracy?
It posted higher reported aggregate scores in both comparisons, but the authors treated the gpt-oss-120b result as an accuracy tie because a repeat run matched ACE. The DeepSeek-V3.2 advantage was larger in the reported test.
Are the benchmark findings independently confirmed?
No independent validation is included in the supplied material. The figures should be read as results reported by ALTK-Evolve’s developers from their own AppWorld evaluation.
Source: Hugging Face
