Paper · Training data
AutoEnvScaling
AutoEnvScaling: Automating the Data Flywheel with Terminal AgentsA framework that treats environment design as a terminal task, using agents to automatically generate and scale verifiable training environments for reinforcement learning.
The loop
A proposer agent builds new reinforcement learning environments from a solver model's rollouts. These environments are used to train the solver, which improves its performance on terminal tasks and generates better rollouts for the next round of environment design.
The loop
AutoEnvScaling- Proposer agent builds environments
- Solver generates rollouts
- Rollouts guide next environments
- Solver trains on new environments
- Proposer agent builds environments
- Solver generates rollouts
- Rollouts guide next environments
- Solver trains on new environments
↻ The improved system does the next round, and the loop turns again.
Why it is a road to recursion
Automating environment generation removes the human bottleneck in creating verifiable training data for reinforcement learning.
Evidence
improves its Terminal-Bench 2.1 score by 4.3 points over the matched Tmax baseline
terminal agentsenvironment generationreinforcement learning
Related loops
More training data →
Nemotron-4 340B
- Aligned model generates training data
- Reward model judges and filters it
- Data trains next intermediate instruct model
- Next model generates better alignment data
- repeat
Training dataNVIDIA · 2024
AdvSim2Real
- Simulator co-evolves tasks and adversaries
- Generates challenging web agent scenarios
- Agent trains on resulting curriculum
- Agent handles harder tasks and attacks
- repeat
Training dataSarim Hashmi et al. · 2026
Recursive Self-Improvement in Unified Multimodal Models
- UMM generates images and programs
- Execution verifies programs against specifications
- Verified renders and programs form training data
- Data trains UMM visual and text capabilities
- repeat
Training dataHuijuan Wang et al. · 2026