Alphabell.
Benchmark · AI R&D evals

InfraDev

InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?A benchmark evaluating coding agents on real-world pull requests to AI infrastructure repositories like Megatron-LM and vLLM.

InfraDev
Figure from the paper, arXiv. Source
The loop

Coding agents develop production-grade AI infrastructure like Megatron-LM and vLLM within the InfraDev benchmark. This improves the evaluation of infrastructure engineering, guiding the development of agents that can build and optimize their own training and inference systems.

The loop
InfraDev
Benchmark · arXiv
  1. Coding agents write infrastructure patches
  2. Patches improve training and inference systems
  3. Improved systems train better foundation models
  4. Better models power stronger coding agents
  1. Coding agents write infrastructure patches
  2. Patches improve training and inference systems
  3. Improved systems train better foundation models
  4. Better models power stronger coding agents
↻ The improved system does the next round, and the loop turns again.

Why it is a road to recursion

Benchmarking agents on production-grade AI infrastructure development is a critical step toward models that can autonomously optimize their own training and inference stacks.

Evidence

Across six frontier models, the best resolves only 38.9% of the tasks.

benchmarkai-infrastructurecoding-agents

Related loops

More ai r&d evals →
PostTrainBench
BenchmarkarXiv
PostTrainBench
  1. Coding agent writes fine-tuning code
  2. Agent post-trains a base LLM
  3. Trained model scored on target benchmark
  4. One turn of model training another
  5. repeat
AI R&D evalsELLIS Institute Tubingen, MPI-IS, University of Tubingen, Thoughtful Lab (Rank et al.) · 2026
VibeTraining-Bench
BenchmarkarXiv
VibeTraining-Bench
  1. Research agents refine target system
  2. Modify training data and harness
  3. Evaluate against validation feedback
  4. Submit improved final agent
  5. repeat
AI R&D evalsPeng et al. · 2026
LLM Speedrunning Benchmark
BenchmarkarXiv
LLM Speedrunning Benchmark
  1. Agent rewrites LLM training script
  2. Script makes LLM training faster
  3. Benchmark scores the recovered speedup
  4. Speeds up agent's own model training
  5. repeat
AI R&D evalsMeta and collaborators (Zhao et al.) · 2025