Alphabell.
Benchmark · AI R&D evals

Singularity-Bench

Singularity-Bench: Can Agents Build Agents?An open-ended benchmark that evaluates whether frontier models can autonomously build and optimize the core infrastructure of AI development, from pre-training pipelines to agent harnesses.

Singularity-Bench
Figure from the paper, arXiv. Source
The loop

Frontier AI agents are tasked with implementing core AI development systems from scratch, including pre-training, post-training, and agent harnesses. By successfully building and optimizing these pipelines, the agents directly construct the infrastructure required to train and run their own successors.

The loop
Singularity-Bench
Benchmark · arXiv
  1. Frontier AI agents
  2. implement AI training pipelines
  3. and cluster orchestration systems
  4. to build their successors
  1. Frontier AI agents
  2. implement AI training pipelines
  3. and cluster orchestration systems
  4. to build their successors
↻ The improved system does the next round, and the loop turns again.

Why it is a road to recursion

It provides an open-ended, end-to-end evaluation of whether current models can autonomously engineer the core infrastructure required to train the next generation of models.

Evidence

Evaluated 11 agents built from 7 frontier models and 2 harnesses against mature human-engineered baselines on identical hardware.

benchmarkagent-engineeringinfrastructure

Related loops

More ai r&d evals →
PostTrainBench
BenchmarkarXiv
PostTrainBench
  1. Coding agent writes fine-tuning code
  2. Agent post-trains a base LLM
  3. Trained model scored on target benchmark
  4. One turn of model training another
  5. repeat
AI R&D evalsELLIS Institute Tubingen, MPI-IS, University of Tubingen, Thoughtful Lab (Rank et al.) · 2026
InfraDev
BenchmarkarXiv
InfraDev
  1. Coding agents write infrastructure patches
  2. Patches improve training and inference systems
  3. Improved systems train better foundation models
  4. Better models power stronger coding agents
  5. repeat
AI R&D evalsYangshen Deng et al. · 2026
VibeTraining-Bench
BenchmarkarXiv
VibeTraining-Bench
  1. Research agents refine target system
  2. Modify training data and harness
  3. Evaluate against validation feedback
  4. Submit improved final agent
  5. repeat
AI R&D evalsPeng et al. · 2026