Alphabell.
Research and evals

AI R&D evals

Benchmarks that measure how well AI does AI research and engineering: the scoreboard for the loop.

BenchmarkarXiv
PostTrainBench
  1. Coding agent writes fine-tuning code
  2. Agent post-trains a base LLM
  3. Trained model scored on target benchmark
  4. One turn of model training another
  5. repeat
AI R&D evalsELLIS Institute Tubingen, MPI-IS, University of Tubingen, Thoughtful Lab (Rank et al.) · 2026
BenchmarkarXiv
LLM Speedrunning Benchmark
  1. Agent rewrites LLM training script
  2. Script makes LLM training faster
  3. Benchmark scores the recovered speedup
  4. Speeds up agent's own model training
  5. repeat
AI R&D evalsMeta and collaborators (Zhao et al.) · 2025
BenchmarkarXiv
PaperBench
  1. Agent replicates published ML research
  2. LLM judge grades the replication
  3. Measures ability to do ML research
  4. Research produces better future models
  5. repeat
AI R&D evalsOpenAI · 2025
BenchmarkGitHub
KernelBench
  1. LLM writes replacement GPU kernel
  2. Kernel is timed against baseline
  3. Workloads are neural network operators
  4. Makes agent's own training faster
  5. repeat
AI R&D evalsOuyang et al. (Stanford) · 2024
BenchmarkMETR
RE-Bench
  1. Agents do AI research tasks
  2. Scored against human expert baselines
  3. Measures automation of AI research
  4. Research improves future AI systems
  5. repeat
AI R&D evalsMETR · 2024
BenchmarkGitHub
MLE-bench
  1. Agents build and train ML models
  2. Submissions scored against human thresholds
  3. Measures open-ended ML research ability
  4. Agents improve their own training code
  5. repeat
AI R&D evalsOpenAI · 2024

Other loop types

Self-modifying agentsTools and harnessTraining dataSelf-reward and self-playKernels and compilersHardware and chipsInference efficiencyArchitecture and optimizer searchResearch automationInterpretability and oversight