Alphabell.
Measure

Benchmarks for the loop

Evals that measure how well AI does AI research and engineering. If AI is going to improve AI, these are the scoreboards that show whether it is working.

The trend

How long a task AI can finish on its ownMETR 50% time horizon on software and AI R&D tasks, by model release date (log scale)
Best so farEach model
1 min4 min15 min60 min4 h16 h64 h20242025202617 h1 min4 min15 min60 min4 h16 h64 h20242025202617 h

From 4 min (Mar 2023) to 17 h at the frontier, about 262x. Data: METR via Epoch AI (CC BY).

Show the data
DateModelValue
14 Mar 2023gpt-4-03144 min
6 Nov 2023gpt-4-1106-preview4 min
29 Feb 2024claude-3-opus-202402294 min
9 Apr 2024gpt-4-turbo-2024-04-094 min
13 May 2024gpt-4o-2024-05-137 min
20 Jun 2024claude-3-5-sonnet-2024062011 min
12 Sep 2024o1-preview-2024-09-1220 min
22 Oct 2024claude-3-5-sonnet-2024102221 min
17 Dec 2024o1-2024-12-17_medium39 min
24 Feb 2025claude-3-7-sonnet-2025021960 min
16 Apr 2025o3-2025-04-16_unknown2 h
22 May 2025claude-opus-4-20250514_unknown1.7 h
5 Aug 2025claude-opus-4-1-202508051.7 h
7 Aug 2025gpt-5-2025-08-07_high3.4 h
18 Nov 2025gemini-3-pro-preview3.7 h
19 Nov 2025gpt-5.1-codex-max3.7 h
24 Nov 2025claude-opus-4-5-202511014.9 h
11 Dec 2025gpt-5.2-2025-12-11_high5.9 h
5 Feb 2026claude-opus-4-6_unknown12 h
5 Feb 2026gpt-5.3-codex5.8 h
19 Feb 2026gemini-3.1-pro-preview6.4 h
5 Mar 2026gpt-5.4-2026-03-05_xhigh5.7 h
7 Apr 2026claude-mythos-preview-early17 h
Record runtime, GPU MODE choleskyB200 kernel runtime, ended 2026-07-30
200 µs500 µs1 ms2 ms15 Jul17 Jul19 Jul21 Jul23 Jul25 Jul27 Jul2.04 ms202 µs200 µs500 µs1 ms2 ms15 Jul20 Jul25 Jul2.04 ms202 µs

69 record-setting submissions took the best kernel from 2.04 ms to 202 µs, 10.1x faster. Data: GPU MODE KernelBot.

Show the data
DateSubmitted byRuntime
14 Jul 2026msaroufim2.04 ms
14 Jul 2026xuan99382.03 ms
14 Jul 2026xuan99381.92 ms
14 Jul 2026xuan99381.91 ms
14 Jul 2026Voldemort43211.74 ms
14 Jul 2026Voldemort43211.73 ms
14 Jul 2026xuan99381.52 ms
15 Jul 2026Voldemort43211.50 ms
15 Jul 2026xuan99381.41 ms
15 Jul 2026xuan99381.41 ms
15 Jul 2026xuan99381.36 ms
15 Jul 2026Voldemort43211.31 ms
15 Jul 2026Voldemort43211.30 ms
15 Jul 2026Voldemort43211.26 ms
15 Jul 2026Voldemort43211.17 ms
15 Jul 2026Voldemort43211.08 ms
15 Jul 2026Voldemort4321981 µs
15 Jul 2026Voldemort4321932 µs
15 Jul 2026seanyang843 µs
15 Jul 2026seanyang834 µs
15 Jul 2026seanyang820 µs
15 Jul 2026seanyang807 µs
15 Jul 2026seanyang780 µs
15 Jul 2026seanyang769 µs
15 Jul 2026seanyang750 µs
16 Jul 2026seanyang730 µs
16 Jul 2026seanyang728 µs
16 Jul 2026seanyang726 µs
16 Jul 2026seanyang724 µs
16 Jul 2026seanyang722 µs
16 Jul 2026seanyang719 µs
16 Jul 2026seanyang703 µs
16 Jul 2026seanyang702 µs
16 Jul 2026seanyang699 µs
16 Jul 2026seanyang698 µs
17 Jul 2026seanyang690 µs
17 Jul 2026seanyang690 µs
17 Jul 2026Voldemort4321657 µs
17 Jul 2026Voldemort4321615 µs
17 Jul 2026Voldemort4321609 µs
18 Jul 2026Olek602 µs
18 Jul 2026Olek601 µs
18 Jul 2026Olek599 µs
18 Jul 2026Olek592 µs
18 Jul 2026Olek584 µs
18 Jul 2026Olek580 µs
19 Jul 2026Olek579 µs
19 Jul 2026Olek576 µs
19 Jul 2026Olek575 µs
19 Jul 2026Olek574 µs
19 Jul 2026Ravi Theja573 µs
19 Jul 2026dannywillowliu-uchi567 µs
19 Jul 2026Ravi Theja533 µs
19 Jul 2026Ravi Theja532 µs
19 Jul 2026Ravi Theja529 µs
19 Jul 2026Ravi Theja522 µs
19 Jul 2026Ravi Theja515 µs
19 Jul 2026Ravi Theja508 µs
19 Jul 2026Ravi Theja506 µs
19 Jul 2026Sebastian Kimberk505 µs
19 Jul 2026Sebastian Kimberk499 µs
20 Jul 2026Sebastian Kimberk498 µs
20 Jul 2026Sebastian Kimberk497 µs
20 Jul 2026zhongmingee375 µs
22 Jul 2026zhongmingee321 µs
23 Jul 2026viridale318 µs
23 Jul 2026zhongmingee226 µs
24 Jul 2026zhongmingee206 µs
28 Jul 2026zhongmingee202 µs
Record runtime, GPU MODE eighB200 kernel runtime, ended 2026-07-15
10 ms20 ms50 ms27 Jun30 Jun3 Jul6 Jul9 Jul12 Jul57 ms6.60 ms10 ms20 ms50 ms27 Jun3 Jul9 Jul57 ms6.60 ms

77 record-setting submissions took the best kernel from 57 ms to 6.60 ms, 8.6x faster. Data: GPU MODE KernelBot.

Show the data
DateSubmitted byRuntime
26 Jun 2026revolutionaryspaces57 ms
26 Jun 2026Sinatras54 ms
26 Jun 2026Sinatras54 ms
26 Jun 2026Sinatras49 ms
27 Jun 2026zyzy07234348 ms
27 Jun 2026zyzy07234348 ms
29 Jun 2026zyzy07234347 ms
30 Jun 2026kumarkrishna40 ms
30 Jun 2026kumarkrishna40 ms
30 Jun 2026kumarkrishna35 ms
30 Jun 2026kumarkrishna32 ms
30 Jun 2026kumarkrishna31 ms
30 Jun 2026kumarkrishna31 ms
30 Jun 2026kumarkrishna31 ms
30 Jun 2026kumarkrishna31 ms
30 Jun 2026kumarkrishna27 ms
1 Jul 2026kumarkrishna22 ms
1 Jul 2026kumarkrishna22 ms
1 Jul 2026kumarkrishna21 ms
1 Jul 2026kumarkrishna21 ms
1 Jul 2026kumarkrishna21 ms
1 Jul 2026kumarkrishna21 ms
1 Jul 2026kumarkrishna20 ms
2 Jul 2026kumarkrishna19 ms
2 Jul 2026kumarkrishna19 ms
2 Jul 2026kumarkrishna19 ms
2 Jul 2026kumarkrishna19 ms
2 Jul 2026kumarkrishna19 ms
2 Jul 2026kumarkrishna19 ms
3 Jul 2026zhongmingee18 ms
3 Jul 2026kumarkrishna18 ms
3 Jul 2026kumarkrishna17 ms
3 Jul 2026kumarkrishna17 ms
3 Jul 2026kumarkrishna15 ms
3 Jul 2026nikhilbarhate9915 ms
3 Jul 2026nikhilbarhate9915 ms
3 Jul 2026nikhilbarhate9915 ms
3 Jul 2026nikhilbarhate9915 ms
3 Jul 2026nikhilbarhate9915 ms
3 Jul 2026nikhilbarhate9914 ms
3 Jul 2026zhongmingee13 ms
4 Jul 2026zhongmingee12 ms
4 Jul 2026nikhilbarhate9912 ms
4 Jul 2026nikhilbarhate9912 ms
4 Jul 2026nikhilbarhate9912 ms
4 Jul 2026nikhilbarhate9912 ms
4 Jul 2026nikhilbarhate9911 ms
4 Jul 2026nikhilbarhate9911 ms
4 Jul 2026nikhilbarhate9911 ms
4 Jul 2026zhongmingee9.26 ms
5 Jul 2026zhongmingee9.17 ms
5 Jul 2026nikhilbarhate999.14 ms
5 Jul 2026nikhilbarhate999.11 ms
5 Jul 2026zhongmingee8.85 ms
5 Jul 2026zhongmingee8.81 ms
5 Jul 2026zhongmingee8.66 ms
6 Jul 2026zhongmingee8.64 ms
6 Jul 2026zhongmingee8.63 ms
6 Jul 2026zhongmingee8.59 ms
6 Jul 2026zhongmingee8.50 ms
6 Jul 2026zhongmingee8.45 ms
6 Jul 2026zhongmingee8.26 ms
6 Jul 2026zhongmingee8.14 ms
7 Jul 2026zhongmingee7.67 ms
8 Jul 2026zhongmingee7.32 ms
8 Jul 2026zhongmingee7.28 ms
8 Jul 2026zhongmingee7.20 ms
11 Jul 2026zhongmingee7.10 ms
11 Jul 2026zhongmingee7.00 ms
11 Jul 2026zhongmingee6.97 ms
11 Jul 2026zhongmingee6.96 ms
11 Jul 2026zhongmingee6.96 ms
12 Jul 2026nikhilbarhate996.89 ms
12 Jul 2026nikhilbarhate996.85 ms
12 Jul 2026nikhilbarhate996.78 ms
14 Jul 2026teelaitila6.74 ms
14 Jul 2026teelaitila6.60 ms

The benchmarks

PostTrainBench
BenchmarkarXiv
PostTrainBench
  1. Coding agent writes fine-tuning code
  2. Agent post-trains a base LLM
  3. Trained model scored on target benchmark
  4. One turn of model training another
  5. repeat
AI R&D evalsELLIS Institute Tubingen, MPI-IS, University of Tubingen, Thoughtful Lab (Rank et al.) · 2026
FlashInfer-Bench
BenchmarkarXiv
FlashInfer-Bench
  1. LLM agents write GPU kernels
  2. Faster kernels make LLM inference cheaper
  3. New serving traces define next kernel tasks
  4. repeat
Inference efficiencyUW, CMU, NVIDIA, UC Berkeley (Xing et al.) · 2026
LLM Speedrunning Benchmark
BenchmarkarXiv
LLM Speedrunning Benchmark
  1. Agent rewrites LLM training script
  2. Script makes LLM training faster
  3. Benchmark scores the recovered speedup
  4. Speeds up agent's own model training
  5. repeat
AI R&D evalsMeta and collaborators (Zhao et al.) · 2025
PaperBench
BenchmarkarXiv
PaperBench
  1. Agent replicates published ML research
  2. LLM judge grades the replication
  3. Measures ability to do ML research
  4. Research produces better future models
  5. repeat
AI R&D evalsOpenAI · 2025
KernelBench
BenchmarkGitHub
KernelBench
  1. LLM writes replacement GPU kernel
  2. Kernel is timed against baseline
  3. Workloads are neural network operators
  4. Makes agent's own training faster
  5. repeat
AI R&D evalsOuyang et al. (Stanford) · 2024
RE-Bench
BenchmarkMETR
RE-Bench
  1. Agents do AI research tasks
  2. Scored against human expert baselines
  3. Measures automation of AI research
  4. Research improves future AI systems
  5. repeat
AI R&D evalsMETR · 2024
MLE-bench
BenchmarkGitHub
MLE-bench
  1. Agents build and train ML models
  2. Submissions scored against human thresholds
  3. Measures open-ended ML research ability
  4. Agents improve their own training code
  5. repeat
AI R&D evalsOpenAI · 2024

Live leaderboards

Snapshots of public leaderboards that score AI doing AI research and engineering, refreshed daily. Full results and credit belong to the organisers.

PostTrainBench
AI agents post-train a base model; score is the trained model's benchmark average
  • Fable 5 (Max) · Claude Code41.8%
  • GPT 5.6 (Sol) (Max) · Codex CLI36.2%
  • Opus 5 · Claude Code35.0%
  • Opus 4.8 (High) · Claude Code33.8%
  • Opus 4.8 (Max) · Claude Code32.9%
Data: Epoch AI Benchmarking Hub (CC BY), 11 Oct 2026
METR time horizons
Length of software and AI R&D tasks a model completes at 50% reliability
  • claude-mythos-preview-early17.4 h
  • claude-opus-4-6_unknown12.0 h
  • gemini-3.1-pro-preview6.4 h
  • gpt-5.2-2025-12-11_high5.9 h
  • gpt-5.3-codex5.8 h
Data: Epoch AI Benchmarking Hub (CC BY), 11 Oct 2026
GSO
Agents optimize real codebases for speed; share of tasks matching expert speedups
  • claude-fable-5-1_unknown · OpenHands88.2%
  • gpt-6-astra_unknown · OpenHands79.4%
  • claude-fable-5_unknown · OpenHands78.4%
  • gpt-5.6-sol_unknown · OpenHands76.5%
  • claude-opus-4-8_unknown · OpenHands47.1%
Data: Epoch AI Benchmarking Hub (CC BY), 11 Oct 2026
AlgoTune
Models speed up numerical programs; score is the average speedup
  • GPT-5.2 (high)2.05x
  • Gemini 3.1 Pro Preview (high)2.02x
  • GPT-5.4 (high)1.85x
  • Gemini 3 Pro Preview (high)1.83x
  • Claude Opus 4.5 (high)1.77x
Data: Epoch AI Benchmarking Hub (CC BY), 11 Oct 2026
SWE-bench Verified
Real GitHub issues fixed by coding agents, the harness behind most AI-built software
  • claude-opus-4-7_max83.5%
  • gpt-5.5-pre-release_xhigh80.6%
  • gemini-3.5-flash_high79.3%
  • claude-opus-4-678.7%
  • glm-5.2_max78.7%
Data: Epoch AI Benchmarking Hub (CC BY), 11 Oct 2026
Terminal-Bench 2.0
Agents complete hard tasks in a terminal; ranked by model and agent harness
  • GPT-5.5 · NexAU-AHE84.7%
  • GPT-5.5 · Capy83.1%
  • GPT-5.5 · Codex CLI82.2%
  • GPT-5.5 · Codex82.0%
  • GPT-5.4 · ForgeCode81.8%
Data: Epoch AI Benchmarking Hub (CC BY), 11 Oct 2026
GPU MODE cholesky
B200 kernel runtime, ended 2026-07-30 · lower is better
  • zhongmingee201.53 µs
  • Frosty40240.68 µs
  • Olek251.11 µs
Data: GPU MODE KernelBot, 11 Oct 2026
GPU MODE eigh
B200 kernel runtime, ended 2026-07-15 · lower is better
  • teelaitila6.60 ms
  • nikhilbarhate996.74 ms
  • elianaive6.80 ms
Data: GPU MODE KernelBot, 11 Oct 2026
GPU MODE qr_v2
B200 kernel runtime, ended 2026-06-30 · lower is better
  • 10billiontokens1.22 ms
  • gau.nernst1.23 ms
  • dhu.randhar1.28 ms
Data: GPU MODE KernelBot, 11 Oct 2026
GPU MODE qr
B200 kernel runtime, ended 2026-06-30 · lower is better
  • nikhilbarhate99704.87 µs
  • badelsteinlelbach1.21 ms
  • Sinatras2.00 ms
Data: GPU MODE KernelBot, 11 Oct 2026
GPU MODE trimul
B200 kernel runtime, ended 2026-05-09 · lower is better
  • josusanmartin553.98 µs
  • shiyegao554.40 µs
  • ajay_a602.21 µs
Data: GPU MODE KernelBot, 11 Oct 2026
GPU MODE princeton_cross_entropy
A100 kernel runtime, ended 2026-04-17 · lower is better
  • fs908411.26 µs
  • lszshu283.75 µs
  • DIodide343.39 µs
Data: GPU MODE KernelBot, 11 Oct 2026

Why these matter

A loop only compounds if each turn is measurably better than the last. General capability benchmarks rarely tell you that. The benchmarks here test the specific skills a loop depends on: reproducing research, running ML experiments, writing fast kernels, and improving an agent's own code. We list a benchmark when its score is a direct read on how far AI can take over its own development.

Built an AI R&D eval?

Send it to us for the index, or apply for a grant to extend it.

Submit a benchmark