Alphabell.
Paper · Self-reward and self-play

The Red Queen Gödel Machine

The Red Queen Gödel Machine: Co-Evolving Agents and Their EvaluatorsAn evolutionary framework that enables recursive self-improvement by co-evolving agents alongside the evaluators that guide their search.

The Red Queen Gödel Machine
Figure from the paper, arXiv. Source
The loop

A coder agent and a reviewer agent co-evolve under non-stationary utilities. The reviewer grades patches to guide the coder's search, while the coder's output helps the reviewer refine its grading rubric, making both better at their respective tasks.

The loop
The Red Queen Gödel Machine
Paper · arXiv
  1. Coder generates patches
  2. Reviewer grades patches to guide search
  3. Coder output helps reviewer refine rubric
  4. Reviewer provides better guidance to coder
  1. Coder generates patches
  2. Reviewer grades patches to guide search
  3. Coder output helps reviewer refine rubric
  4. Reviewer provides better guidance to coder
↻ The improved system does the next round, and the loop turns again.

Why it is a road to recursion

Co-evolving the evaluator alongside the agent removes the ceiling imposed by static reward functions, enabling open-ended recursive self-improvement.

Evidence

At low reasoning effort, the RQGM coder passes 82.1% of held-out tasks against the baseline's 75.0%.

co-evolutionagentic-codinglearned-evaluators

Related loops

More self-reward and self-play →
Self-Rewarding LMs
PaperarXiv
Self-Rewarding LMs
  1. Model generates candidate responses to prompts
  2. Model scores them itself as a judge
  3. Scores become preference pairs for DPO
  4. Trained model becomes next generator and judge
  5. repeat
Self-reward and self-playMeta, NYU (Yuan et al.) · 2024
Absolute Zero
PaperarXiv
Absolute Zero
  1. Model proposes code reasoning tasks
  2. Model solves them for accuracy reward
  3. Python executor verifies the answers
  4. Curriculum shifts as solver improves
  5. repeat
Self-reward and self-playTsinghua, BIGAI, Penn State (Zhao et al.) · 2025
rStar-Math
PaperarXiv
rStar-Math
  1. Policy model runs Monte Carlo Tree Search
  2. Verified trajectories train next policy and PPM
  3. PPM guides search to produce better data
  4. Self-evolution improves both models
  5. repeat
Self-reward and self-playMicrosoft Research Asia · 2025