Alphabell.
Data and training signal

Self-reward and self-play

Models that supply their own training signal: judges, critics, verifiers and self-play opponents.

PaperarXiv
Self-Rewarding LMs
  1. Model generates candidate responses to prompts
  2. Model scores them itself as a judge
  3. Scores become preference pairs for DPO
  4. Trained model becomes next generator and judge
  5. repeat
Self-reward and self-playMeta, NYU (Yuan et al.) · 2024
PaperarXiv
Absolute Zero
  1. Model proposes code reasoning tasks
  2. Model solves them for accuracy reward
  3. Python executor verifies the answers
  4. Curriculum shifts as solver improves
  5. repeat
Self-reward and self-playTsinghua, BIGAI, Penn State (Zhao et al.) · 2025
PaperarXiv
rStar-Math
  1. Policy model runs Monte Carlo Tree Search
  2. Verified trajectories train next policy and PPM
  3. PPM guides search to produce better data
  4. Self-evolution improves both models
  5. repeat
Self-reward and self-playMicrosoft Research Asia · 2025
PaperarXiv
Self-Taught Evaluator
  1. LLM builds worse answers for preference pairs
  2. Judge samples reasoning traces and verdicts
  3. Judge is fine-tuned on correct verdicts
  4. Improved judge labels next iteration's data
  5. repeat
Self-reward and self-playMeta FAIR · 2024
FoundationarXiv
Constitutional AI
  1. Model critiques and revises its own responses
  2. Model is fine-tuned on the revisions
  3. Model judges responses to train preference model
  4. Preference model provides RL reward signal
  5. repeat
Self-reward and self-playBai et al. (Anthropic) · 2022
FoundationarXiv
AlphaZero
  1. Network plays games against itself
  2. Games become the training data
  3. Same network trains on its own moves
  4. Updated network plays next self-play games
  5. repeat
Self-reward and self-playSilver et al. (DeepMind) · 2017

Recently on the radar

More →
PaperarXiv·29 Sep 2026·Self-reward and self-play

RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

The loop: A policy generates its own feedback insights from failed attempts at difficult tasks. It then internalizes these insights through training, breaking through learning barriers and improving its success rate on future rollouts.

loop fit 9/10Michael Kirchhof, Eleonora Gualdoni, Andrew Szot et al.via arXiv

Other loop types

Self-modifying agentsTools and harnessTraining dataKernels and compilersHardware and chipsInference efficiencyArchitecture and optimizer searchResearch automationAI R&D evalsInterpretability and oversight