Alphabell.
Paper · Self-reward and self-play

Self-Taught Evaluator

Self-Taught EvaluatorsA method for training an LLM-as-a-Judge with no human preference labels, using only synthetic contrasting response pairs and the judge's own filtered reasoning.

The loop

An LLM builds a worse answer for each instruction by answering a subtly modified version of it, which yields a preference pair with a known winner. The current judge samples reasoning traces and verdicts, only verdicts that pick the known winner are kept, the judge is fine-tuned on them, and the improved judge labels the next iteration's data.

The loop
Self-Taught Evaluator
Paper · arXiv
  1. LLM builds worse answers for preference pairs
  2. Judge samples reasoning traces and verdicts
  3. Judge is fine-tuned on correct verdicts
  4. Improved judge labels next iteration's data
  1. LLM builds worse answers for preference pairs
  2. Judge samples reasoning traces and verdicts
  3. Judge is fine-tuned on correct verdicts
  4. Improved judge labels next iteration's data
↻ The improved system does the next round, and the loop turns again.

Why it is a road to recursion

Judges supply the reward signal for training other models, so an evaluator that improves itself without human labels removes a human bottleneck on the feedback that trains every model downstream.

Evidence

Without labeled preference data, the method raised Llama3-70B-Instruct from 75.4 to 88.3 on RewardBench (88.7 with majority vote), matching top reward models trained on labeled examples.

llm-as-judgereward-modelsynthetic-preferencesiterative-trainingevaluation

Related loops

More self-reward and self-play →
PaperarXiv
Self-Rewarding LMs
  1. Model generates candidate responses to prompts
  2. Model scores them itself as a judge
  3. Scores become preference pairs for DPO
  4. Trained model becomes next generator and judge
  5. repeat
Self-reward and self-playMeta, NYU (Yuan et al.) · 2024
PaperarXiv
Absolute Zero
  1. Model proposes code reasoning tasks
  2. Model solves them for accuracy reward
  3. Python executor verifies the answers
  4. Curriculum shifts as solver improves
  5. repeat
Self-reward and self-playTsinghua, BIGAI, Penn State (Zhao et al.) · 2025
PaperarXiv
rStar-Math
  1. Policy model runs Monte Carlo Tree Search
  2. Verified trajectories train next policy and PPM
  3. PPM guides search to produce better data
  4. Self-evolution improves both models
  5. repeat
Self-reward and self-playMicrosoft Research Asia · 2025