Alphabell.
Paper · Architecture and optimizer search

Genesys

Language Modeling by Language Models (Genesys)Genesys is a multi-agent LLM system that simulates the research cycle for new language-model architectures, from ideation and literature search to code, pre-training and evaluation, on a genetic-programming backbone.

The loop

LLM designer agents propose, adversarially review and implement new LM architecture designs, and verifier agents pre-train and evaluate selected designs on a Ladder of Scales (14M to 350M parameters) with fewer models trained at each larger scale. Verified results feed the evolutionary population the designers draw from next, so language models search for better language-model architectures.

The loop
Genesys
Paper · arXiv
  1. Designer agents propose language model architectures
  2. Verifier agents pre-train and evaluate designs
  3. Verified results feed evolutionary population
  4. Language models search for better architectures
  1. Designer agents propose language model architectures
  2. Verifier agents pre-train and evaluate designs
  3. Verified results feed evolutionary population
  4. Language models search for better architectures
↻ The improved system does the next round, and the loop turns again.

Why it is a road to recursion

It is an explicit setup of language models discovering language-model designs, the shape a recursive loop takes once the discovered designs train the next designers.

Evidence

Of 1,162 newly discovered designs (1,062 verified through pre-training), the best outperform GPT2, Mamba2 and others on 6 of 9 common benchmarks, and the genetic-programming backbone gave about an 86 percentage point improvement in successful design generation over direct prompting.

architecture-searchgenetic-programmingmulti-agentlanguage-modelsai2

Related loops

More architecture and optimizer search →
FoundationarXiv
Self-training learned optimizers
  1. Learned optimizers update other learned optimizers
  2. Training keeps best performing optimizer parameters
  3. Better optimizers train each other faster
  4. repeat
Architecture and optimizer searchMetz et al. (Google Research, Brain Team) · 2021
ProjectGitHub
ShinkaEvolve
  1. LLM ensemble rewrites candidate programs
  2. Evaluator scores and selects best programs
  3. Evolved mixture-of-experts load-balancing loss
  4. Loss improves LLM training
  5. repeat
Architecture and optimizer searchSakana AI · 2025
PaperarXiv
DiscoPOP
  1. GPT-4 proposes preference-optimization losses
  2. Candidate losses fine-tune language models
  3. Evaluation scores feed back into prompt
  4. Best loss aligns other LLMs
  5. repeat
Architecture and optimizer searchSakana AI, FLAIR (Oxford), University of Cambridge · 2024