Search
Architecture and optimizer search
AI searching over architectures, optimizers, losses and training recipes for the next model.
Self-training learned optimizers
- Learned optimizers update other learned optimizers
- Training keeps best performing optimizer parameters
- Better optimizers train each other faster
- repeat
Architecture and optimizer searchMetz et al. (Google Research, Brain Team) · 2021
ShinkaEvolve
- LLM ensemble rewrites candidate programs
- Evaluator scores and selects best programs
- Evolved mixture-of-experts load-balancing loss
- Loss improves LLM training
- repeat
Architecture and optimizer searchSakana AI · 2025
Genesys
- Designer agents propose language model architectures
- Verifier agents pre-train and evaluate designs
- Verified results feed evolutionary population
- Language models search for better architectures
- repeat
Architecture and optimizer searchCheng, Clark, Richardson (Allen Institute for AI, Dartmouth) · 2025
DiscoPOP
- GPT-4 proposes preference-optimization losses
- Candidate losses fine-tune language models
- Evaluation scores feed back into prompt
- Best loss aligns other LLMs
- repeat
Architecture and optimizer searchSakana AI, FLAIR (Oxford), University of Cambridge · 2024
NAS with RL
- RNN controller writes child network architecture
- Child validation accuracy returns as reward
- Updates make controller propose better architectures
- repeat
Architecture and optimizer searchZoph and Le (Google Brain) · 2016