Skip to content

Compute stack

Inference efficiency

AI that makes AI cheaper to run: distillation, quantization search, draft models and routing.

Context Language ModelsNew

Model updates context file
Skill-optimization evolves instructions
Model uses improved instructions
Model manages context better
repeat

Inference efficiencyRu-Lin Shao et al. · 2026

FlashInfer-Bench

LLM agents write GPU kernels
Faster kernels make LLM inference cheaper
New serving traces define next kernel tasks
repeat

Inference efficiencyUW, CMU, NVIDIA, UC Berkeley (Xing et al.) · 2026

LLM agents write inference cluster code
Designs lower latency and GPU cost
Lowers cost of serving LLMs running Glia
repeat

Inference efficiencyHamadanian et al. (MIT CSAIL) · 2025

LLMs propose and mutate load balancer code
Best programs seed the next generation
Output is faster serving code for LLMs
repeat

Inference efficiencyUC Berkeley (Sky Computing Lab) · 2025

Recently on the radar

PaperarXiv·29 Sep 2026·Inference efficiency

Context Language Models

The loop: A language model natively manages its own context by updating a context file. The model's context-management instructions are then evolved through a skill-optimization loop, improving its accuracy and efficiency on subsequent tasks.

loop fit 10/10Ru-Lin Shao, Shannon Zejiang Shen, J. Yin et al.via Semantic Scholar

Other loop types

Self-modifying agents Tools and harness Training data Self-reward and self-play Kernels and compilers Hardware and chips Architecture and optimizer search Research automation AI R&D evals Interpretability and oversight