Today's edition highlights the growing trend of treating agent harnesses and memory as dynamic, self-evolving components rather than static wrappers.
The loop: Anthropic AI systems perform AI R&D work, which improves Anthropic's models and tools. This makes the systems or their direct successors better at performing further AI R&D work.
METR conducted an independent assessment of Anthropic's internal use of AI to accelerate its own AI research and development.
The loop: A coding agent uses Monte Carlo Tree Search to iteratively search for and refine its own executable harness. This optimized harness directly governs the agent's interaction with tools and the environment, improving its performance on subsequent tasks.
A framework that automates the design of agent harnesses by using a coding agent and MCTS to search for and refine harness mechanisms.
The loop: LLM agents write research proposals based on a literature snapshot, which are scored against actual future publications. This benchmark measures AI performance on AI research tasks, providing the necessary evaluation signal to improve future AI research agents.
A benchmark that evaluates whether LLMs can propose research ideas that match the actual future trajectory of human scientific communities.
The loop: The Recursa foundation model system uses parallel learners to execute tasks and synthesize local improvements. These improvements are validated and merged into the system's own knowledge repository, enabling the model to adapt to new domains through recursive self-improvement.
A foundation model system that adapts to new domains by learning from its own experience and storing domain-specific skills in a self-refined knowledge repository.
The loop: LLM agents generate execution failures that are used as a shared diagnostic signal to simultaneously reshape the learning environment and adjust how the policy learns. This keeps the training environment aligned with the agent's evolving needs, directly improving the agent's subsequent training.
A framework that uses agent execution failures to co-evolve the agent's policy and its learning environment.
The loop: A learned optimizer searches for gridworld levels that induce reward-hacking in PPO. This automated discovery of honeypots improves the evaluation of reinforcement learning algorithms, helping developers build more robust training processes.
A project that trains a small optimizer to automatically discover gridworld levels where PPO agents learn to cheat.
The loop: A swarm of agents generates diffs to their own skills, prompts, and harnesses. An aggregator acts as an optimizer to apply these changes, improving the performance of the agents in the swarm.
A parallel, asynchronous framework that treats agents as parameters and uses a gradient-descent-like process to optimize their prompts and harnesses.
The loop: Claude uses a dreaming mechanism to organize its memory and correct past mistakes. This process improves its internal state, making the model better at learning from experience in subsequent interactions.
An overview of how Claude uses a dreaming mechanism to organize memory and correct mistakes to learn from experience.
The loop: AI systems at Anthropic are used to accelerate their own R&D, a process independently verified by METR.
Redaction summary: As part of this assessment, Anthropic had rights to redact any non-public information in this post. Per our approach to transparency , we had the right to disclose if Anthropic exercised redaction rights, and whether or...
The loop: Anthropic AI systems perform AI R&D work, which improves Anthropic's models and tools, which makes the systems better at AI R&D work.
Redaction summary: As part of this assessment, Anthropic had rights to redact any non-public information in this post. Per our approach to transparency , we had the right to disclose if Anthropic exercised redaction rights, and whether or...
The loop: This benchmark evaluates predictive safety guardrails within an agent harness, providing data to improve agent safety mechanisms.
Listed in Awesome-AI4AI under: evidence/benchmarks, targets/harness
The loop: The system measures gaps between agent behavior in evals versus production to improve the reliability of AI safety benchmarks.
We study agent behavior in safety evaluations and in production, and explore how models' interpretations of evaluation objectives relate to eval-production performance gaps.
