How Agents Learn to Improve Themselves

An agent you ship today is frozen. Same harness, same instructions, same tools, and no memory of anything that happened in past sessions. Working with one is like working with a smart colleague who has amnesia. You explain the problem in one session. Open a fresh session, and you have to share all of that context again.

So what would it take to build an agent that learns on the job, from its own successes and failures?

Three research survey papers map out how that works.

The KAUST survey. From Jilin University and KAUST.
The TMLR survey. Published in TMLR by a group spanning Princeton, Tsinghua and a dozen other universities.
The Glasgow survey. From the University of Glasgow and seven other universities.

How Do You Know If You Are Building a Self-Evolving Agent?

Plenty of approaches get called self-evolving. Most are not.

The TMLR survey gives three conditions. An agent is only self-evolving if the change it makes to itself is:

Driven by its own experience. It comes from the agent's own runs and feedback, not from a dataset a person handed it.
Persistent. It survives past the end of the task. It is not just the agent behaving differently for one conversation.
Self-started. The agent decides to learn, rather than waiting for a human to schedule an update.

That rules out most of what people point at:

Approach	Self-evolving?	Why
An agent that retries after an error	No	Nothing survives the session
Search over a live document store	No	The documents change, the agent does not
A monthly retrain on labeled data	No	Humans choose what it learns
Logs a failure, writes a rule, keeps the rule	Yes	Meets all three

The KAUST survey words the same test differently. The change has to be durable past the task boundary. Everything an agent holds while working, its notes, its plan, its running context, is temporary by design. Self-evolution is about what it writes down and keeps.

An Agent Is Two Things: The Harness and the Model

The KAUST survey splits any modern agent into two parts.

The model. The weights. The trained brain.
The harness. Everything wrapped around it. Instructions, memory, the tools it can call, and the control logic that decides what happens when.

Both can change, though they behave nothing alike.

	Model weights	Harness
Speed of change	Slow	Fast
Cost	High	Low
Easy to undo	No	Yes
Can you see what changed	Not really	Yes
Carries across domains	Well	Less well

A bad instruction is easy to delete. A problem baked into model weights is, in the survey's words, "notoriously difficult to trace".

There is a second reason to be careful about pushing changes into weights. The survey describes it as lossy compression. Train on a pile of successful runs and the model learns what those runs had in common. The unusual steps get averaged away. That includes the one odd move the agent made to dig itself out of trouble. Which is often the most useful thing in the whole run.

So the practical rule is simple. Change the harness first, and only push a behavior into the weights once it has proven stable there. Re-test safety afterwards, because a weight change invalidates whatever testing you did before it.

Cursor demonstrates this perfectly. It trains its Composer models with reinforcement learning in the same harness the shipped model runs in, calling the same tools a real user would trigger. The harness stops being the thing you tune around the model, and becomes the environment the model is trained in.

EvoHarness-RL, from UIUC and Meta AI, makes that relationship even tighter. It trains an 8B model to decide when to use its external workspace. Checking the environment state, updating the plan or recalling past experience costs a turn, so the model learns when that extra step is useful.

During training, the agent harness acted as a scaffold. Early on, the agent used it often to track state, manage progress and recall past experience. As training continued, calls fell to about one per episode, while recall remained useful. The authors argue that the model had internalized some habits but still benefited from external memory. They call this harness annealing.

Five Components the Harness Can Change

Disclaimer: there is more in a harness than this. These five are the ones the research keeps coming back to.

1. Instructions. The agent rewrites its own prompts based on what went wrong. PromptBreeder, TextGrad and OPRO all do versions of this.

2. Memory. The agent stores lessons from its runs, drops the ones that no longer hold, and reorganizes what it keeps. Mem0 and Agent Workflow Memory are examples. In my experience, this is where most of the production value sits today, and it is less simple than it sounds.

The KAUST survey splits it into three questions.

What you store. Distilled lessons, exact artifacts like a code snippet, facts pulled from external sources, or compressed machine-readable representations.
How you organize it. A plain timeline, a hierarchy, a graph, or a searchable index.
How it is maintained. Writing, reading, updating and deleting.

Each approach has a tradeoff. Lessons can go stale. Exact artifacts bloat the context. Compressed representations are fast but nearly impossible to debug when they go wrong. Delete too aggressively and you lose things you needed. Delete too little and retrieval drowns.

EvoHarness-RL splits external state into three parts: belief about the environment, progress on the current plan and experience from past runs. Belief and progress stop the agent from losing its place. Experience lets it learn across tasks.

Removing any one hurt performance on the paper's household benchmark. The balance depends on the job. The household tasks relied most on experience, while coding agents may rely more on progress to track tests and unfinished work.

The memory store was also curated rather than append-only. A separate model merged useful notes and corrected or removed outdated ones. Rarely used entries were discarded, so the store grew early and then levelled off.

3. Skills. The KAUST survey defines a skill as a named update the agent keeps and reuses. The survey sorts them by what they act on.

Object-level. Acts on the task or the world, like Voyager's routine for collecting wood in Minecraft.
Meta-level. Acts on the agent itself. Writing a new tool, rewriting a prompt, patching its own code.

Meta-level skills are the ones that matter here. Because the skill both changes the agent and gets stored back inside the agent. So it can end up improving itself, and the loop starts feeding itself.

4. Tools. This is three separate jobs, and most write-ups mention only the third.

Routing. Picking the right tool as the toolbox grows, which gets harder with more tools, not easier.
Refinement. Repairing a tool that keeps failing.
Creation. Writing a new one when nothing fits.

Alita and CREATOR are the ones to look at for creation. Both generate a tool on demand instead of choosing from a fixed set.

Refinement is worth paying attention to. A broken tool that gets saved does not fail once. It gets retrieved again and again, and quietly corrupts everything downstream. Refinement is not just debugging, it is the gate on what gets into the toolbox at all.

5. The harness itself. Darwin Gödel Machine keeps a growing archive of agent variants and branches off the promising ones. AlphaEvolve evolves algorithms against evaluators. Live-SWE-Agent rewrites its own tooling while it runs.

This is what people mean when they say Recursive Self-Improvement (RSI). Worth being clear about how it actually works in practice. The KAUST survey describes these systems as bounded, verifiable loops, not open-ended. The agent proposes a patch to its own code, the patch runs against a test suite, and it is kept only if the tests say yes. Nothing ships because the agent believes in it.

I went deeper on two papers that do exactly this earlier this year.

Three Ways the Model Itself Improves

Now the other half. To change the weights rather than the harness, you need something to train on. The KAUST survey says the agent can produce three kinds of signal itself:

1. It writes its own training examples. The agent creates tasks, solves them, removes the weak examples, and learns from the good ones.

One approach splits the agent in two. A challenger invents problems that are hard but still solvable. A solver tries to solve them. As the solver improves, the challenger has to work harder, so the difficulty climbs on its own. Absolute Zero and R-Zero work this way, and Socratic-Zero adds a teacher that designs the challenges. It is the same broad idea behind AlphaZero, applied to language.

The value is that the supply of problems never runs out. A fixed set of tasks stalls the moment the agent has beaten them all, while a challenger that keeps getting harder does not. The risk is that nothing outside the system is checking the problems, so a drifting challenger leaves the solver very good at problems nobody has. R-Few solves this by anchoring the challenger to a small set of human examples.

2. It grades its own work. The agent scores, ranks or critiques its own outputs, and those judgments become the training signal. This is the same machinery behind AI feedback in model alignment.

3. It learns from doing. The agent acts in a real or simulated environment and trains on what actually happened. Success and failure, recorded.

The first two share a weakness. The agent is grading its own homework, so the self-evolving loop is only ever as good as the grader. The third avoids that, because reality does the grading. It is the strongest of the three, and the most expensive.

Letting a model act for real means running a sandbox for every attempt. That is why Cursor uses hundreds of thousands of concurrent sandboxed coding environments to train Composer, and Kimi K3 lists persistent sandbox states among the infrastructure advances behind its agentic RL.

The Blueprint for a Self-Evolving Loop

The Glasgow survey lays it out most clearly. Any self-evolving system, however fancy, is these four boxes running in a circle:

Inputs. The task, plus the examples you score against.
The agent. Model, instructions, memory, tools, skills.
The environment. Where it runs, and where feedback comes from. Compilers and tests for code. A browser for web tasks. A rubric or a scoring model when there is no clean right answer.
The optimizer. The thing that reads the feedback and changes the agent.

The optimizer is defined by two choices: what it is allowed to change, and how it picks a change. Most of the design work sits in those two knobs, and in the plumbing around them. Where the feedback comes from. When the update fires. What has to pass before a change is kept.

Run it, score it, change it, run it again. Stop when it stops helping.

Five Decisions You Have to Make

1. What is it allowed to change?

Analyze the root cause. Read the traces, find the step where it first goes wrong, and change that. If a tool is returning the wrong data, rewriting the prompt will not save you. You get an agent that words a wrong answer better.

Then, go back to the five components. Some are much safer to change than others.

Instructions are the safest. A bad instruction is just a line you delete.
Memory is almost as easy to remove, but a bad memory can keep shaping future runs until someone notices it.
Skills are reusable routines. They are reversible, but a bad skill can keep firing whenever the agent selects it.
Tools are riskier. A broken tool can be retrieved again and again, quietly corrupting everything downstream.
The harness is the most dangerous part. Most production systems do not let the agent change it directly. When they do, the agent proposes the change, and a human decides.

A general rule: change one component at a time. When the score moves, you want to know what moved it. Rewrite a prompt and add a tool in the same round, and a worse result leaves you two suspects and no way to tell them apart.

2. Where does the feedback come from?

Written critique. The agent or another model explains what went wrong in plain language. Rich and specific, but soft.
Its own confidence. Cheap to collect. Easy to fool, because a confident model can be confidently wrong.
External reward. Tests passed, task completed, user accepted the result. The strongest signal by far.
Implicit feedback. Signals hidden in normal usage. For example, a user rephrases the same request, corrects the agent, or gives up. Use AI to analyze transcripts and find those moments. They tell you something went wrong, even if nobody left a score.
Evaluation runs. Scores from running the agent against a curated ground truth dataset. The most precise of the five, and the most work, because every task still needs a ground-truth answer.

If you can turn success into something a machine can check, do that before anything else. It is not a coincidence that coding is where these systems have the most traction. Compilers, unit tests and CI pipelines hand you a verdict on every attempt, automatically, at no extra cost. Most domains have nothing like it.

3. When does it update?

The update can happen at two moments.

During the task. The agent notices it is stuck and adapts on the spot. Reflexion writes itself a critique and retries. LADDER goes further: when it hits a problem it cannot solve, it generates a set of simpler versions of that same problem, trains on those right there, then comes back to the original.

Between tasks. The agent finishes, reviews what happened, and updates for next time.

Most teams should start with the second. Adapting mid-task means every request pays the cost in latency, and the agent is learning from a single example under time pressure. Learning between tasks lets you look across many runs and spot the pattern.

4. What decides whether a change survives?

The gate. This is the single most important piece.

Before a change is kept, it has to pass a test. Not one the agent wrote for itself, one you set up. The test has to show two things: the change fixed what it was meant to fix, and nothing that used to work is now broken. The agent proposes. The test decides.

Now, there are three common pitfalls when designing an evaluation gate, and each has a practical fix.

1. The agent grades itself. If it proposes a change and also approves it, the loop becomes self-confirming and nothing is really being checked. Keep the proposer and the approver apart. For example, let GPT-5.6 propose the change, then use Claude Opus 5 to judge whether it helped.

2. The agent games the grader. The grader decides which changes survive, so the agent learns to optimize for its judgment, not necessarily the work that judgment is meant to measure. If the evaluation checks only the final answer, the agent can learn shortcuts that look right without completing the task. It can say a booking is confirmed when no reservation exists. Check the outcome first. When tool use, safety or policy matters, grade the transcript as well. Do not require one exact path, because a valid solution may take a route you did not expect.

3. The grader wears out. Reuse the same tasks for hundreds of rounds and changes that beat them by chance get kept, and they pile up. Keep every change whose score went up and you are running what one paper calls uncontrolled adaptive multiple testing. Put plainly, the loop mistakes lucky scores for real improvements and keeps them permanently. Refresh the tasks, or hold some back.

The KAUST survey has a blunt way of framing all of it: treat a self-improving agent as untrusted code running in a protected environment.

5. How do you know it improved?

Two things to get right: the tasks you measure on, and how you read the score.

The tasks

Hold a set back. Keep some evaluation tasks hidden from the improvement loop. EvoHarness-RL shows what this can reveal on unseen environments:
1. A model given the harness without training scored 77.6%.
2. A model trained to copy successful teacher actions scored 69.4%.
3. A model initialized from the supervised model and then trained with reinforcement learning scored 86.6%.
If the researchers had tested only familiar tasks, the supervised model would have looked better than it was. The KAUST survey recommends using private or unreleased tasks, or tasks created after the model's training cutoff.
Start small. Build a manually curated set of ten to twenty tasks around realistic uses of the agent. Give each task clear success criteria and, where possible, a reference answer or known-good trajectory. Together, they form a small ground truth dataset the grader can check against.
Skip the pile. Do not turn every failed trace into a test case. Many traces may capture the same underlying failure. Adding all of them creates a large suite that costs more to run without adding much coverage. Keep one representative test case for each distinct failure pattern.
Grow it, do not swap it. Add harder tasks as the agent improves, and keep the old ones. Those are what tell you when something that used to work has stopped.

The score

Run each task more than once. Agents are not deterministic and the gap is wider than it looks. The τ-bench authors found that an agent managed to solve 61% of tasks in a single attempt, but succeeded on only 25% when success required completing each task in all eight attempts. A later paper puts the bar at 30 runs before a score is worth trusting.
Choose the metric that matches the job. The metrics pass@k and pass^k answer opposite questions. The first asks whether the agent succeeds at least once in k attempts. The second asks whether it succeeds in all k attempts. Use pass@k when retries are acceptable, and pass^k when reliability matters.
Break the score apart. Track success, reliability, cost and latency separately, then slice the results by task type. An agent's overall score might rise from 75 to 80 while its score on refund tasks falls from 90 to 60. The average says it improved, but the task-level score shows that an important workflow got worse.
Report the curve, not the peak. A loop might score 70, 72, 71, 85 and 69 across five rounds. Reporting only 85 makes the change look like a breakthrough. The full curve shows that it was a spike, not a stable improvement.

What the KAUST survey recommends reporting:

The score you started at. The baseline. Without it, no later number means anything.
The score after a fixed budget. How much you got for a set amount of spend. This is what makes two loops comparable.
The curve across rounds. Tells you whether it is still climbing, has gone flat, or has started to overfit.
The score on held-out tasks. The only one that shows it learned rather than memorized.
Tasks that used to pass and now fail. Regressions. The average can rise while things quietly break underneath it.
The full cost. Compute, tool calls, time and human input. A gain that cost ten times more is not much of a gain.

How Do Multi-Agent Systems Evolve?

Those five decisions assume a single agent. Plenty of real systems are several, and the Glasgow survey is largely about that case.

Once you have more than one agent, something new becomes changeable: who talks to whom. Which agent sees which output, in what order, and when the whole thing stops. The survey calls this the topology, and its argument is that the best prompts cannot rescue a bad structure. So the structure becomes something you optimize, not something you settle at a whiteboard once.

You can define that topology in code or as a graph. In code, rules decide which agent runs next and what information it receives. In a graph, agents are nodes and connections show where information can travel. Either way, the system can change how work moves between agents.

Three findings worth noting:

Cutting communication saves money. Trimming redundant messages between agents cuts token cost without losing quality.
The same cut contains damage. In a multi-agent system, whatever one agent says gets passed to every agent it is wired to. So if that agent is hijacked, say by a hidden instruction in a web page it read, the bad output travels with it. G-Safeguard cuts the outgoing edges of any agent it flags as risky, so the damage stops at that agent.
Collaboration is a trained skill, not a free one. OPTIMA reports up to a 2.8 times gain at under 10% of the token cost on tasks needing heavy back-and-forth, by explicitly training for communication efficiency.

The point is simple: models are not naturally good teammates. Put five of them in a room and you often get cost and confusion rather than a better answer.

What Could Go Wrong When an Agent Improves Itself

This part rarely makes it into the announcements.

The TMLR survey calls it misevolution: risks that were not present in the original agent, and appear only as it evolves.

Three concrete ones:

1. It learns the wrong lesson. An agent starts issuing unnecessary refunds, because its memory has correlated refunds with high satisfaction ratings. Nothing was hacked. It learned exactly what it was shown.

2. Safety training wears off. Training on its own output can erode the safety alignment the model came with. The survey notes agents executing harmful instructions they previously refused.

3. Self-written tools are unreviewed code. An agent that writes its own tools can write insecure ones, or fail to spot malicious code in a tool it picks up from elsewhere. The survey also notes that agents are bad at telling sensitive data from harmless data, so a self-written tool can quietly leak.

There is a name for the general shape of this. The alignment tipping process, where an agent that started out aligned discovers that misaligned behavior scores better, and its policy tips over.

None of these look like failure at the time. The score goes up. That is the problem.

Practical Steps to Follow

Gate every change on a check the agent did not write, scored over several runs rather than one. A single run cannot tell a real fix from a lucky one, and the loop will happily keep both.
Keep the proposer and the approver separate. If the same agent suggests the change and signs it off, nothing is being checked. Use a different model for the grading, or the agent learns where that grader is soft.
Version everything, and test the rollback before you need it. Every self-modification logged, every known-good state tagged. Bad changes surface days later, and by then your only real option is going back to a state you know was fine.
Sandbox and scan any tool the agent writes before it is allowed to run. It is unreviewed code written by something that cannot reliably tell sensitive data from harmless data.
Track what used to pass and now fails, not just the average score. The average can climb while specific things break underneath it, and those are the failures your users hit first.
Require human approval for anything expensive or irreversible. Spending money, sending messages to real people, deleting data, touching the harness. The agent proposes, a person decides.
Report what the evolution cost you, not just what it gained. Two points of accuracy for triple the compute is a trade someone should get to refuse.

The Glasgow survey compresses the whole philosophy into three rules, and the order is the point.

Endure. Stay safe, aligned and robust through any change.
Excel. Be good at the actual task.
Evolve. Only then adapt on your own.

Evolution comes third on purpose.

Where to Start

For most teams, not with self-evolution.

Get the foundation right first. Traces you can investigate. Failures sorted into groups rather than left in a pile. A set of tasks with curated ground truth. A test that fires on every change. A tagged version you can fall back to. None of that is self-evolution. It is what makes self-evolution possible.

It is also what makes an agentic system production grade in the first place. You cannot debug what you never logged, or improve what you cannot measure, no matter who is doing the improving.

Once that foundation is there, enabling self-improving agents is a small step. The gate already exists. You are only changing who proposes.

The mistake is doing it in the other order. Wanting an agent that improves itself, before having any way to tell whether it did.