Self-Improving Agents: Google's Harness² (harness2) vs Octomind

Self-improving agents promise to get better at your work without anyone retraining the model. The same coding agent can take the same request twice and fail in two different ways. One system responds by rerunning that request with edited instructions. Another waits for similar sessions and tests whether a learned rule changes outcomes.

I build Octomind, so I have a stake in one side of this comparison. The useful question is narrower than which system is “more self-improving”: which layer changes, what evidence accepts the change, and how much extra inference does the loop spend?

The short answer: Harness² (also typed harness2) improves the harness around fixed model weights for the current task, then stores supported lessons for later tasks. One k=1 step uses about four task executions by construction, plus editor work. Octomind's evolution learns across ordinary sessions and promotes generated behavior only after a concurrent, execution-verified trial clears statistical and cost checks. Pick Harness² for research runs where extra probes on the same task are useful; pick a cross-session loop when replay is expensive or can repeat side effects.

What are self-improving agents, and what does Harness² change?

A self-improving AI agent changes something that affects future behavior based on what happened before. Harness² changes the execution scaffold around a frozen model: instructions, memory, tools and orchestration. It doesn't train new weights.

The paper calls the process recursive agent harnessing. “Recursive” has two scopes: improve the current task's harness, then improve the mechanism that will do harness editing on later tasks. The Harness² paper (2026), from Google Cloud AI Research and Northwestern University, exposes eight files an editor may rewrite. Four are always in context (the system prompt, guardrails, memory, and notes on how to use tools); four load on demand (skills, standalone scripts, subagents, and a TypeScript plugin that runs inside the tool loop).

The project is a framework wrapped around agents, not a competing foundation model or a complete end-user agent.

The eight pieces matter because “it learns” can mean very different things. A system that saves a user preference, one that tunes a prompt on a labeled dataset, and one that rewrites its own source code all make different promises.

If the word harness is new, our harness engineering explainer describes the guides and checks around an agent. And for a smaller unit of reusable behavior, see how agent skills activate.

How does Harness² improve one task without a hidden answer key?

It compares executions of the same task under matched conditions, then asks a composer to make a selective harness edit from the behavioral differences. The contrast is evidence about how the agent behaved, not proof its answer is correct.

At k=1, the procedure is:

1. Run the base harness once as a reference.
2. Propose two candidates: one for an invariant agent weakness such as planning or tool use, one for a domain-specific issue.
3. Probe each candidate by running the same task again.
4. Compose a successor harness and execute it for the final deliverable.

That's about four task executions, a derived count from the paper's procedure, not a price quoted by the authors. Each execution also has its own model-token cost, while proposal, composition and reflection add editor sessions. Increasing k scales the exploration budget. Parallel probes can lower elapsed time when capacity allows, but they still use compute.

The composer reads the reference run, the candidate harnesses, their diffs and the execution evidence, but never verifier scores or benchmark rubrics. That avoids directly optimizing on a hidden answer key. It also means a candidate that looks behaviorally promising can still be wrong.

With a larger budget (k > 1), several compositions run and a final selector picks one deliverable from anonymized outputs; in one failure case from the paper, it chose an output scored 55 over one scored 95. Stochastic runs and shared model blind spots are weaker evidence than a direct test.

After the task, a reflector stores the adopted harness with an EXPERIENCE.md note in a per-domain library, but only when execution contrasts support a lesson; with too little evidence it adds nothing. Later proposals can reuse that record.

In the paper's ablations (Gemini 3.5 Flash in OpenCode, k=1), the full method scored 77.5 on WorkBuddy-Bench and 36.0 on JobBench; removing the library dropped that to 74.4 and 34.2, removing the invariant/domain split to 74.6 and 32.3, and removing composition to 75.9 and 34.1. Those results support the design in the studied setup; they don't establish a correctness guarantee or transfer to every task.

What did Harness² find, and how much should I trust the result?

The paper reports gains over base harnesses on three professional benchmarks, with the largest one-step gains of 11.5 WorkBuddy-Bench points, 4.5 JobBench points and 11.8 percentage points on LAB Partial pass rate. Those are benchmark outcomes, not a claim that every task gets better.

The paper's setup uses GLM-5 and Gemini 3.5 Flash in OpenCode and Codex harnesses; Claude Opus 4.8 edits harnesses by default. The benchmark guide (2026) describes WorkBuddy's office, web and code tasks, JobBench's professional tasks, and Harvey's legal tasks with rubric-based grading. These are distinct domains and graders, which makes the cross-benchmark result more useful than a single coding score.

Every benchmark in the study is public, and JobBench's per-task grading rubrics sit on Hugging Face. The experiment guide (2026) says executor network access is unrestricted and documents a Codex run that downloaded a JobBench task rubric without instruction. The repo recommends blocking access to benchmark hosts during evaluation. That's a benchmark-integrity concern, not evidence that every reported score is invalid, but it belongs beside the result.

The authors also name the failure mode: a model judging behavioral contrasts can select the wrong output; an incorrect premise can propagate into multiple candidates; and extra recursion costs inference. That's why the strongest use case is an experiment with bounded budgets, a clear task stream and a human-readable record of why the edit was adopted.

Is Harness² recursive self-improvement in AI?

Yes, in the paper's broad harness-recursion sense; no, if by recursive self-improvement you mean an AI system improving its model weights or designing more capable successor models. Harness² keeps the executor model fixed and changes the scaffold around it.

Search for what recursive self-improvement in AI means, and results often describe the stronger idea of a system repeatedly upgrading the system that builds the next system. Here, recursion happens in harness design: a task's behavior informs edits, and the editing process leaves reusable lessons for future edits. The paper's definition (2026) explicitly describes a frozen base model and persistent harness changes.

That distinction helps answer “does the agent learn from itself?” It can learn behaviorally in the limited sense that its future context and tools change. It doesn't mean its underlying model weights learned from the conversation. For a prompt-optimization loop scored by graders (rule checks, similarity to a source, and an LLM judge), the OpenAI cookbook's GEPA example is a different design point.

Harness² vs Octomind: which self-improvement loop fits?

Harness² spends extra executions on the current task and uses an LLM composer to decide whether edits are supported. Octomind learns across normal sessions and requires a statistical treatment/control result before generated behavior can become active. The comparison below was checked 2026-10-19; Octomind's entries come from the Octomind learning docs, including changes that landed on master and are not released yet.

They aren't apples-to-apples benchmark competitors, and there are no benchmark claims for Octomind's new evolution features.

Choose Harness² if

• The same task is safe to rerun. You can afford about four executions for a first-step comparison, then raise k if the added compute is justified.
• You want task-specific adaptation now. The final deliverable can benefit from a harness tailored to this task, while the experience library preserves lessons for later work.
• You can inspect the evidence trail. Review candidate diffs and execution contrasts because behavioral preference isn't a correctness oracle.
• You have the benchmark setup. Linux, model credentials and benchmark dependencies are part of the research workflow, not an incidental install detail.

Choose a cross-session loop if

• The same task has side effects. Avoid having a self-improving coding agent replay real repository actions, deployments or workflow mutations just to compare prompt candidates.
• You can collect matched outcomes over time. A randomized control arm gives a more defensible test than “the agent said the new rule helped.”
• You can wait for evidence. A candidate needs enough verified sessions to clear the trial gate; it doesn't improve the session that generated it.
• You want changes to stay controlled. Shadow, trial, active and retired states make the promotion lifecycle visible.

For background on writing the evaluation and verification signals a learning loop needs, see our LLM evaluation workflow. It's the difference between measuring a change and merely recording that it happened.

What is the practical cost of a self-improving AI agent?

For Harness², count all model calls in the refinement loop, not just the final answer. For Octomind, the cost is delayed across shadow and trial sessions, with promotion also conditioned on its API-call budget.

Harness²'s repo recommends beginning with a small task selection and checking model pricing. Its setup guide (2026) requires Linux and Python 3.12 for benchmark runs, Node.js 20+, uv, Google Cloud CLI and Vertex configuration; Pandoc/Poppler and Docker are needed for specific benchmarks. That's a practical research stack rather than a one-command local agent install.

Estimate the cost of repeated execution against the value of a better answer on this task. Report solves next to total cost and time, the way our real pull request benchmark write-up does, so a loop that adds executions has to pay for them in results.

What Octomind took from Harness², and what it left out

We read the paper and adopted the three parts that fit a cross-session loop. All three landed on master this week and ship in the next release; none has a measured gain yet.

What we left out:

• Rerunning the same task to compare candidates. About four executions per step works against token efficiency, and a real session can change repositories and external systems. The trade-off is real: cross-session trials take longer to gather evidence and never improve the current task on the spot.
• Adopting an edit on a model's judgment alone. A generated behavior still has to win a randomized trial.
• Recall that loads behavior. Naming a skill to a session in the control arm would hand it the treatment and break the comparison.

FAQ

Does Harness² change the model weights?

No. The executor's weights stay fixed while Harness² edits the surrounding files, such as prompts, memory, tool guidance and scripts. The Harness² paper (2026) calls this recursive harnessing.

How many executions does Harness² add?

At k=1, about four: one reference run, two candidate probes and one composed run. That's a derived count from the Harness² procedure (2026), before counting the editor sessions; larger recursion budgets add more inference cost.

Does Harness² use ground truth to decide which edit wins?

The improver doesn't see verifier output or benchmark rubrics. It compares run behavior, which can help identify an edit but can't prove the output is correct; the paper's failure analysis (2026) documents a case where the selector preferred a lower-scoring result.

Is Harness² open source?

The repository is licensed Apache 2.0, and the Harness² README (2026) says it is not an officially supported Google product. The benchmark and dataset licenses can differ from the framework license.

What should you try first?

Start by writing down the behavior you want to improve and how you'll recognize a regression. If you can replay a task in isolation, run a bounded Harness² experiment and inspect its diffs. If improvements must accumulate from ordinary work, keep them in shadow until a concurrent, verified comparison supports promotion.

My rule is to match the loop to the unit of evidence. Per-task contrasts answer, “Did this harness change this run?” Randomized sessions answer, “Does this behavior improve the class of future work it applies to?” Neither answers every question, and neither should turn an unverified hunch into durable behavior.

Next, we measure Octomind against Codex on Harvey's Legal Agent Benchmark with the same ChatGPT model and LAB's standard judges. We have no result to report yet.