On October 2 I gave the same model the same one-line bug six times. Three runs shipped a broken fix. Three didn't. The model, the prompt, and the code were identical every time; the only difference was what ran around the model.
The prompt was the kind of thing you'd actually type:
full_name in names.py runs the first and last name together with no space. Fix it.Every run started the same way: find full_name, read the file, make the obvious edit. Every run wrote nearly the same first message — "Fixed: full_name now joins first and last name with a space." Half of them stopped right there, with a test still failing. The other half got that failing test pushed back at them, handled the case they'd missed, ran the suite, and passed a completion check before they were allowed to finish.
Harness engineering is the practice of designing everything around an AI model — the instructions it starts with, the tools it can call, the checks that run after it acts, and the rules for when it may say "done" — so a capable model reliably produces work you'd merge. The model writes the code. The harness decides whether that code ships.
What is harness engineering?
Harness engineering is building the system that turns a model's output into dependable work: guides that steer the agent before it acts, sensors that check what it did, and a loop that keeps going until the sensors agree. Birgitta Böckeler's piece on martinfowler.com puts it in one line: the harness is "everything in an AI agent except the model itself."
The term went mainstream in February 2026, when OpenAI published Harness engineering: leveraging Codex in an agent-first world. Ryan Lopopolo's team spent five months building an internal product with no manually written code — roughly a million lines across about 1,500 pull requests — and summed up the job in four words: "Humans steer. Agents execute." The engineers stopped writing code and started designing the environment the agent writes it in.
Addy Osmani's O'Reilly essay has the line I'd tape to every monitor: "A decent model with a great harness beats a great model with a bad harness."
None of this was news to us. We've been building Octomind on that bet since 2025: the model is a commodity you rent, and the harness is the product you own. What changed in 2026 is that the bet got a name, and we got numbers.
Prompt engineering vs context engineering vs harness engineering
The three are nested. Prompt engineering tunes the words you send. Context engineering tunes everything the model reads on each turn. Harness engineering tunes the loop itself: what runs before and after every turn, and what counts as finished.
Here's how each one would have tried to fix my six runs:
| Layer | What you change | The fix for full_name | Where it breaks |
|---|---|---|---|
| Prompt engineering | The instruction | "…and handle an empty last name" | You have to know the bug before you ask |
| Context engineering | What the model sees each turn | Put the test file in context, or a rule in AGENTS.md | The model can still read the rule and skip the check |
| Harness engineering | What runs around the model, every turn | A script runs the tests after every turn and reports failures | Only as good as the tests it runs |
Prompt work still matters. Context engineering matters more — it's where most of the cost lives. But both are requests the model can ignore. A harness check isn't a request. The failing test comes back whether the model planned for it or not.
The experiment: same model, two harnesses
I ran six non-interactive sessions with Octomind 0.55.2, all on GLM-5.3, from a two-file fixture: a names.py that glues the names together with no space, and a test_names.py with two tests — the obvious one ("Ada Lovelace") and the trap (full_name("Ada", "") should return "Ada", with no trailing space).
The bare harness was a role with three octofs tools (view, text_editor, shell), the supervisor switched off, and nothing else. The harnessed version switched the supervisor on, completion gate included, and added one guardrail validator — a script Octomind runs at the end of every assistant turn:
#!/usr/bin/env python3
import json
import subprocess
import sys
json.load(sys.stdin)
run = subprocess.run([sys.executable, "-m", "unittest", "-v"], capture_output=True, text=True)
if run.returncode:
print("The project test suite failed. Fix names.py without weakening test_names.py.")
print(run.stdout + run.stderr)
sys.exit(1)Registered in .agents/guardrails.toml:
[[validator]]
name = "name-tests"
roles = ["completion"]
script = ".agents/validators/tests"Results across three runs per arm:
| Harness | Tests passing at the end | Tool calls per run | Cost per run |
|---|---|---|---|
| Bare | 0 of 3 | 2–3 | $0.0021–$0.0054 |
| Harnessed | 3 of 3 | 5–6 | $0.0095–$0.0117 |
Every bare run made the same edit, return first + " " + last, announced "Fixed," and exited with test_single_name failing. The model wasn't careless by its own standards. Nothing in its loop ever asked it to look.
In every harnessed run, the model's first answer was the same wrong one. Then the validator fired, and this landed in the session as a guardrail_validator event (trimmed):
<validation validator="name-tests">
The project test suite failed. Fix names.py without weakening test_names.py.
test_full_name (test_names.NameTests.test_full_name) ... ok
test_single_name (test_names.NameTests.test_single_name) ... FAILThe model fixed the edge case, ran the test suite itself, and reported both tests passing. Then the completion gate checked that claim against the request and the recorded tool output before the run was allowed to end.
The harness roughly tripled the cost: about a cent per run instead of a third of a cent. It also turned 0 working fixes out of 3 into 3 out of 3. I'll take that trade on every task I'd otherwise have to re-check by hand.
One more thing the runs taught me, and it's the most important part. Run one fixed the trap with if last:, which returns " Lovelace" — with a leading space — when the first name is empty. Runs two and three used .strip(), which would also eat whitespace you meant to keep. Both versions pass both tests. A harness enforces exactly what its sensors measure, and nothing more. If the test suite doesn't ask, the harness can't either.
What harness engineering looks like at benchmark scale
Six toy runs show the mechanism. Our benchmarks measured the size of the effect: same model, better harness, more tasks solved for less money.
In our coding agent benchmark on 25 real pull requests, Octomind and opencode ran the same model, glm-5.2, on the same endpoint at the same prices. That's a pure harness A/B. Octomind solved 24 of 25; opencode solved 19. Octomind's total cost was $63.43 against opencode's $129.54. On one task, opencode pushed 30M input tokens through the model where Octomind needed 8M. The cases opencode dropped were exactly the deep, multi-trap ones where nothing questioned an early "done."
Earlier, in Octomind 0.33, we held the model fixed and changed only the supervisor: the solve rate on SWE-bench-Live tasks went from 71% (10/14) to 93% (13/14), and cost per solved task dropped from $3.24 to $2.90. It was a small study, 7 tasks run twice each, and we said so at the time. But the direction was the same as in the PR benchmark.
The research says it too. In the SWE-agent paper, the same model resolved 18.0% of SWE-bench Lite with an edit tool that returned lint feedback, 15.0% without the feedback, and 10.3% with shell-only editing (arXiv 2405.15793). The editing interface alone was worth nearly half the score. That's why we designed octofs around what the model needs on the next call after a failed edit.
Guides and sensors: what a harness is made of
A harness has two kinds of parts. Guides act before the agent does, to make the first attempt better. Sensors act after, to catch what the first attempt got wrong. Either kind can be computational (a script, a counter: fast, free, deterministic) or inferential (a model judging a model: slower, costs money, catches what scripts can't). That's Birgitta Böckeler's framing, and it's the most useful one I've found.
Here's how that maps onto the parts of Octomind I'd actually reach for:
| Surface | Guide or sensor | Computational or inferential | What it catches |
|---|---|---|---|
AGENTS.md | Guide | Inferential — the model reads it | Project conventions the model would otherwise guess |
[[pipe]] | Guide | Computational | Rewrites or rejects input before the model sees it |
[[guard]] | Guide | Computational | Blocks a tool call before it runs |
[[hook]] and [[validator]] | Sensor | Computational | Failures in a tool result or at the end of a turn |
| Loop and no-progress detectors | Sensor | Computational | Identical calls for 3 rounds; 5 rounds with nothing new |
| Completion gate | Sensor | Inferential | A "done" the evidence doesn't support |
The ordering matters more than the inventory. Our supervisor runs on one rule: free signals gate the model, and model calls are rare. Counters run every turn at zero cost; the gate wakes a model only when the agent claims it's finished. To make those yes-or-no judgments faster and cheaper still, the supervisor can hand them to a model built for calibrated decisions — the story is in our Jev write-up.
Only the top row is something the model can choose to ignore. The system prompt belongs up there with it: worth writing well, and still the weakest lever you have. Every row below it runs whether the model cooperates or not.
Claude Code harness engineering uses the same parts
If you run Claude Code, you're already doing harness engineering whether you call it that or not. CLAUDE.md is a guide. A PreToolUse hook that exits with code 2 blocks the tool call — that's a guard. A Stop hook that exits with code 2 stops Claude from finishing and sends it back to work — that's the validator from my experiment, under a different name.
The skill transfers across tools because it isn't really about any tool. Every time the agent fails, you decide which layer should catch that failure next time, then build the cheapest check that will.
How to learn harness engineering: start from a failure
You learn harness engineering by fixing real failures one at a time. Here's the loop we run:
- Collect a failure. A transcript where the agent said "done" and wasn't. Not a hypothetical — a real miss.
- Ask whether a script could have caught it. A test, a linter, a type check, a schema. If yes, make it a sensor that runs every turn. Computational checks are free and never get tired.
- Only if no script can, add an inferential check. A model judging the result against the request. Keep it rare and narrow.
- Re-run the same task with the same model. If the outcome doesn't change, your check isn't wired where you think it is.
- Touch the prompt last.
Osmani calls the result a ratchet, and his test for it is worth borrowing: "Every line in a good AGENTS.md should be traceable back to a specific thing that went wrong." The same goes for every guard and validator.
Measure cost per solved task, not tokens per call. A harness that adds a check will always cost more per run, like my experiment did. It pays for itself when the run actually lands.
FAQ
What is harness engineering in AI?
Harness engineering is designing the system around an AI model: the instructions and tools it starts with, the checks that run after each action, and the rules that decide when work is finished. The goal is reliability. A good harness makes a capable model's correct output the default instead of a lucky run.
What is OpenAI harness engineering?
It's the approach OpenAI described on February 11, 2026, in a post by Ryan Lopopolo. A small team built an internal product over five months with no manually written code, roughly a million lines across about 1,500 pull requests, by designing the environment, intent, and feedback loops Codex worked inside. Their summary: "Humans steer. Agents execute."
What's the difference between prompt, context, and harness engineering?
Prompt engineering optimizes the instruction. Context engineering optimizes everything the model reads on each turn: files, tool results, memory. Harness engineering optimizes the loop around the model: checks before and after actions and the definition of done. Each contains the one before it.
How do I learn harness engineering?
Start with a real failure from your own agent, then add the cheapest check that would have caught it — a test or linter first, a model-based check only when a script can't do the job. Re-run the same task on the same model to confirm the check works. Repeat. Your harness grows from actual misses, not guesses.
The next model will still say "Fixed" too early
Models will keep getting better, and every one of them will still sometimes announce a fix one turn before it's true. That's not a flaw you can prompt away. Your job is deciding what your loop does when it happens. Mine runs the tests.
