Octomind 0.33.0: It's the Framework, Not the Model

For two years the only question that mattered was which model. That question is mostly settled — everyone calls the same frontier models through the same APIs. The model is becoming a commodity. What separates a coding agent that finishes the job from one that burns your budget looping isn't the model anymore. It's the framework that steers it — how it manages context, catches its own failure modes, verifies its work, and recovers when it goes off the rails.

Octomind is a bet on exactly that. 0.33.0 is the release where we rebuilt the framework, the Supervisor, and started measuring it. Held the model fixed, changed only the harness, and watched the results move: same model, 93% of tasks solved vs 71%, at a lower cost per solved task. That single fact is the whole argument of this post.

The model is commoditized. The harness is the product.

Give two agents the same frontier model and the same task and you'll get wildly different outcomes. One reads the right three files and edits them. The other greps the same term five times, re-reads a file it already has, confidently invents a function that doesn't exist, declares victory, and never runs the test. Same model. The difference is entirely the scaffolding around the model, the part that decides what goes into the context window, notices when the agent is spinning, and refuses to accept "done" without proof.

That scaffolding is where the real engineering is now. It's context engineering, loop detection, self-verification, recovery, the unglamorous control plane that turns a capable model into a reliable agent. It's also the part that's measurable: if you fix the model and only change the framework, any difference in outcome is the framework's doing.

So that's what we did. The rest of this post is the new framework, and the numbers that show it works.

The Supervisor, recapped

The Supervisor runs beside the agent loop, never in your transcript. Every turn it reads two free signals (the agent's self-reported state (exploring / progressing / blocked / done) and a set of deterministic counters over the tool stream), and only when those two disagree does it spend a model call. Cheap signals do the gating; the model is woken rarely. Everything it injects is an advisory system note, never a silent rewrite of your context.

0.31 shipped that idea with one detector and a verify-gate. 0.33 makes it a real framework.

Six detectors, almost all free

Catching the agent's common failure modes shouldn't cost a token. In 0.33 these are counters over the tool stream, not model calls:

Truncation, dedup, drift, and over-sequencing are new in 0.33. Five of the six are pure counters, zero model cost. Only drift spends anything (one embedding per sizable result, scored against a self-referential centroid of recent on-task work), and it's off by default.

Steering, rewritten against the research

Catching a failure mode is half the job. The message you inject to break it is the other half. And a bad message is worse than none, because the model learns to tune out <system-reminder> blocks entirely. So we rewrote every steer in 0.33 against what the literature actually says about redirecting a model mid-task:

• Positive-forward, not prohibition. "Do Y instead" beats "don't do X." Negation is the construct LLMs handle worst. They miss it, misread it, and larger models often get worse (inverse scaling). Each steer leads with the concrete next action and puts it last, in the recency slot.
• An escalation ladder, not repetition. Re-sending the same words loses force fast. When a signal re-fires, the steer reframes: soft diagnostic → directive with a self-check → firm "different approach, or report blocked" → persistent "small adjustments aren't working; change the path." It never softens once the agent is stuck.
• Recite the goal. On long, compacted sessions the objective is buried mid-transcript where attention is weak. The Supervisor re-emits a tiny goal block at the context tail each turn, pure reuse of the existing anchor, no model call. The live goal stays in view.
• Name the conflict. The one case the Supervisor was built for: the counters say no progress while the agent insists it's progressing. 0.33 surfaces that contradiction directly instead of a generic nudge.

Each detector also gets its own framing: a loop ("same result"), a truncation ("output is capped"), and drift ("off the goal") are different problems, and saying them differently is what stops the agent from discounting all of them at once.

Evidence-bound claims

Agents state confident facts about your code that aren't true, the AI slop tax in miniature. 0.33 adds an opt-in contract: when the agent asserts a load-bearing fact — a path, signature, value, behavior — it backs it with a verbatim quote it actually saw in a tool result, in a fixed [evidence: <locator> «exact text»] form. A deterministic check confirms each quote really occurs in some tool output. Fabricated citations are caught for free and re-grounded through the verify-gate's bounded re-run. The agent has to show its work.

A circuit-breaker for stuck loops

A steer is advisory. The agent can ignore it, which means in the worst case it can ignore it forever. 0.33 adds max_consecutive_steers: after N rounds that emitted a steer without the agent breaking out, the turn hard-stops instead of spiraling. Not hypothetical. In the benchmark below, 0.32.0 (no breaker) hung for over an hour on one task. 0.33.0 caught it.

octobench: measuring the framework, not the model

Claims about agent quality are usually vibes. We wanted a number, so we built octobench — an open benchmark harness whose entire design is to hold the model constant and measure the framework.

How it works. octobench runs an agent end-to-end against SWE-bench-Live — real, post-2024 GitHub issues, contamination-resistant. The agent fixes the repo inside the instance's own Docker image, and the instance's own test suite is the verdict: FAIL_TO_PASS must flip, PASS_TO_PASS must stay green. No self-report, no LLM grading the pass/fail — real tests as the oracle. It captures normalized token telemetry per run, and a separate judge scores solution quality on the runs that pass.

The headline metric is cost-per-solved-task — total tokens (or dollars) ÷ tasks actually solved. It's the only honest number: raw "tokens per call" rewards an agent that gives up faster, which is strictly worse. Cost-per-solved punishes exactly that, and rewards spending the budget on work that lands.

The result — 0.32.0 vs 0.33.0, same model (glm), identical config, only the framework changed:

0.33.0 solves more (+22 points), its passing solutions are rated higher (+18 on quality), and because it converts work into more passes, the cost per solved task falls ~10% — even though it spends a touch more per individual task. Time is a wash.

What it is not is "fewer tokens per call." Per-task token use is mixed — 0.33.0 is much cheaper on some tasks (one dropped from 7.07M → 4.27M tokens; another from 0.64M → 0.20M), and more expensive on a couple of hard ones it now actually finishes. The win isn't shaving tokens off every call. It's not lighting the budget on fire in a loop — and the framework, not the model, is what changed.

Honesty about scope. This is a focused study (7 SWE-bench-Live tasks, 2 runs each), not a thousand-instance leaderboard. It's enough to show the direction is real and consistent (and to kill a scary single-run fluke that briefly suggested 0.33.0 was 3× more expensive; it wasn't. That was one lucky-cheap 0.32.0 run, erased by averaging). The harness, the frozen baseline, and the exact per-task numbers ship in the Octomind repo under bench/, so you can reproduce every figure.

What's next: Octomind vs Claude Code, Codex, opencode

Octobench matters beyond our own changelog because it fixes the model, the task, and the oracle. The only variable left is the framework, which means it can compare different agents on a level field, not just different versions of Octomind.

octobench already treats the agent as a pluggable provider: Claude Code, Codex, and Octomind run through the same harness today, on the same SWE-bench-Live tasks, judged by the same real tests, with the same normalized telemetry. opencode and other clients are next on the adapter list.

That's the benchmark we actually want to publish: given the same model, which agent framework solves the most real issues at the lowest cost per solved task? "Which model is smartest" you can read anywhere. Same model, same tasks, same tests. Pure framework. The 0.32→0.33 numbers above are the first cut, run against ourselves to prove the methodology. The cross-client board is what we're building toward, and we'll publish it openly — harness and all — because a benchmark you can't reproduce is just marketing.

If the model is commoditized, this is the scoreboard that's left.

Release notes — 0.33.0

✨ Features

• Supervisor: truncation, dedup, drift, and over-sequencing detectors — the full free-signal suite over the tool stream.
• Supervisor: rotating, research-grounded steering — escalating per-signal framing, anti-habituation, goal recitation, conflict framing.
• Supervisor: evidence-bound claim verification — «quotes» checked deterministically, for free.
• Supervisor: goal recitation + mutation gating — re-anchor the goal at the context tail; refuse a done claim when code changed but no check ran.
• Supervisor: steer circuit-breaker (max_consecutive_steers) — hard-stop a turn that keeps getting steered without breaking out.
• Session: system-managed user messages — supervisor notes, recall, and continuations tracked as synthetic turns; excluded from task capture and compaction.
• Session: tool-call deduplication upstream — duplicate results elided to a placeholder before they reach the model.
• Config: orchestration MCP server — tap and schedule tooling migrated onto the new runtime module.
• bench: token-efficiency benchmarking suite — the octobench harness above, committed and reproducible.

🐛 Fixes & hardening

• Prevent greedy quote stripping; resolve YAML escaping asymmetry in supervisor parsing.
• Exclude synthetic user messages from task selection during compaction.
• Refine tool-response deduplication and truncation-notice clarity.
• Consolidate steering signals across parallel tool batches (one steer per round, by priority).

Upgrade

# Homebrew (macOS / Linux)
brew install muvon/tap/octomind

# or from crates.io
cargo install octomind

Prebuilt binaries for every platform are on the releases page. Your existing config keeps working — the new Supervisor fields ship with sensible defaults (drift and over-sequencing off by default; the rest on). If you've run the Supervisor since 0.31, you get the new detectors, the rewritten steering, and the circuit-breaker for free.

The Supervisor was a promise in 0.31: a session that finishes what it starts. 0.33 is where we can prove it — and where we start measuring whether any agent, ours or not, actually keeps that promise. The model isn't the moat. The framework is.