Octomind vs Codex CLI: Same Solves, 25% Fewer Tokens Across 150 Runs

Octomind and OpenAI's Codex CLI can run the same model, gpt-6.1-sol, on the same ChatGPT plan. Put both on the same coding tasks and the model stops being a variable. What is left is the model harness: the program around the model that gives it tools, feeds it context and decides when the work is done.

Over five paired rounds of GOLD, our new benchmark of real pull requests, the two clients solved the same amount of work: each passed 38 of 65 one-shot runs (one pull request, one instruction) and all 140 scored turns of two 15-turn sessions. Octomind used 0.75× Codex's tokens and 0.87× the Codex cost per task, and took 1.08× the time. A week earlier, before the fixes in this post, Octomind used up to 3.1× Codex's tokens and 2.3× its time on its six worst tasks.

This post is how we got there. Benchmarking against Codex turned up a string of bugs in Octomind, and together they taught us one thing: gpt-6.1-sol behaves like a model trained inside Codex's harness, and it works best when a harness lets it work in the shapes it learned there, such as one tool call per reply, or an edit and its test in one shell command. Most of the bugs were places where Octomind's tools or messages asked the model for a different shape.

I make Octomind, so weigh my framing with that in mind. The final results come from five paired rounds on one machine, November 8 to 10, 2026; the bug numbers come from the runs that led up to them. The tasks, scripts, per-round records and raw latency samples are public in the octobench repository.

What GOLD is

GOLD is a frozen set of 13 one-shot tasks and two 15-turn sequences, built from real pull requests merged into maintained open-source projects: Vue, React, DuckDB, mypy, Polars, Pydantic, Wasmtime, Nushell, Cargo and ESLint. It was frozen on October 15, 2026, and lives in octobench, our open-source benchmark harness. GOLD compares clients on the same model, not models against each other. The GOLD reference lists every item, and the GOLD protocol fixes how rounds are run and published.

Tasks newer than the model's training data

A public benchmark ages. Once a fix sits on GitHub, the next model may have trained on it, and its score starts measuring memory instead of work. SWE-bench showed how far that drift goes; we wrote about it in SWE-bench is broken.

OpenAI lists the knowledge cutoff of both gpt-6.1-sol and gpt-6-astra as April 30, 2026. Every GOLD item was merged after that:

• 11 of the 13 one-shot tasks were mined for GOLD from pull requests merged between September 8 and October 12, 2026.
• The React task (merged July 28) and both sequences (merged July 3 to August 20) were carried over from our earlier cases. React stayed as the suite's one light task, because React is too widely used to leave out.

So even the oldest fix is two months newer than the models' training data. The agent also runs on a sealed network that reaches only the model endpoint, so it cannot look the fix up during a run. Changes whose authors disclosed AI-generated code or tests were left out.

How a task is judged

Only the maintainers' own tests decide pass or fail. They are restored byte for byte from the merged commit and hidden from the agent. Before an item was admitted, it had to pass four gates:

1. The code before the pull request failed the tests, and the merged code passed them, three times in a row.
2. An audit traced every test assertion to the task's instruction or a visible convention in the repository, so no test checks a requirement the agent was never told.
3. A plausible fix that misses the main behavior still fails.
4. Test runners and harness files are restored from the merged commit before validation, so an agent's changes cannot manufacture a pass.

The instructions keep the three ways real work arrives: an acceptance spec, an issue report, or a short request from a maintainer. Candidates were then calibrated on gpt-6.1-sol with both clients, and only the heavy ones stayed: roughly 2M or more tokens per run, or consistently missed. Light tasks that every client always passed were dropped.

One-shot tasks and sequences

GOLD has two kinds of items.

• A one-shot task is one pull request. The agent starts from a fresh checkout of the code just before the merge, gets one instruction and has 90 minutes. It passes if the hidden upstream tests pass.
• A sequence is 15 consecutive real pull requests from one project, sent as 15 turns of one persistent session. The agent keeps its conversation and its own earlier changes, like a developer working through a backlog for an afternoon. Each turn is judged by that pull request's tests. A sequence runs long enough to outgrow the model's context window, so it also tests how the client compacts, or summarizes, older conversation.

Two sequence turns are unscored: DuckDB turn 6 and ESLint turn 7. No upstream test checks their core behavior, so a wrong fix could pass. For those turns the merged change is applied without asking the agent, so later turns start from the right state, and they count toward nothing. That leaves 14 scored turns per sequence, 28 per round and 140 over five rounds.

How we run the comparison

Both clients run at the same moment

The model behind an API is not a fixed thing. Providers update it, load rises and falls by the hour, and a slow afternoon can add seconds to every call. A comparison run on different days measures those changes as much as the clients: a week-apart comparison of ours once showed 11.9 s per model call against 7.2 s, which measured the provider, not the clients.

So GOLD runs the two clients in parallel. The same task starts for both at the same moment, each in its own container on its own six of the machine's twelve cores, and the core sets swap every round. Whatever the provider does during a task, a model update, a slow hour or a busy region, hits both clients at once and cancels out of the ratio.

Everything else held equal

• Same model and Codex reasoning effort. Both run gpt-6.1-sol at reasoning effort low, which is Codex's default for this model, each on its own ChatGPT login.
• Same instructions. Both get the same system prompt, byte for byte, synced from one file.
• Five rounds. One run says little. Across five rounds, Octomind's own token use on the Vue transition task ranged from 6.5M to 17.0M.
• Like against like. Ratios are Octomind ÷ Codex per task and round. The "solved by both" rows compare only runs where both passed, because a run that fails early is cheap for the wrong reason.
• Checked traces. Every agent trace is audited for tools the run forbids; none of the 1,684 traces used one. An LLM judge also scores each run, but only as a diagnostic; it never decides pass or fail.

One difference cannot be held equal: the network route each client is allowed to use. On a ChatGPT plan, a request through Octomind's route takes about 1.1× as long as the same request through Codex's. The route latency section at the end has the measurement.

The bugs we found in Octomind

Each fix below came from reading transcripts until we found the call where a run went wrong, then checking the fix with paired runs. All of them ship in Octomind 0.70.1 and octofs 0.23.3, Octomind's file and shell server, except three settings in our benchmark config, marked below.

The model's own reasoning was lost between requests

On a ChatGPT plan the client keeps the conversation, not the server. It must send back the model's earlier output exactly: each message, its reply type, and the encrypted reasoning. Octolib, the model library under Octomind, flattened the messages into one text, dropped the reply type, dropped reasoning after text-only replies and copied it onto every tool call. The model was reading a history it had never written. Given the same task, Octomind's model reasoned about half as much per run as Codex's.

Octolib now sends the output items back verbatim, the way Codex does (octolib 5cfd1a1, Octomind d182e2de, released in Octomind 0.64.0). On the Cargo task, reasoning per run went from 1,182 tokens to 2,552, against Codex's 2,430, and passes went from 1 of 17 runs to 2 of 6. That is a small sample from October; the matching reasoning depth on that task is the clearer signal. Across the five final GOLD rounds, Octomind's model still produced about half Codex's reasoning tokens (96K against 196K per round) for the same number of solves, and we have not traced why. Cargo stays hard for both clients: neither solved it in the final rounds, and in our earlier runs Codex solved it once in 13 tries.

An edit and its test took two model calls

The largest token cost was not in what Octomind sent but in how many times it sent it. Every model call re-reads the whole conversation, so one extra call late in a task costs the full context again.

Comparing call by call, the pattern was the same on every task. Codex wrote a change and ran its check in one shell call, such as python3 - <<'PY' … PY && pnpm test …. Octomind's reply was one edit call; the test came in the next reply, after the edit result came back. Same work, twice the calls.

We first tried wording: the prompt told the model to put an edit and the command that checks it in the same reply. In 13 probe runs it changed nothing. The model did what the tools made natural: an edit tool returns an edit result, so the next step waits for it.

So we changed the tool. Octofs now takes an optional check on its edit tools: a shell command that runs right after the edit is written, exactly as the shell tool would run it, with its output appended to the edit result (octofs f6a27a68, released in octofs 0.23.0).

The model decides when to use it; the tool description says most edits omit it. It does not run when the edit fails. It exists only in sessions allowed to run shell commands: without shell access the edit tools do not offer check, and a request that uses it is rejected, so an edit cannot become a way around a denied shell (octofs 7c7023bf). In probes, 5 of 5 runs then edited and tested in one call.

On the Vue task, the paired round after this change used 14.4M tokens instead of 30.3M, and 140 model calls instead of 237. On the five tasks with a comparable pair before and after (pass against pass, or fail against fail), Octomind's token use relative to Codex changed as follows:

Those are single paired rounds right after the change. Over the five final rounds the medians were 0.45× on React, 0.97× on Nushell, 0.64× on Vue and 0.90× on Wasmtime. Pydantic is the exception: its median went back up to 1.57×, so the gain from this round did not hold there.

A status reply the model could not end

While a build ran in the background, Octomind told the model to reply with a brief status and no tool call. In three 300-second timeouts we traced, the connection was healthy and the model kept generating: a status, then an invented "[WAITING FOR BACKGROUND TASKS]" line, then whitespace about 14,000 times until the deadline. The instruction now tells the model to end its turn with a short status, because the result arrives on its own as the next message (Octomind 15b85899, released in 0.70.0). The next section explains why that one sentence mattered so much.

Requirements fell off the end of the checklist

Octomind turns each request into a checklist of stated requirements, which it repeats to the agent and later checks the finished work against. The list was capped at 24 items. Long, precise issue reports produce more than that, and 10 of 30 runs hit exactly 24. The items cut were always the last ones, and in one Vue run the cut paragraph, about hydration, was exactly the part whose two tests failed. The cap is gone, and each numbered requirement now stays one item, however many sentences it has (Octomind b3e20610).

The completion check saw the gap but never said so

On the hardest Vue task, Octomind's tests covered a Vapor Transition by itself, while the failing hidden tests covered Vapor content inside a VDOM Transition. Codex ran that interop test file 11 times; Octomind ran it zero times. The completion check had noticed: it reported the empty-slot requirement as not demonstrated.

It never told the agent, because the note was gated on the checker's own confidence score reaching 0.8, and across 40 runs that score never went above 0.7. The score also did not separate failed runs from passed ones. Whether any requirement was left unproven did: all 12 failed runs had one, against 13 of 28 passed runs.

We removed the score. Each unproven requirement now goes to a separate calibrated evaluation model with one yes-or-no question: did a recorded check exercise this specific case, not just the suite around it? The requirements it cannot confirm go into one note to the agent per turn, telling it to check that exact case (Octomind 10f4c8d5). The note itself is on by default. The evaluation model that filters it is off by default; our benchmark config turns it on with [decision] supervisor = true.

Waiting that was not the model

With tokens under control, most of the time gap left at that point was time when neither the agent nor the model was working.

• The completion check thought for too long. After the agent says it is done, a small supervisor model checks the work against the checklist. At reasoning effort low it spent about 46 s and 6.3K reasoning tokens per check. Without reasoning a check took 3–28 s, so our benchmark config runs it without reasoning.
• A silent stream waited out the full deadline. In 4 of 1,894 model calls, the response stream went silent and Octomind waited out the 300-second request deadline before retrying. Every other call finished within 77 s, and Codex's longest of 9,556 was 105 s. We set a 120-second silence limit in our benchmark config (stream_timeout_seconds, off by default), but in a later run a hang still lasted 336 s, so the limit did not catch it, and the cause is still open. In the five final rounds no Octomind run had a silent gap longer than 31 s.
• The run waited for a summary nobody would read. Octomind compacts long conversations in the background. When a one-shot run finished during a compaction, it waited for it before exiting: in 8 of 37 runs, 57–303 s after the final answer. The run now exits at once and hands the unfinished compaction to a detached process, which a later resumed session picks up (Octomind 5a9bedcd).
• Building the checklist blocked the first request. It now runs alongside the agent's first steps and lands at the next turn boundary (Octomind dbf55510).

What we tried that did not help

Not every idea held up in paired runs. These measured as no gain, and none of them shipped:

• Prompt wording for batching. Telling the model to send an edit and its test in one reply: 0 of 13 probes changed.
• Removing Octomind's parallel-tool-calls prompt block. Replies kept carrying about 2.4 calls.
• An apply_patch-style edit tool in octofs. Pass rate did not move. The patch syntax alone was not what the model missed.
• Higher reasoning effort. Medium instead of low did not change the pass rate.
• Turning the supervisor off. Cargo passed 0 of 3.
• Build parallelism. Octomind's model built DuckDB with -j 3, Codex's with -j 4 to -j 6. We suspected Octomind's prompt; replaying that decision with six cores, the model chose -j 6 six times out of six. Not the cause.
• Earlier octofs experiments. Longer shell timeouts instead of a progress heartbeat, a soft shell warning instead of a block, and batching advice in the server instructions all lost to the simpler version; the tool calling post has those numbers.

The pattern across all of them: changes to wording rarely moved the model, while changes to tool shape and runtime behavior did.

What the bugs taught us: the model learned Codex's harness

Look at the fixes together and they point the same way. The model did fine with the tasks; it struggled wherever Octomind asked it to work in a shape it rarely produced in training.

OpenAI has said its earlier models were trained with its own tools. The Codex prompting guide, written for an earlier Codex model, recommends "our exact apply_patch implementation as the model has been trained to excel at this diff format", and describes the model as one "which was primarily trained using a dedicated terminal tool". The GPT-4.1 prompting guide ships "a reference implementation of the apply_patch tool that we used as part of model training."

The Codex transcripts from our runs show what that training looks like on gpt-6.1-sol:

• One tool call per reply. In our October 26 analysis, Codex's model never put more than one tool call in a reply, even though Codex sends parallel_tool_calls: true. Octomind's replies carried 2.4 to 2.7 calls on average.
• Work packed into the shell instead of parallel calls. Of 3,862 shell commands in 67 Codex runs from October, 1,716 printed a line range with sed -n, 778 piped rg into sed or head, and 509 were Python scripts applying two or more replacements. Each Codex edit call carried 4.3 to 5.8 replacements, while Octomind packed 1.6 to 2.2 edits into each reply that edited. The LLM tool calling post has the full breakdown.
• Edit and test in one command. A change and its check usually travel together in one shell call.
• Strict reply types. The Responses API marks each assistant message as commentary (work in progress) or final_answer. In the Codex transcripts we checked, a message with no tool call was always a final_answer (51 of 51), and a commentary message was always followed by a tool call (190 of 190).

Now read the bugs against that list. The lost replay fed the model a history in a shape it had never written. The edit tool split an edit and its test into two calls, where the model's habit is one. The status instruction asked for a work-in-progress message with no tool call, a combination that never appeared in 241 Codex messages, and the model could not end it. The fixes that worked restored the shapes the model knows: verbatim history, edit and check in one call, and a status that ends the turn.

That does not mean the model can only work in Codex. Octomind gives it line-addressed file tools and a shell that rejects reading and editing commands such as cat, grep and sed -i. The model moved its reading and editing onto those tools, and the shell rejected only 1 of 150 calls. Octomind now uses fewer tokens than Codex on the same tasks. The lesson is narrower: a harness pays an extra round trip, or worse, wherever its tools ask for a shape the model rarely produced in training. We have measured this only for OpenAI's models so far; the harness engineering post covers the general idea.

Where Octomind's savings come from: the supervisor

The fixes above removed waste. The savings themselves come from what Octomind adds around the model and Codex does not: the supervisor, a set of checks and side models that run beside the agent. It:

• turns the request into a checklist and keeps reciting it;
• keeps the task plan outside the conversation and feeds it back;
• condenses large tool results;
• stops repeated identical tool rounds;
• checks a "done" against the checklist and the recorded evidence.

The 0.31.0 post introduced it, 0.44.2 made it demand proof, and what is harness engineering explains the idea behind it.

The savings on GOLD are largest where that machinery has the most to manage: 0.41× to 0.44× Codex's tokens on the long DuckDB one-shots, 0.43× on the DuckDB sequence, and 0.61× across both sequences, which run longer than the model's context window. Per round, Octomind re-read 73.7M cached tokens against Codex's 119.8M. We have not switched each part off on GOLD, so this is where the savings show up, not a measured split. The one part we did isolate, on six GOLD tasks in October, was the plan manager: turning it off cost 1.49× the tokens and 91 more model calls (tool calling post). Earlier, the rebuilt supervisor in Octomind 0.33 raised the solve rate on 14 SWE-bench-Live runs from 10 (0.32.0) to 13, with the model and config unchanged.

Results: Octomind vs Codex CLI cost, tokens and time per task

With the fixes above released in Octomind 0.70.1 (changelog) and octofs 0.23.3, plus the three benchmark settings marked above, we ran the full GOLD suite against Codex CLI 0.167.0: five paired rounds, 75 paired items (13 one-shots and 2 sequences per round), November 8 to 10. Every published number is in BENCHMARK.md, with per-round records for Codex and Octomind.

The solve rates share a 95% interval of 46.3–69.6%. Both clients ran on a ChatGPT plan, so costs are what the same usage would cost at API prices: the main model at OpenAI's prices, plus Octomind's side models (supervisor and compaction) at their providers' reported prices. Token counts cover the main model only.

Per task, the ratios are Octomind ÷ Codex for each task and round: geometric mean with a 95% bootstrap interval, then the median.

Solve rate is a tie: each client solved the same count, and neither ever solved the two Vue tasks, Polars or Cargo. The tasks they split went both ways. Octomind solved DuckDB Parquet 3 times to Codex's 2 and React 5 to 4; Codex solved DuckDB SQL export 2 to 1 and mypy 5 to 4.

The token savings come from the heavy work. On the two DuckDB one-shots Octomind used 0.41× and 0.44× Codex's tokens, on the DuckDB sequence 0.43×, and on React 0.45×. On the short Rust tasks, which both clients solved every round, it is close to even (0.90× to 1.06×). One task went against it: on Pydantic, Octomind used 1.57× the tokens and 1.30× the time.

Time is the one column Octomind still loses: 1.14× on the work both solved. That is close to the route's measured per-request ratio below, so the route likely explains most of it.

Route latency: the part no client can fix

On a ChatGPT plan, Codex sends requests to chatgpt.com/backend-api/codex, an endpoint OpenAI reserves for its own client. A third-party client has to use the documented route, api.openai.com/v1/responses. OpenAI's Sign in with ChatGPT guide says not to point third-party clients at the backend-api endpoints.

We sent the same single request through both routes: a one-word answer after 0, 20K, 40K or 80K tokens of padding, 15 rounds in shuffled order, a fresh process and connection every time, 60 pairs per measurement. We measured twice: on November 7, and on November 10 with the final versions, Octomind 0.70.1 and Codex CLI 0.167.0.

Octomind's requests carried about 6K fewer input tokens and still took longer. On November 7 the gap was clear: 1.11× over all sizes and 1.15× once the prompt passes 20K tokens, which covers most requests in a real coding session. On November 10 every interval included 1.0, so that run alone does not prove the gap, but its point estimates, 1.10 to 1.15, match the first. OpenAI does not document a difference between the routes; we measured the effect, not its cause. Every third-party client on a ChatGPT plan has to use this route, so we expect each to pay the same gap, whatever it does on its own side.

Our first version of this test, in October, reported +1.8 s per request and was wrong, because Octomind ran at a higher reasoning effort than Codex. The route latency write-up has the method, the millisecond medians and the raw samples.

Full benchmark summary

Everything from the five rounds in one table. "Mean per round" shows the range over rounds in brackets.

Totals weigh big tasks more, so they favor Octomind's large savings on DuckDB: 0.63× the tokens in total, against 0.75× for the typical task. The per-task ratios give every task the same weight; read those for what to expect on one task of your own.

Our findings in short

• Same solves, fewer tokens. On GOLD with gpt-6.1-sol, Octomind 0.70.1 and Codex CLI 0.167.0 each solved 38 of 65 one-shot runs and 140 of 140 sequence turns. Octomind used 0.75× the tokens and 0.87× the cost per task.
• Time is still 1.08×. On work both solved it is 1.14×, close to the 1.1× per-request cost of the third-party route.
• The model learned Codex's harness. One tool call per reply, edits and tests packed into shell commands, and strict reply types. Octomind's biggest bugs were places that asked for other shapes.
• Tool shape beat wording. The edit check in octofs 0.23.0 did what prompt wording could not in 13 probe runs.
• Replay matters. Sending the model's own output back verbatim (Octomind 0.64.0) brought its reasoning depth on the Cargo task level with Codex's.
• Checks must speak. A completion check that sees a gap and stays silent is worth nothing; the checklist cap and the confidence gate are gone in Octomind 0.70.1.
• Run clients in parallel. Provider speed changes by the hour; only same-moment pairs measure the clients.

What we did not change

The model, its reasoning effort and the system prompt are the same for both clients, and we did not tune Octomind for these tasks: every change above is a general tool or runtime fix that applies to any task. When a task's tests checked a requirement its instruction never stated, we fixed the instruction for both clients, not the agent.

The route gap stays. It is the cost of using the endpoint a third-party client is allowed to use, and it applies to every client on a ChatGPT plan that is not Codex.

Next: the same suite on gpt-6-astra

The same 15 items are running now on gpt-6-astra, with the same knowledge cutoff, the same pairing and five rounds. It will show whether the harness lesson holds for OpenAI's larger model. We will publish both columns the same way.

FAQ

What is the GOLD benchmark?

GOLD is a frozen set of 13 one-shot tasks and two 15-turn sequences from real pull requests merged between July and October 2026, judged only by the maintainers' own hidden tests. It is part of the open-source octobench harness and compares coding agents on the same model.

Can the model have memorized the GOLD fixes?

Not from training: OpenAI lists April 30, 2026 as the knowledge cutoff for gpt-6.1-sol and gpt-6-astra, and every fix was merged after that. The agent's network reaches only the model endpoint, so it cannot fetch the merged pull request during a run.

What is the difference between a one-shot task and a sequence?

A one-shot task is one pull request with one instruction and a fresh checkout. A sequence is 15 consecutive pull requests from one project, sent as turns of one persistent session, so the agent builds on its own earlier changes and has to manage a conversation longer than its context window.

Why run both clients at the same time?

Providers update models and change speed during the day. Running both clients on the same task at the same moment exposes them to the same conditions, so the ratio measures the clients instead of the provider's mood that hour.

Is Octomind faster than Codex CLI on the same model?

No. It takes 1.08× Codex's time per task overall and 1.14× on the work both solved, in line with the third-party route being about 1.1× slower per request. It uses 0.75× the tokens and 0.87× the cost per task.

Why do OpenAI models work best in Codex?

OpenAI's prompting guides say its earlier models were trained with its own tools, the kind Codex ships, and gpt-6.1-sol behaves the same way. In our transcripts the model made one tool call per reply and packed edits and tests into single shell commands. A different harness can match or beat Codex on tokens, but wherever its tools ask for other shapes, it pays in extra round trips.

octobench on GitHub · Octomind 0.70.1 changelog · octofs on GitHub · Related reading: What is harness engineering · LLM tool calling on the same model · Why we built octofs