LLM Tool Calling: A Shell-Loving Model Did Better With Hard Limits

Inside Codex CLI, OpenAI's gpt-6.1-sol does almost everything through the shell. It pipes rg into sed, prints line ranges with sed -n, and edits files with Python one-liners that chain several replacements. You'd expect that model to struggle when a harness takes the shell away from it.

On our benchmark, it did better. We ran the same model in Octomind, where octofs, our MCP file server, blocks shell reads and offers line-addressed view, search and edit tools instead. It solved the same six tasks as Codex with 0.69x the tokens and 0.80x the cost. Loosening the block to a warning didn't help.

Our reading, and the LLM tool calling lesson of this post: a model that loves batching shell commands still performs better with hard limits and good dedicated tools. That's our interpretation of a small sample: six tasks, one run per arm, compared with Codex's five-round median. And wall time isn't a win. This post walks through the evidence chain, the limits, and the octofs work that made the dedicated tools worth using.

How we run a same-model coding agent comparison

The tasks come from GOLD, the frozen client-comparison suite in our open-source octobench harness. It has 13 one-shot tasks and two 15-turn sequences, all built from real, merged, human-authored pull requests from August to October 2026, in projects like Vue, React, DuckDB, Polars, mypy, Pydantic, Cargo, Nushell, Wasmtime and ESLint.

The rules make the client the only variable:

• Same model, same system prompt. Every client gets the identical model and a shared system prompt.
• Sealed network. The tasks come from merged pull requests, so web access would hand over the answer key. The agent can reach only the model endpoint.
• Only the maintainers' tests judge. Validation restores the project's own upstream tests byte-for-byte from the merged commit. Every assertion has to trace back to the instruction or a visible repository convention.
• Solve rate, tokens and time stay separate. We don't compare tool-call counts, because a Codex shell call and an Octomind view call aren't the same unit.

The Codex column is published: Codex CLI 0.162.1 on gpt-6.1-sol, five rounds. It solved 41 of 65 one-shot runs (63.1%, 95% CI 50.9–73.8%) and passed 140 of 140 scored sequence turns, at about 2.7M tokens and $0.52 per solve. The per-round record links every task to its case.

Five rounds is the protocol minimum for a reason: Codex's own wall time on one Cargo task ranged from 168 to 402 seconds across them. Our Octomind numbers below are one run per task, so read them as a first measurement. For the case for benchmarking on real pull requests, see our coding agent benchmark write-up.

The habit: inside Codex CLI, the model lives in the shell

We parsed the shell commands in 67 Codex runs of the one-shot tasks (the five rounds' 65 runs plus two that were rerun after infrastructure failures). Out of 3,862 commands:

• 1,716 (44%) used sed -n to print a line range of a file.
• 778 piped rg into sed or head.
• 509 were Python scripts applying two or more .replace() edits.
• By our rough parse, about 69% started a command with a program that octofs's shell gate covers (cat, head, tail, grep, rg, find, ls).

That's not misuse. It's how this model navigates and edits code inside a harness built around one shell tool. Search with rg, cut the hit out with sed -n, patch it with a Python script, often all in one call. If tool habits were fixed properties of the model, this is the model you'd expect to fight a harness that blocks those commands.

Same model, hard limits: it complies

Octofs has a shell gate. A command that starts with a read or search program, like cat, grep, rg, find or ls, is rejected in full with a message naming the dedicated tool: view for reading and searching, batch_edit and text_editor for edits. In-place sed -i and content writes into the project are rejected too. Pipelines aren't split, so cargo test 2>&1 | tail -50 still runs.

We expected gpt-6.1-sol to keep hitting that wall. It barely touched it. In Octomind's first GOLD round, the block rejected 3 of 144 shell calls across five tasks. In the six-task run below, it rejected 1 of 150. The model read the tool descriptions and moved its reading and editing onto the dedicated tools. On the Wasmtime task, for example, it made 34 octofs edits and wrote six Python scripts through the shell.

The same thing happened earlier on a different model. On GLM-5.3, we A/B tested the block, and it was a win on tokens and quality, even counting the one to five rejected calls per task. The rejection pushes the model onto tools that return line-addressed, cacheable output instead of raw shell dumps.

The result: fewer tokens and lower cost, but not less time

Here are the six tasks that Codex solved in all five of its rounds: Nushell, Wasmtime, Pydantic, mypy, DuckDB and React. Octomind ran them with Octomind 0.58.8 and stock octofs 0.18.3 on gpt-6.1-sol at low reasoning effort, with lesson learning off, one run per task. Ratios compare Octomind with Codex's five-round median on each task.

Per task, Octomind's tokens against Codex's median were 0.46x on DuckDB, 0.64x on mypy, 0.73x on Nushell, 0.82x on Pydantic, 1.04x on React and 1.27x on Wasmtime. Four of six came in clearly lower, React was about even and Wasmtime was higher.

The time row needs a word. The two clients' model calls ran on different days at different provider speeds, so raw wall time mixes client behavior with API latency. Normalized time charges Octomind's call count at Codex's median seconds per call, then adds Octomind's own non-model time. That's an estimate, and it says Octomind is slower: its non-model time was 1,485 seconds against Codex's 751, mostly builds and tests.

The heavy C++ task: under half of Codex's token usage

The strongest single result came from duckdb_sql_export_sources, the heaviest of the six: a C++ change in DuckDB with a long build and test cycle.

Octomind came in below Codex's best round on both tokens and cost, with about half the model calls. Codex's median was 8.16M tokens. Every model call re-sends the context, so fewer calls compound on a task this size. It's one run, so treat it as one strong data point, not a distribution. And wall time wasn't a win here either: Octomind took 24–29 minutes across its two runs of this task (hard and soft) against Codex's 19–27, again C++ build and test time outside the model.

Where the time goes

A forensic read of four round-one tasks (one Octomind round against five Codex rounds, so directional only) found two causes:

• Long commands hold the whole tool batch. An octofs shell call returns as soon as its command finishes. A command that runs longer than 10 seconds holds the call for the full 10 seconds before moving to the background. Octomind waits for every tool in a batch, so the batch waits too. Codex asks its shell to yield after one second (yield_time_ms: 1000). Across the four tasks, that was about 230 of Octomind's 518 seconds of non-model time. It's an estimate from counts and source, and it's an open item.
• Edits come in smaller pieces. Octomind used 92 edit rounds against a Codex mean of 41.8. It packed 1.6–2.2 edits into each round, while each Codex edit call carried 4.3–5.8 replacements. This is the one place where the shell habit clearly pays: one script, many edits, one round trip.

That round also had lesson learning on. Distillation spent 158.8K tokens across four tasks and wrote nothing, because every benchmark task starts from a fresh state. So we turned learning off for benchmark runs. On the Nushell task, side-model tokens dropped from 279K to 110K. In everyday use, where the lesson store persists, learning stays on by default.

Loosening the limit didn't help

If the shell habit is real, a softer gate should beat a hard one: let the command run, attach a warning naming the dedicated tool, and save the model a round trip. Octofs briefly had that mode in August. We added it as an option on August 1, removed it on August 10 to make the block unconditional, and later GLM-5.3 A/B runs backed that call. For gpt-6.1-sol, we tested it again on the same six tasks, same config, one run per task.

Soft cost 5% more tokens and 8% more cost than hard. But the warning never fired, so most of that gap is run-to-run noise, not the policy itself. Per task it went both ways: on DuckDB, soft came in even lower than hard (3.36M tokens, $0.68, 50 calls). The time row shows how loud that noise can get. On mypy, the soft run took 20.5 minutes and 3.16M tokens, against the hard run's 8.0 minutes and 2.00M. The soft run's agent re-ran mypy's full testcheck suite several times, once for 662 seconds in parallel-worker mode. That has nothing to do with the shell policy, and it accounts for most of the 1.45x.

One difference wasn't noise in kind, though it's an anecdote from a single task. The soft-mode tool description says gate-covered commands "are discouraged and come back with a warning" instead of "are rejected." On Wasmtime, under that looser wording, the model drifted back to Codex-style edits: seven Python write_text scripts through the shell plus four octofs edits, against six scripts and 34 octofs edits under the hard wording. No warning fired. The description alone moved it.

So loosening the cap bought nothing we could measure, and the one visible behavior change went toward the old habit.

Why we think the harness shapes the habit

Put the chain together. Inside a shell-first harness, gpt-6.1-sol navigates and edits through shell chains. Give the same model line-addressed tools with a firm boundary, and it switches to them almost without friction, then finishes the same tasks with fewer tokens, lower cost and fewer model calls. Soften the boundary, and nothing improves, while the tool wording alone can pull it back toward scripts.

Our interpretation is that the habit comes from the harness, not from the model. gpt-6.1-sol uses the shell for everything in Codex because that's the tool it has there. GLM-5.3 showed the same pattern earlier, when the block measured as a win. One older data point fits too: in August, on gpt-5.6-luna, an older octofs build with a freer shell was 2.2 times slower on one identical turn (587 against 273 seconds, 59 against 23 requests), mostly because the model swapped batch_edit for 21 single text_editor calls. That's one turn, so it's consistent with the thesis but doesn't prove anything on its own.

What this doesn't show: that hard limits make Octomind faster (they don't, yet), that this holds beyond six tasks, or that it holds for every model. A model that keeps re-sending blocked commands would be a different story, and that's what the rejection count is for.

The octofs fixes that made the dedicated tools worth using

A block only helps if the tools it points to are good. Most of our octofs work over the past months came from reading benchmark transcripts until we found the call where a run went wrong, and surprisingly often it was the tool. The design principle is the one from why we built octofs: every error message is a prompt, and every rejection costs a whole round trip.

• Silent builds turned into polling loops (fixed in 0.10.2). Octomind's client gave an octofs shell call 30 seconds without output before cancelling it, and builds redirected to a log are silent. So agents backgrounded builds and polled them, and on GLM-5.3 runs polling was 62 of 250 tool calls on one C++ sequence and 102 of 504 on another. A longer timeout didn't fix it (we tried and reverted). A 10-second MCP progress heartbeat did: on the seven-turn sequence, turns passed went from 6/7 to 7/7, cost from $3.66 to $2.55, and tokens from 10.24M to 7.76M, with zero polling.
• One CRLF line rewrote a whole file (fixed in 0.15.5). If you've ever debugged CRLF vs LF, you'll recognize this one. A Fastify doc file with 2,516 LF lines and 5 CRLF lines turned a one-word edit into a 5,022-line whitespace-only diff, and the task timed out at the 45-minute cap four runs in a row. Now each line keeps its own ending. The task passed in both runs after the fix.
• Read-only sed and awk were blocked (fixed in 0.15.6). The gate treated sed -n '1,5p' and awk 'length>88' as edits, and agents retried variations across at least six tasks of one run. Now only in-place sed -i is blocked.
• Rejecting what the model clearly meant (0.15 series). In a 20-task GLM-5.3 run, 27 views had start and end swapped, 10 named several candidate files, and 20 batch edits sent content as a list of lines. Octofs now serves the obvious intent. Failed calls were only 0.7% of calls there, but each one was a full round trip.

New in octofs 0.19.0: hard by default, soft as an opt-in

Octofs 0.19.0 adds a --hint-mode flag.

• hard (the default): the whole command is rejected and nothing runs. The message says so explicitly, so the model can split a compound command instead of retrying it.
• soft (opt-in): a command the gate covers runs anyway, and the result carries a warning naming the dedicated tool.

Everything else about the gate stays the same in both modes: pipelines aren't split, read-only sed and awk pass, and a write to a scratch path outside the project runs with a hint. The shell tool's description states the hard policy and switches to the soft wording under --hint-mode soft.

The octofs docs give the guidance in one line. Hard suits models that follow tool steering: it was a measured win on GLM-5.3, and gpt-6.1-sol rarely hits it (1 of 150 shell calls over six tasks). Try soft for a model that keeps re-sending blocked commands, and watch the rejection count.

To opt a model into soft mode:

{
	"mcpServers": {
		"octofs": {
			"command": "octofs",
			"args": ["mcp", "--path", "/your/project", "--hint-mode", "soft"]
		}
	}
}

Octofs is a standard MCP shell server and file server, so this works the same under Octomind, Claude Code, Codex CLI or anything else that speaks MCP.

We also tried extending octofs's MCP server instructions with round-trip guidance. It told the model to put every change a file needs into one batch_edit, send edits to different files as parallel calls, search for several names with one regex alternation, and keep working while a build runs. We tested that text, it didn't help, and it stays out of 0.19.0. The results are in the next section.

Every arm we ran on the same six tasks

After the hard-versus-soft comparison, we varied one Octomind setting at a time on the same six tasks. Same model at low effort, Octomind 0.58.8, one run per task per arm. Ratios are against Codex's five-round median, and Codex solved all six with 308 model calls.

Only one setting clearly mattered. Turning off Octomind's plan manager, which keeps the task plan outside the conversation and feeds it back to the model, cost 1.49x the tokens of the first row and 91 more model calls. That erased the lead over Codex. The external plan is load-bearing. Everything else stayed within run-to-run noise or got worse:

• The efficiency instructions changed behavior, not cost. We injected the batching guidance through the prompt. The model stopped writing Python edit scripts through the shell entirely and used octofs's edit tools, which fits the thesis about harnesses shaping habits. But it made more calls, not fewer: 289 against 252. Octomind's fix that passes MCP server instructions to the model is prepared but hasn't been tested as a build, since this test used the prompt instead.
• The verification gate off gained nothing. The gate never held back a "done" in any session, so removing it saved nothing.
• Later compaction and condensing gained nothing. Raising either threshold didn't beat the first row.

Three arms (later compaction, efficiency instructions and plan manager off) failed mypy the same way: each broke the same existing upstream test, testPartialNoneTypeClassMethod. That's a real model regression, not infrastructure, and a reminder of how much a single run can vary.

The broader point holds across the table. Every arm with learning off, except the one without the plan manager, beat Codex on tokens (0.69–0.83x) and cost (0.80–0.95x), with equal or near-equal solves. None of them is a wall-time win, and each is one run per task.

What's next

• More rounds. Everything above is one run per task. The protocol calls for five rounds before a column counts, and the first-row configuration is the one worth repeating.
• MCP instructions as a build. The fix that shows octofs's server instructions to the model is ready but untested as a binary.

Takeaways for LLM tool calling design

• Don't design tools around habits a model learned in another harness. In our runs, the shell-first model followed dedicated tools as soon as it had good ones and a firm boundary.
• A block is only as good as the tools behind it. Line-addressed views and edits that return diffs are what the model moves to. Most of our octofs work went into making them reliable.
• Count what your gates actually do. "We block grep" says nothing until you know how often it fires. Ours fired once in 150 shell calls with gpt-6.1-sol.
• Tool descriptions are prompts too. Changing "rejected" to "discouraged" changed one model's edit style on one task, with zero warnings fired. More guidance isn't automatically better, though: our batching instructions changed how the model edited but raised its call count.
• Measure the scaffolding, not just the tools. Of six settings we changed one at a time, only one clearly mattered: without the plan manager, tokens rose 1.49x.
• Every rejection is a full round trip. Serve the unambiguous intent whenever you can.
• Report liveness instead of tuning timeouts. A heartbeat fixed what longer timeouts only postponed.
• Don't trust one run. On mypy, one run took 20.5 minutes because its agent kept re-running a test suite, and the other took 8.0.

If you're tracking Codex token usage or running your own coding agent comparison, the published Codex per-task records are in octobench, so you can check our numbers against your own runs. For the efficiency side, see where agent tokens actually go and Octomind vs OpenCode on the same model.

FAQ

Do shell-heavy models need a shell-first harness?

Not in our data. gpt-6.1-sol works almost entirely through shell chains inside Codex CLI, but inside Octomind it followed a hard shell block (1 rejection in 150 calls) and solved the same six tasks with 0.69x the tokens and 0.80x the cost. That's six tasks and one run per arm, so it's a first measurement, not a law.

Is Octomind faster than Codex CLI on the same model?

Not yet. On those six tasks Octomind's time, normalized to Codex's model speed, was 1.09x, and its non-model time was about double Codex's, mostly builds and tests. The biggest identified causes are long shell commands holding a tool batch for 10 seconds and smaller edit batches.

Should I block grep and cat in my agent's shell?

If your dedicated tools are good, our data says yes, at least as a default. Blocking was a measured win on GLM-5.3 and cost gpt-6.1-sol nothing measurable, because it rarely tried those commands. Watch the rejection count. If a model keeps re-sending blocked commands, try a softer policy and measure.

What is octofs?

Octofs is an open-source MCP server written in Rust that gives coding agents file viewing, search, editing and a shell. It's the file layer for Octomind, but it works with any MCP client. The design write-up explains its fuzzy edit matching and diff-as-result edits.

What are the best tool calling models for this kind of work?

We can't rank models on tool calling from this data. GOLD compares clients on the same model, not models against each other. What it does suggest is that a model's tool habits depend heavily on the harness it runs in.

What changed in octofs 0.19.0?

The --hint-mode flag: hard is the default and rejects shell reads, searches, in-place edits and content writes unexecuted; soft is an opt-in that runs them and attaches a warning. The release notes on GitHub list everything else in the release.

Octofs on GitHub · Octomind on GitHub · octobench on GitHub · Related reading: What is harness engineering · MCP vs CLI vs skills