Prompt Engineering for Different Models: Test What Each One Reads

Prompt engineering for different models got concrete for us when MiniMax restated our rule correctly, "N = after line N", and then answered a test question with: "append_line N means after line N, so 13 places the block directly after line 12." It could repeat the rule and still choose the wrong number. Adding another abstract explanation didn't help. A worked example did.

I make Octomind, an open-source agent runtime that routes to any model — that's why we had to test this. We need shared tool descriptions and system rules that survive a model switch. On October 27, 2026, we tested those texts across six models. We found misreadings, mistakes in our own prose, and a harness that sometimes supplied the answer itself.

The short answer: Keep one shared prompt when the job and constraints are shared, but test its rules on every model you route to. Give each model short scenarios and ask it to restate the rules. Fix the phrase behind a miss, then rerun every model. Use a full LLM evaluation workflow to measure task quality; use model-specific prompts when the required inputs or output formats differ.

Can prompt engineering for different models work with one prompt?

Yes, provided you check what each model reads. Our tests covered 26 distinct source texts across five sessions: seven Octofs tool/server texts, four Octomind main-agent texts, five async-tool texts, six context/compression texts, and four supervisor-note texts. The a1 session's additional t5 combined previously tested material; it was a composite retest, not another source text.

We routed GLM-5.3, MiniMax M3.1 Flash, MiMo V2.6 Flash, Qwen 3.8 Flash, Claude Opus 5.5, and GPT-6.1-sol through OpenRouter. The plain role specifies temperature 0.5; we didn't record what each provider route actually applied. These were single runs per model per iteration, sometimes with runtime retries, not repeated reliability trials.

Across the four reports that record edits, I count 11 rule changes linked to reported misreadings and eight code-alignment or consistency changes. A change means one target rule, regardless of how many words or attempts it took. The split is a judgment call: two of the edits could sit in either bucket. The 11 include restatement omissions and diagnoses from imperfectly isolated runs; they aren't 11 independently validated fixes. The fifth report records no edits.

Vendor advice supports testing the shared baseline. OpenAI’s 2026 guide recommends checking prompts across model types and snapshots. Anthropic’s 2026 guidance cautions against transferring model-specific techniques without evaluation. Google’s Gemini guide (2026) describes response changes from rephrasing.

The FormatSpread paper (Sclar et al., ICLR 2024) measures sensitivity to equivalent prompt formats; the PromptBridge preprint (2025) studies cross-model transfer. Neither validates our particular scenarios.

What did the models misread?

The useful result was a specific phrase we could repair, rather than a request to make the whole prompt “clearer.” “Held” means the recorded check passed, with the isolation limits noted below.

MiniMax's insertion mistake survived our first repair: “(N lines precede the new block).” Only the numerical example coincided with a pass. That's a more useful prompt-editing lesson than “add detail”: show the relationship the reader must preserve.

The async example exposed a different problem. The original text implied that polling was unnecessary but contained no explicit no-poll sentence. All six selected the waiting answer in scenario c. Their restatements were uneven: GLM didn't express the prohibition, Qwen said polling was unnecessary, and MiMo described delivery rather than a clear ban. Calling that three models dropping a written instruction would overstate the evidence. We made the intended prohibition explicit; all six restated it at i2.

In our sample, the report tables list three scenario misses: two from MiniMax (both Octofs cases) and one from GPT (the async /status case). MiMo twice gave a wrong answer and corrected it on a runtime retry. GLM and Qwen exposed errors through restatements, including clamping and stderr trust. Opus had no reported scenario miss or WRONG/OMIT verdict in these report tables. This is not a ranking: texts and difficulty differed, some initial runs retained supervisor influence, and each iteration was run once.

Can the harness accidentally answer its own test?

Yes. Our first testing mistake was leaving runtime instructions around the text under test. Supervisor rules such as SELF_REPORT_INSTRUCTION reached every model's system prompt; auto-activated capabilities could also expose tools. A correct answer could come from those instructions instead of our test text.

The a1 session discarded 24 confounded outputs. The a2 session moved 12 outputs aside; a3 records six contaminated calls. That's 42 set-aside runs across a1–a3. Octofs separately invalidated its initial 42-run round, with extra retries recorded outside that baseline. These are contamination counts, not a clean-call total.

The current runner uses an empty sandbox/ working directory, disables skill activation, supervisor and learning, and sends the assembled prompt on stdin. The scorer marks CONTAM when output contains a tool-box marker (╭) or auto-activated. That catches visible contamination, not hidden system-prompt influence.

We found the limit in a1 round 2. MiniMax restated an old loop/stall rule that was absent from the supplied text. The report's diagnosis, not independently reproduced: the installed binary still injected the old wording, because turning off the supervisor doesn't remove the runtime's built-in prompt. The report leaves the final combined main-agent wording untested after its rewrite.

For any harness, start with the bare model and a one-line system prompt. Inspect what's actually sent, including injected messages. When testing built-in runtime rules, retest against the binary you intend to ship. An edited source file isn't evidence that the running instrument changed.

What if the prose is wrong about the code?

Write the intended rule from the implementation or observed behavior before testing the prose. Otherwise a comprehension test can reward a faithful reading of a false instruction.

Our review found these mismatches:

• Workdir. “Required unless reset” missed a supported operation: omitting both path and reset reads the current directory.
• Text editor. The description grouped no-match and multiple-match errors together. Multiple matches return their line IDs; no match returns nearby candidates.
• Scheduling. when and every were described as optional. A timed interval needs a time-based when; idle scheduling has different pairing rules. We rewrote those cases explicitly.
• Supervisor notes. The main prompt treated loop notes as immediate orders, while the notes called themselves “Advisory” and allowed continuing a justified approach. Round 2 separated advice from mandatory action.

Those changes belong in a different count from model misreadings. If two instructions disagree, a model's choice between them can't tell you which sentence merely needs a better example.

How do you build a five-to-eight-question prompt test?

Copy the exact instruction, list its required conclusions, and write five to eight scenarios covering requirements, exceptions and boundaries. Check the answer key yourself. Add a separate restatement step so correct letters can't hide a missing rule.

One real Octofs question, quoted from the saved tail:

d. The copied lines must land directly after line 12 of the target file. A) append_line 13 B) append_line 12 C) append_line 11

The key is B. It separates a correct relative position from the tempting next-line number.

Balance answer letters and option lengths. Our a3 generator produced a faulty key; the questions stayed unchanged while the key was rebuilt and outputs rescored. Some tests were too easy and letter-skewed. Replace a giveaway with a realistic edge case, and record changed questions separately from wording edits. A new question is a new measurement condition.

Keep revisions small: the skill recommends usually under 15 added words per misreading, not a hard ceiling. Name the missing boundary, actor, exception or example, then rerun all target models. Qwen’s function-calling guide likewise recommends revising prompts or templates when protocol adherence falls short.

Run it yourself

The following is docs-from-repo, not newly run. You need Octomind on PATH with model credentials configured, Bash, Python 3, and the prompt-comprehension skill available. It's Apache-2.0; find it in the tap registry. Check octomind complete run lists assistant:plain.

Inside Octomind, ask for a “comprehension test” and supply the instruction paths, intended rules and model IDs. That phrase matches the skill's activation rules. For manual use, copy its two scripts into a fresh test directory. This small fixture tests insertion positions; expand your own test to cover every rule in the real instruction.

Create texts/t1.i1.txt containing:

For append_line, zero selects the beginning and -1 selects the end.
A positive value N inserts immediately after existing line N.

Create tail-t1.txt with the restatement, scenarios and answer format:

First print RESTATE and describe each rule in your own words, one bullet per rule.
Then print SCENARIOS. Answer each question as a: B - reason, using its own
letter and chosen option. Stop after the final answer.

a. Insert after existing line 12. Which value?
   A) 13
   B) 12
   C) 11
b. Insert at the file's end. Which value?
   A) -1
   B) 0
   C) 1
c. Insert at the file's beginning. Which value?
   A) -1
   B) 1
   C) 0
d. append_line is 5. Where does the new block start?
   A) Before existing line 5
   B) After existing line 5
   C) After existing line 6
e. Insert after existing line 1. Which value?
   A) 1
   B) 2
   C) 0

Save this matching example as key-t1.json:

{ "a": "B", "b": "A", "c": "C", "d": "B", "e": "A" }

The runner/scorer interfaces are bash run_models.sh <id> <iteration> <model>... and python3 score_answers.py key-<id>.json out/<id>-i<N>-*.txt. Run:

bash run_models.sh t1 1 openrouter:minimax/minimax-m3.1-flash openrouter:openai/gpt-6.1-sol
python3 score_answers.py key-t1.json out/t1-i1-*.txt

The quoted execution line inside the runner is:

(cd sandbox && octomind run assistant:plain -m "$model" --format plain \
  -c skills.auto_activation=false -c supervisor.enabled=false -c supervisor.learning.enabled=false \
  <"$work/prompts/$id-i$iter.txt" >"$work/out/$id-i$iter-$slug.txt" 2>"$work/out/$id-i$iter-$slug.err") &

The script creates the directories and supplies those variables. Expect one output per model and a score such as 5/5, possibly with CONTAM and wrong-answer details. Read restatements manually and record OK, OMIT or WRONG alongside the scenario score. Set contaminated outputs aside; log format slips and key repairs without treating them as wording failures.

With plain API calls, send the same text and tail as the user message under a one-line system prompt, save responses, and use the same key and restatement review.

How many calls should you budget?

One text across six models for three iterations costs at most 18 planned calls before retries: 1 × 6 × 3 = 18. The skill caps a session at 60 calls, retries included; reduce texts, models or iterations if texts × models × 3 exceeds that. Our historical sessions had expanded budgets; Octofs reached i4. They aren't examples of staying within today's default limits.

Keep text versions, questions, keys and every attempt in a ledger. Some of ours were incomplete, so the reports and output files were necessary to reconstruct results. Don't sum incompatible report totals into an experimental sample size. Our same-model tool-calling comparison asks the complementary question: what changes when the model stays fixed and the harness changes?

Does a comprehension pass prove the agent will do the task?

No. Choosing the right option and restating a rule measure reading under these conditions. They don't establish compliance during a long task, after context changes, or through tool failures.

Use this kind of LLM prompt testing for a narrow written contract. Then test the resulting work with a task evaluation suite. Save unresolved omissions and label final edits UNTESTED when they were not rerun. We caught useful wording defects this way; we also caught reasons not to trust our first green scores.

Prompt-comprehension skill on GitHub · Related reading: LLM evaluation workflow · What is harness engineering · Same-model tool-calling comparison