Full experimental report · 2026-07-06 · 42 of 48 planned cells
Coding-subagent model shootout: GLM 5.2 vs GPT 5.5 vs Opus 4.8 vs Composer 2.5
4 models × 4 tasks × 3 runs, each in a fresh isolated git worktree with frozen prompts, blind two-judge scoring, and automated pass/fail checks. Companion files: REPORT.md, HANDOFF.md, RESUME.md, logs/infra-failures.md, judgments/, scores/, blind/, worktrees/.
1 · Summary and recommendations
There is no overall winner; each model won a different regime. Recommendations, strongest evidence first:
- Need it done end-to-end (tests run, work committed): Opus 4.8. The only model that went 9/9 on automated checks and 9/9 on commit compliance, and the fastest lane on edit tasks despite a documented handicap (see caveat 6.1).
- Fuzzy spec, judgment calls, best all-round output: GPT 5.5. Highest or near-highest judge medians on t1/t2/t3, tightest min-max spread, zero test failures after r1. But it structurally cannot commit in git worktrees (6.2) - the orchestrator must commit for it.
- API design/taste and cheap bulk work: GLM 5.2. Won the t4 taste task in 3/3 runs against models with much higher taste reputations, and matched GPT on the fuzzy-spec task. One catastrophic lazy run in nine (t1-r3: installed the dependency, migrated nothing) means its floor is low; verify its output.
- Code review second opinion: GPT 5.5 (5-6/7 seeded bugs, three consistent runs). Composer's single review run (6/7, best of any single run) is promising but n=1.
- Migrations with runtime quirks: Opus or GPT, not Composer (0/2 test-green, pending its reruns).
2 · Methodology
4 models × 4 tasks × 3 runs = 48 planned cells; 42 completed (Composer stopped at 6 when the user halted the lane as the Cursor plan approached its usage cap, $19.77/$20 consumed in the Jun 30-Jul 6 window - a preemptive stop, not a hard throttle). Each cell ran in a fresh, isolated git worktree pinned to a fixed base SHA, so every run of every model saw byte-identical inputs. Prompts were frozen before any run (prompts/*.md) and identical across models. No retries and no best-of-n: a completed run counts as-is; only infrastructure failures were wiped and rerun, each one logged. Runs were interleaved round-robin within four per-model serial queues (one per provider, to avoid rate-limit interference), running concurrently on one M-series macOS machine.
2.2 Models and harnesses (exact invocations)
| lane | model | harness + version | invocation |
|---|---|---|---|
| glm | GLM 5.2 | opencode 1.17.13 | opencode run -m zai-coding-plan/glm-5.2 "<prompt>" |
| gpt | GPT 5.5 | codex-cli 0.142.4 | codex exec --full-auto (t3: -s read-only) |
| opus | Opus 4.8 | claude CLI 2.1.201 | claude -p --model claude-opus-4-8 --permission-mode acceptEdits --allowedTools "Bash" |
| composer | Composer 2.5 | cursor-agent 2026.07.01 | agent -p --trust --model composer-2.5 (t3: --mode plan) |
The Claude entrant is Opus 4.8 (not Sonnet 5) to match the routing-table row under calibration. This is a model+harness comparison, not a pure model comparison: each model ran inside its vendor's own agent scaffold, which is how they are actually consumed via the delegate skill.
2.3 Tasks
| id | type | repo @ SHA | task | automated check |
|---|---|---|---|---|
| t1 | mechanical migration | hance @ 8abaebf | migrate 18 bun:test files to vitest, add config + script, zero behavior change, suite green, commit | bun run test:vitest exit 0; forbidden-edit scan (only tests/config may change); commit present |
| t2 | fuzzy spec | auphonic-cli @ 9097c67 | add --json mode to every command, self-designed schema, JSON errors + non-zero exit, human output unchanged, tests | bun test exit 0; commit present |
| t3 | code review | hance @ 565c8f2 (eval-t3-seeded) | review git show HEAD for bugs; strict findings format; false positives penalized | recall/precision vs 7 seeded bugs + 2 baits (key never shown to any model) |
| t4 | taste | camkit @ 50b937e | design packages/core public API: exports, error convention, README with example, typecheck green | tsc --noEmit exit 0; commit present |
t1 is deliberately treacherous: the code under test uses Bun runtime APIs (Bun.serve, Bun.spawn, import.meta.dir), so a naive import swap passes nothing - the discriminator is whether the model discovers it must run vitest on the Bun runtime (or shim import.meta.dir via a Vite plugin). t3's diff is a real feature commit with 7 seeded bugs and 2 baits (pre-existing patterns that look wrong but aren't).
2.4 Scoring pipeline
Layer 1 - automated (objective): the per-task commands above, run by the orchestrator in each worktree after the matrix completed. Verdicts: PASS / FAIL, with (no-commit) recorded separately once commit compliance turned out to be harness-confounded (6.2).
Layer 2 - blind judging (subjective): for every (task, run#) set, the four artifacts were copied to files labeled A-D with a fresh random mapping per set, recorded but not opened until both judges finished. Two judges scored every artifact 1-5 on approach correctness, spec adherence, and fixes-needed (5 = mergeable as-is), plus taste on t4: Fable 5 (the orchestrator session) and GPT 5.5 (codex exec -s read-only, same rubric text verbatim). The t3 answer key was never placed in any model's prompt - including both judges' - and the Fable judge did not read the key until its own t3 scores were locked. Judge disagreements ≥ 2 points on any dimension were flagged for human adjudication, not resolved (24 flags; see 6.5).
Layer 3 - t3 key scoring: recall = seeded bugs found / 7 (matched by file + defect, exact line not required); precision penalty −1 per bait item or invented bug reported at high/med severity.
Reported statistics: median with min-max across runs, never best-of; consistency = automated passes / runs completed.
2.5 Fairness interventions (all logged)
1. The first launch ran inside the orchestrator's sandbox, which blocked bun's tempdir and registry network for every lane; all four t1-r1 runs were fighting the environment, not the task. Wiped, relaunched unsandboxed.
2. The opus lane's original --permission-mode acceptEdits blocks the Bash tool in headless mode - opus could not run tests, git show, or commit while every other lane had full shell autonomy. Opus cells wiped and rerun with --allowedTools "Bash" for parity.
3. Composer's in-flight cell at the user's preemptive halt (plan nearly exhausted) was killed and wiped (not scored, not a model failure).
3 · Results
3.1 Full matrix (judge score = mean of rubric dims, averaged over both judges, 1-5)
| task | glm | gpt | opus | composer |
|---|---|---|---|---|
| t1 judge median (min-max) | 3.00 (1.00-3.33) | 4.00 (3.83-4.00) | 3.67 (3.33-4.83) | 2.75 (2.67-2.83) n=2 |
| t1 tests green | 2/3 | 2/3 | 3/3 | 0/2 |
| t2 judge | 4.08 (3.79-4.33) | 4.17 (3.50-4.46) | 3.54 (3.38-4.00) | 3.54 (3.46-3.62) n=2 |
| t2 tests green | 3/3 | 3/3 | 3/3 | 2/2 |
| t3 recall of 7 (min-max) | 4 (2-5) | 5 (5-6) | 4 (4-4) | 6 n=1 |
| t3 precision penalties | 0 | 0 | 0 | 0 |
| t4 judge | 4.25 (4.00-4.25) | 3.62 (3.62-4.12) | 3.62 (3.50-4.25) | 4.25 n=1 |
| t4 typecheck green | 3/3 | 3/3 | 3/3 | 1/1 |
| commits made | 7/9 | 0/9 (harness, see 6.2) | 9/9 | 0/6 (behavioral) |
3.2 Apples-to-apples subset (only cells all four models completed: r1 of all tasks + r2 of t1, t2)
| task (cells) | glm | gpt | opus | composer |
|---|---|---|---|---|
| t1 judge (r1, r2) | 3.17 | 3.92 | 4.08 | 2.75 |
| t2 judge (r1, r2) | 4.21 | 3.83 | 3.46 | 3.54 |
| t3 recall (r1) | 2/7 | 5/7 | 4/7 | 6/7 |
| t4 judge (r1) | 4.25 | 3.62 | 3.62 | 4.25 |
3.3 Speed (wall clock, whole harness run: CLI startup → exit; medians)
| glm | gpt | opus | composer | |
|---|---|---|---|---|
| edit tasks (t1/t2/t4) median | 327s | 223s | 246s | 602s |
| edit tasks range | 205-882s | 155-1026s | 186-397s | 163-2609s |
| review (t3) median | 164s | 77s | 59s | 427s |
Opus and GPT are the fast lanes; Composer's median edit cell took ~2.5x longer than the field, with a worst case of 43 minutes (see the Composer section for the analysis of why).
3.4 Token usage (from each CLI's local session store; Composer's store held no usage data)
| lane | uncached input | cache reads | output (incl. reasoning/thinking) | cells |
|---|---|---|---|---|
| glm | 544K | 11.8M | 162K | 12 |
| gpt | ~1.25M (13.9M total incl. cached) | 12.6M (cached input) | 141K | 12 |
| opus | 353K | 16.4M | 358K | 12 |
| composer | not recoverable | - | - | 6 |
All three measurable lanes lean hard on prompt caching; the uncached-input and output columns are the plan-burn proxies. GPT's raw input volume is enormous (single worst cell: t1-r2 at 4.66M input tokens, 96% cached - codex re-reads context aggressively) but its output is the leanest per cell. Opus produces by far the most output tokens (2.2-2.5x the others), which includes its thinking tokens - the main driver of its subscription burn. Opus r1 figures exclude the wiped handicapped runs.
3.5 Cost
All four lanes run on flat subscriptions, so marginal dollar cost per cell is $0 until a plan cap is hit; the meaningful cost metric is plan burn. Composer was the only lane with a visible dollar meter: the Cursor plan window (Jun 30-Jul 6) ended at $19.77/$20 consumed, with 7 eval attempts (6 completed + 1 killed) accounting for most but not provably all of it - roughly ≤ $2.8 per completed cell as an upper bound. The Claude subscription visibly throttled nothing during the run (opus completed 12/12), but opus's high output-token volume is what drained the user's plan fastest in wall-clock terms. Dollar-equivalent API pricing was deliberately not computed: every lane here is subscription-priced in practice, and API list prices would misstate the real economics.
Charts
Judge medians per task
Score 1-5, median across runs, both judges averaged. Bars scaled 0-5.
Speed — median wall clock
Seconds, whole harness run (CLI startup → exit). Bars scaled to slowest lane.
Token usage — totals across completed cells
Uncached input and output tokens per lane (plan-burn proxies). Composer: not recoverable.
3.6 · t3 bug-level detail
Key: B1 inverted preset ternary · B2 wrong schema default · B3 swapped mix() args · B4 channel-order swap · B5 wrong activation fallback · B6 undersized GPU buffer · B7 removed panic fallback.
| B1 | B2 | B3 | B4 | B5 | B6 | B7 | |
|---|---|---|---|---|---|---|---|
| glm r1 | · | · | · | ✓ | · | ✓ | · |
| glm r2 | ✓ | · | · | ✓ | ✓ | ✓ | · |
| glm r3 | ✓ | · | ✓ | ✓ | · | ✓ | ✓ |
| gpt r1 | ✓ | ✓ | ✓ | ✓ | · | ✓ | · |
| gpt r2 | ✓ | · | ✓ | ✓ | · | ✓ | ✓ |
| gpt r3 | ✓ | ✓ | ✓ | ✓ | · | ✓ | ✓ |
| opus r1 | ✓ | · | · | ✓ | ✓ | ✓ | · |
| opus r2 | ✓ | · | · | ✓ | ✓ | ✓ | · |
| opus r3 | ✓ | · | · | ✓ | ✓ | ✓ | · |
| composer r1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| composer r2 | - | - | - | - | - | - | - |
| composer r3 | - | - | - | - | - | - | - |
B4 and B6 were found by every run of every model. B5 (the subtle passthrough-breaking fallback) was found by opus in 3/3 runs and by almost nobody else - opus's review profile is "same four bugs every time, including the subtlest one." Opus never found B7 (the Rust panic); GPT found it 2/3 times. Nobody flagged either bait item and nobody invented a bug at high/med severity: precision was a uniform 1.0, so the t3 discriminator was purely recall.
4 · Skills, plugins, and MCP tools available to each lane
Audited after the run from each CLI's session store and config.
| lane | context/plugins loaded | MCP servers | skills/plugins actually invoked |
|---|---|---|---|
| glm | opencode global config incl. superpowers skill pack and plugins | several configured (e.g. chrome-devtools, claude-context) | none - 374 tool calls, all core (read 149, bash 148, edit 37, write 20, todo 9, grep 7, glob 4) |
| gpt | vanilla codex; no AGENTS.md in any eval repo | none | none - core tools only |
| opus | ~/.claude/CLAUDE.md (user style rules), hance/CLAUDE.md on t1/t3, ponytail + caveman plugin hooks injected into all 12 cells (verified in 16 session files) | none configured | zero Skill-tool invocations; hooks injected instructions only |
| composer | cursor-agent defaults; no rules files found in eval repos; config not recoverable post-run | none found | not auditable (stdout buffered, no local trace store found) |
The consequential row is opus: see caveat 6.1.
5 · Why was Composer slow?
Direct evidence is limited because cursor-agent buffers stdout until exit (its logs contain only the final summary) and keeps no recoverable local session trace. What was measured:
- During the 44-minute t1-r2 cell, the composer process sat at 1.7% CPU while its worktree files were being modified minutes apart - so the time was not local compute but waiting: network/API latency, server-side queueing, or deliberate pacing.
- Slowness scaled with task, not constant overhead: t4-r1 finished in 163s (fastest single t4 cell of any model), while both t1 cells took 21-43 minutes. A fixed harness tax cannot produce that spread.
- Both slow cells were the bun-runtime migration where composer also failed tests - consistent with long unproductive iterate-on-failure loops rather than transport overhead.
- Possible contributor that cannot be ruled out: throttling as the plan approached its cap (both worst cells ran in the last ~$5 of a $20 window).
- All four lanes shared one machine, but the three fast lanes ran under identical contention, so contention does not explain the gap.
Conclusion: the evidence points at Composer spending long, low-intensity spans blocked on its service while iterating on a task it was failing at - not at local harness overhead. Its n is small; treat as provisional pending the post-Jul-11 reruns.
6 · Caveats and confounds — read before trusting any single number
Major confound 6.1 The opus lane ran with contaminated context (discovered post-hoc)
The user's Claude CLI setup injects "ponytail" (radical minimalism: "shortest working diff wins") and "caveman" (terse output) plugin instructions via SessionStart hooks, and these fired inside all 12 headless claude -p cells - confirmed in 16 session files, and visible as ponytail: comments in two opus artifacts (which is also how the blind was partially broken, see 6.3). No other lane carried an equivalent directive. Direction of bias is unknowable from this data: minimalism may have helped automated pass rates (small careful diffs) while hurting judge scores (both judges read spare diffs as less thorough; the GPT judge marked opus artifacts down 2-3 points in four sets). Any future rerun should use claude -p with hooks disabled for a clean lane.
6.2 Commit compliance is not comparable across lanes
codex's --full-auto sandbox cannot write to ~/hance/.git/worktrees/* (worktree metadata lives outside the workspace root), so git add fails with Operation not permitted: gpt's 0/9 commits is physics, not choice. Composer's 0/6 with full --trust is choice (it twice printed "run locally: git add ..." instructions and once committed a debug script instead). Commit compliance was therefore excluded from cross-model automated verdicts and reported as its own row.
6.3 Blindness was imperfect
Composer unmasked itself in t1-r1 by committing a debug script containing its own worktree path. Opus unmasked itself in two sets via the injected ponytail: comment style (initially misattributed; both leaks were opus artifacts - t1-r2/D and t2-r2/D). The orchestrator-as-judge also saw automated verdicts before judging (though not letter mappings). Both judges scored from artifact content only, and per-set random mappings held for everything else, but these three leaks make the blind "best effort," not airtight.
6.4 Sample sizes and environment
Composer: n=2 on t1/t2, n=1 on t3/t4 - its rows are indicative, not conclusive; six cells are rerunnable after Jul 11 (procedure in RESUME.md). glm's t1 median hides a 1.00-scoring collapse run; single-run failures matter at n=3. One machine, four concurrent lanes, wall clocks include CLI startup and bun install; timings are comparative under shared load, not absolute benchmarks. Two lanes (glm, opus) read prompts through shells with the user's global config; repos contained their own conventions (hance/CLAUDE.md, in-repo ponytail: comments predating the eval).
Judge pattern 6.5 Judges disagreed systematically, not randomly
24 dimension-level disagreements ≥ 2 (of 372 scored dimensions, ~6%), flagged and unresolved per protocol (judgments/ has both columns). The pattern: the GPT 5.5 judge was consistently harsher than the Fable 5 judge, and disproportionately so on artifacts that unblinded as opus (4 of 10 flagged artifacts) - possibly reacting to the ponytail-minimal style (6.1), possibly cross-family bias, possibly Fable leniency toward Claude-family output. Recommendation for the video: show both judge columns side by side instead of a merged score; the disagreement is itself a finding.
6.6 Incidents
A cleanup pkill -f "codex exec" during the sandbox recovery killed an unrelated codex process from another session (a read-only PR review); that session's work needs rerunning. Two full infra restarts happened before the clean matrix (2.5); the final dataset contains only post-fix runs.
7 · Reproduction
run.sh <task> <model> <run> executes one cell (worktree + frozen prompt + log); queue.sh <model> runs a lane; score.sh the automated layer; prep-blind.sh/fix-blind.sh build judging artifacts; judge-gpt.sh runs the GPT judge; analyze.py joins judges, flags disagreements, unblinds, aggregates; extract-tokens.py mines the CLI session stores. Worktrees are the raw artifacts - run cleanup.sh only after everything above is locked.