RESULTS.md

Results — 2026-07-24 (52 trials, Codex CLI 0.145.0, gpt-5.6-sol, ChatGPT Pro auth)

All trials: fresh codex exec session, read-only sandbox, alternating arm order, MCP servers / plugins / memories / multi-agent disabled, arms differ only in a batching instruction vs a length-matched placebo instruction. See README for methodology; drafts of the upstream reports are linked from the repo description.

Effort sweep — synthetic 48-module repo (3 trials/arm; ultra 2/arm)

Placebo ÷ batching (i.e., how much more the uninstructed arm consumed):

effort	model cycles	raw tokens	weighted credits*	uninstructed behavior
low	1.00×	1.11×	0.98×	Code Mode, batches fine
medium	1.14×	1.01×	0.90×	mixed
high	1.06×	0.96×	0.89×	shell-batches via exec_command loops
xhigh	1.50×	1.56×	1.16×	serial plain calls, one per cycle
max	1.33×	1.36×	0.99×	serial plain calls, one per cycle
ultra	4.83×	4.44×	3.65×	23–25-session swarm, zero Promise use

* 125 / 12.5 / 750 credits per 1M uncached-input / cached-input / output.

Savings concentrate in cached input (context re-read on extra outer cycles); uncached input and output are nearly flat — same work, fewer round trips. Answer quality was graded against generated ground truth: 48/48 modules and 40/40 unused config keys, both arms, every effort level including ultra.

Real repositories — xhigh, 2 trials/arm each

Three private production Python repos (a pipeline monorepo with ~150 non-test files under its app dir; an agent-scaffolding repo with ~170 py files; a web-app backend with ~130 non-migration files). Task: 24-file inventory table after git ls-files enumeration. Arms injected via -c developer_instructions=...; repos never modified.

repo	cycles	raw tokens	credits
A	1.50×	1.41×	0.98×
B	2.79×	3.15×	1.47×
C	2.30×	2.66×	1.46×

Worst placebo trial: 25 cycles / 1.71M raw tokens; the instructed arm did the identical task in 7 cycles / 400k. Deliverables were equivalent (25–26-row tables, comparable length) in every pair.

Ultra findings

• Ultra spawned 23–25 sub-sessions per uninstructed trial despite features.multi_agent=false; the batching instruction shrank the swarm to 6–8 sessions and introduced 18–19 Promise.allSettled cells (vs 0).
• Uninstructed ultra: 305–652 cycles, 7.5–17.6M raw tokens, 267–605 credits per read-only task; the worst trial moved a Pro weekly meter 1% on its own.
• Even batched, ultra cost ~10× batched-xhigh for an identically-graded answer on this task.

Sub-agent config inheritance

0/48 placebo-arm ultra sub-sessions contained the batching text that was simultaneously present in the global ~/.codex/config.toml: sub-agents inherit the parent process's resolved config (including -c developer_instructions overrides) and do not re-read the global file.

Output-volume ("truncation") audit

Per the concern raised in #35050 (batched savings partly reflecting reduced evidence ingestion): summing all tool-output characters that entered each trial's transcript (audit_ingestion.py):

run	placebo	batching	ratio
synthetic-large @ xhigh	55.8k	55.9k	1.00×
real repo A @ xhigh	92.2k	70.4k	1.31×
real repo B @ xhigh	245.9k	169.7k	1.45×
real repo C @ xhigh	109.8k	70.7k	1.55×
synthetic-large @ ultra	849k	420k	2.02×

On the evidence-bounded synthetic task at xhigh, seen ingestion was identical to within 0.2% while consumption differed 1.56×. On real repos the batched arm saw 24–36% less tool output.

Elision follow-up (prompted by openai/codex#35421): a full-text scan of all …N tokens truncated… markers shows no 1 MiB capture-cap events anywhere in the 52 trials (no constant ≈252k claims; largest single output 156 KB) — the silent-discard path did not fire here. However, budget-level elision was concentrated in the batched exec path (33 markers, median 1,235 / max 48,634 claimed tokens — multi-call Promise.allSettled results exceeding the model-chosen max_output_tokens), while placebo shell-path markers were trivial (median 5 tokens). Crediting claimed-elided volume back as "attempted ingestion" compresses the real-repo gaps almost entirely: seen ratios 1.31× / 1.45× / 1.55× become attempted ratios ≈1.09× / 0.90× / 1.36×. The batched arm fetched roughly as much repository data as the placebo arm; the formatter trimmed the presentation.

Net: the consumption gaps (1.41–3.15×) are not explained by reduced reading. Both arms' answers stayed equivalent (synthetic: graded perfect at every tier), so budget elision did not measurably harm quality on these tasks — but it does argue for the output-safety guidance proposed for the exec-description fix (narrow ranges, deliberate max_output_tokens). Claimed elision figures are treated as approximate given #35421's finding that the shell-path constant is unreliable. At ultra, the uninstructed swarm read the ~250KB corpus ≈3.4× over (seen) vs ≈1.7× batched — redundant re-reading across 23–25 sub-agents, not deeper evidence.

Interpretation

The serialization pathology described in openai/codex#32503 / #35050 is effort-gated: absent at low/medium/high (the model batches on its own, often at the shell level), present at xhigh/max, extreme at ultra. A fix validated only at high effort will not observe it. The instruction that fixes xhigh/max/ultra is mildly negative at high (0.89× credits), so a naive unconditional prompt change may regress currently-healthy tiers — effort-conditional guidance looks necessary.