refactor(doto): orchestrate benchmark runs with Codex Skills

[gaoxiaobei]

Summary

Reworks the 23rd DOTO integration from a framework-owned LLM harness loop into a Codex-orchestrated benchmark built from versioned Skills and deterministic atomic tools.

Codex now decides which public-training evidence to inspect, what complete playerAI.cpp child to create, whether another iteration is justified, and which complete training version to select. AgentBenchFramework no longer owns an API client or automatic LLM iteration loop.

What changed

Harness → four Codex Skills

• doto-benchmark-run: authoritative Run lifecycle, training selection, one-shot finalization, validation, and export
• doto-game-rules: official mechanics, observations, legal macro-actions, action mask, terminal states, and strict KL support
• doto-agent-authoring: fixed native SDK contract and safe complete playerAI.cpp authoring
• doto-replay-reader: replay/trace schemas, events, evidence joins, and diagnosis

The old llm_client.py, loop.py, loop_config.py, doto loop command, loop example, and loop tests are removed. Build, match, replay, evaluation, IG, and lifecycle interfaces remain available for Codex to call.

Human population and evaluation

• freezes 43 complete human-policy identities at benchmark version doto-human-43-v1
• deterministic seed-11 split: 15 public training snapshots and 28 sealed hidden-test snapshots
• packages the 15 public source snapshots in one path-safe archive to keep the PR reviewable while preserving complete-source hashes
• evaluates both candidate seats
• public matrix: 30 cells; hidden matrix: 56 cells
• a human counts as defeated only when both seats terminate normally and candidate-oriented mean score difference is positive
• adaptive multiprocess scheduler targets 70% aggregate CPU with an optional worker hard cap; running matches are never cancelled

Atomic authoritative lifecycle

The run and iteration commands provide explicit state transitions:

1. initialize and snapshot source, pool identities, and exact Skills;
2. build an iteration;
3. run/resume the complete public matrix;
4. record strict IG or an exact missing reason;
5. close the iteration and align score/IG curves;
6. durably select a training candidate;
7. finalize the sealed hidden matrix exactly once;
8. validate and export the compatibility projection.

Completed cells are immutable. Build failures, timeouts, crashes, protocol errors, corrupt evidence, missing cells, and incomplete matrices remain explicit rather than being converted into ordinary losses or valid scores.

Metrics and Results

DOTO's continuous joint action space keeps strict deterministic KL semantics:

• identical canonical actions: 0;
• different deterministic supports: infinite, stored with numeric value: null and a reason;
• unavailable or invalid comparison: missing with an exact reason.

No score gain, coordinate distance, or action-change rate is relabeled as IG.

The complete authority is written to the independent DotoResults repository. The unchanged AgentBenchResults format receives exactly five derived files:

• run.toml
• summary.json
• score_curve.json
• ig_curve.json
• doto_results_ref.json

Recommended Codex prompt

docs/doto-harness.md now documents the zero-touch entry point: open the shared AgentBenchmark directory in Codex and send one copy-ready task. Codex locates the PR worktree or merged Framework checkout, reads all four Skills, prepares both pools, initializes the Run, iterates on public evidence, fixes the candidate before one sealed test, validates DotoResults, and exports the AgentBenchResults projection without manual commands.

Verification

• DOTO test suite on the PR branch: 89 passed, 1 skipped
  • the skipped audit requires the external AgentBench DOTO corpus, which is intentionally absent from an isolated checkout
• prompt/Skill documentation tests: 23 passed
• all four Skill packages pass quick_validate.py
• native build and official protocol paths are covered by the DOTO suite
• no generated formal training or hidden-test Run is included in this PR

Documentation

• docs/doto-harness.md
• docs/doto-official-acceptance.md
• skills/doto-benchmark-run/SKILL.md
• skills/doto-game-rules/SKILL.md
• skills/doto-agent-authoring/SKILL.md
• skills/doto-replay-reader/SKILL.md