feat(miracle): add replay-driven LLM harness loop

[gaoxiaobei]

Summary

This PR adds a runnable, replay-driven LLM harness for the 24th Miracle game:

• ports the official Miracle game logic and exposes one battle/replay CLI;
• defines the strategy interface, observation/action space, action mask, termination rules, and strict deterministic KL status;
• adds an OpenAI-compatible LLM iteration loop with SSE streaming enabled by default;
• saves immutable strategy versions, battles, official replays, readable traces, failures, budgets, score/IG curves, and AgentBenchResults-compatible Run metadata;
• adds formal Harness and Replay Reader Skills plus runnable examples.

Quick start

Run all commands from the AgentBenchFramework repository root. The only Miracle entry point is:

uv run python -m agentbench_frame.miracle <match|replay|ig|loop>

Show help with:

uv run python -m agentbench_frame.miracle --help
uv run python -m agentbench_frame.miracle match --help
uv run python -m agentbench_frame.miracle replay --help
uv run python -m agentbench_frame.miracle ig --help
uv run python -m agentbench_frame.miracle loop --help

1. Strategy interface

A Miracle strategy implements MiracleAgent:

from agentbench_frame.miracle.agent_bridge import MiracleAgent


class CandidateAgent(MiracleAgent):
    def choose_cards(self, camp):
        return {
            "artifacts": ["HolyLight"],
            "creatures": ["Archer", "Swordsman", "BlackBat"],
        }

    def act(self, obs):
        return {
            "operation_type": "endround",
            "operation_parameters": {},
        }

choose_cards(camp) selects one artifact and three creatures. act(obs) returns one official operation:

• summon: {type, level, position};
• move: {mover, position};
• attack: {attacker, target};
• endround: {};
• surrender: {};
• artifact actions supported by the official logic.

For manual match/ig commands, register a named strategy once in AGENTS at the bottom of src/agentbench_frame/miracle/agent_bridge.py:

AGENTS = {
    "endround": EndRoundAgent,
    "sample": SampleAgent,
    "my_agent": MyAgent,
}

The LLM loop does not register v0/v1/v2 globally. It loads the configured initial strategy and every generated CandidateAgent from immutable per-iteration source files.

2. Run a battle and save replay

uv run python -m agentbench_frame.miracle match \
  --agent0 sample \
  --agent1 endround \
  --seed 11 \
  --output-dir agentbench_data/replays/24_miracle \
  --tag sample-vs-endround

• agent0 is first player/camp 0; agent1 is second player/camp 1.
• For a seat-balanced comparison, swap the agents and run a second battle with the same seed.
• seed fixes official-logic randomness such as map type and day/night values.
• stdout is JSON containing winner, scores, rounds, terminated_by, errors, replay, and trace.
• A fully valid battle has terminated_by="normal" and errors=[].

Each battle saves:

• *.mrc: official binary replay;
• *.mrc.trace.jsonl: readable host/logic traffic, including observations and actions.

The same interface is callable from Python without registration:

from agentbench_frame.miracle import run_match
from my_agent import CandidateAgent, OpponentAgent

result = run_match(
    CandidateAgent(),
    OpponentAgent(),
    seed=11,
    replay_dir="agentbench_data/replays/24_miracle",
    tag="candidate-vs-opponent",
)
print(result.winner, result.scores, result.replay_path, result.trace_path)

3. Read a replay

Parse the official .mrc into an event timeline:

uv run python -m agentbench_frame.miracle replay \
  --path agentbench_data/replays/24_miracle/<match>.mrc \
  --jsonl agentbench_data/replays/24_miracle/<match>.events.jsonl

The command prints the winner, final round, and event counts. --jsonl is optional.

Use the paired .mrc.trace.jsonl to inspect the exact observations sent to an agent and the operations returned by it. Important replay semantics:

• .mrc is binary, not JSON;
• trace payload.content[0] has a six-character length prefix before its JSON body;
• outer trace state is not the game round; use the decoded observation's round;
• score 30000 is an official win/loss marker, not 30,000 ordinary points;
• units[i] is an 18-field array documented in skills/miracle-replay-reader/SKILL.md;
• CAN_MOVE/CAN_ATK = 0 means unavailable this turn, often because the unit was just summoned or already acted;
• attack target=camp means attack that camp's Miracle;
• obstacles and the complete boundary table are not included in obs; official logic remains the final legality arbiter.

The full event-code, unit-field, creature/artifact-code, damage-type, and buff-type tables are in skills/miracle-replay-reader/SKILL.md.

4. Compare old and new deterministic policies (strict KL status)

First generate a real trace with match, then replay the same observation sequence through two registered strategies:

uv run python -m agentbench_frame.miracle ig \
  --trace agentbench_data/replays/24_miracle/<match>.mrc.trace.jsonl \
  --old old_agent \
  --new new_agent \
  --camp 0 \
  --iteration 1 \
  --output-dir agentbench_data/ig/24_miracle

Run again with --camp 1 to compare decisions for the other seat. Outputs include per-decision records and an aggregated ig_curve.json.

Strict deterministic KL reporting is intentionally not replaced by a proxy metric:

• unchanged_ratio: old/new legal actions match, so strict KL is 0;
• infinite_ratio: deterministic actions differ, so strict KL diverges;
• missing_ratio: an action is invalid, unparsable, or outside the finite action support;
• finite_kl_mean: mean of genuinely finite KL values only; otherwise null.

The finite support is generated from each observation and includes end-round, surrender, legal summons, moves, attacks, and artifact uses. Official logic still performs final adjudication.

5. Run the OpenAI-compatible LLM harness loop

Copy and edit examples/miracle-loop.toml:

agent = "tested_llm"
initial_strategy = "miracle-initial-strategy.py"
opponent = "sample"

[llm]
base_url = "https://api.openai.com"
api_key_env = "OPENAI_API_KEY"
model = "your-model-name"
temperature = 0.0
timeout_seconds = 120
reasoning_effort = "low" # optional; omit if unsupported
stream = true
max_context_tokens = 1000000

[evaluation]
seeds = [11]
seats = [0, 1]

[budget]
max_iterations = 3
max_rollouts = 8
max_episode_reads = 3
max_decision_reads = 500
max_total_tokens = 3000000
max_wall_seconds = 1800

initial_strategy is resolved relative to the TOML file. It must contain a complete CandidateAgent implementation.

Set the key only through the configured environment variable and start the Run:

export OPENAI_API_KEY='<API key>'

uv run python -m agentbench_frame.miracle loop \
  --config examples/miracle-loop.toml \
  --data-dir ../AgentBenchResults

The API contract is OpenAI-compatible POST /v1/chat/completions. SSE streaming is enabled by default to survive long generations and gateway idle timeouts. Set stream=false only for a provider that does not support streaming.

max_tokens is omitted by default and is sent only when explicitly configured. max_context_tokens defaults to 1,000,000 and is checked against authoritative prompt + completion usage returned by the provider; the harness does not pretend that character counts are token counts.

What the LLM sees

Every update request contains:

1. a system contract requiring exactly one JSON object with complete source;
2. the complete miracle-harness Skill;
3. the complete miracle-replay-reader Skill;
4. the current accepted strategy source;
5. the immediately previous iteration's metrics/status;
6. selected structured replay evidence (episode outcome plus budget-limited observations);
7. cumulative budget consumption.

The LLM must return:

{
  "analysis": "problem found and reason for this update",
  "strategy_code": "complete Python source defining CandidateAgent"
}

Patches, shell commands, or partial methods are rejected. The harness validates, saves, loads, battles, and compares the returned source itself.

Budget semantics

All limits are cumulative across one Run:

• max_iterations: LLM update attempts, excluding baseline iteration 0;
• max_rollouts: battles actually started;
• max_episode_reads: replay episodes exposed to the LLM;
• max_decision_reads: individual structured observations exposed to the LLM;
• max_total_tokens: provider-reported total tokens;
• max_context_tokens: provider-reported tokens in any single request;
• max_wall_seconds: total Run wall-clock time.

A Run is one complete benchmark execution; an iteration is one strategy-update attempt; an episode is one battle; a round is one in-game round.

Invalid model output, invalid strategies, API errors, battle timeouts, incomplete evaluations, and missing IG are retained with explicit status/reason. They are never silently removed from curves.

6. Output layout and interpretation

--data-dir ../AgentBenchResults produces:

AgentBenchResults/
└── runs/24_miracle/<agent>/<run_id>/
    ├── run.toml
    ├── config.json
    ├── summary.json
    ├── events.jsonl
    ├── score_curve.json
    ├── ig_curve.json
    ├── skills/
    └── iterations/
        ├── iteration-0000/
        │   ├── strategy.py
        │   └── episodes/
        └── iteration-0001/
            ├── llm_request.json
            ├── llm_response.json
            ├── candidate.py
            ├── strategy.py
            ├── iteration.json
            ├── episodes/
            └── ig/

• iteration-0000 is the raw baseline.
• strategy.py is the immutable accepted strategy for that iteration.
• candidate.py preserves generated source before/while validation.
• episodes/ contains result JSON, official .mrc, and readable .trace.jsonl.
• events.jsonl is the ordered audit log for API calls, battles, failures, and iteration transitions.
• summary.json contains raw/evolved score, gain, best iteration, AUC, failure counts, and cumulative budgets.
• score_curve.json keeps version-aligned raw, evo, gain, win/completion rates, rollouts, token, episode-read, and time axes.
• ig_curve.json keeps version-aligned strict-KL status ratios. Missing/incomplete points remain visible.

API keys are read from the configured environment variable and are not saved in Run files.

Validate exported Results with:

uv run agentbench data check --data-dir ../AgentBenchResults

7. Verification

PYTHONPATH=src python -m pytest tests/miracle -q

• 61 Miracle tests passed.
• A real streamed three-iteration Run completed.
• Export validation reported 6 valid, 0 invalid across retained Runs.

Artifact policy

Generated Runs, .mrc replays, traces, temporary API configuration, caches, credentials, and AgentBenchResults are intentionally excluded from this PR. They are reproducible runtime artifacts, not source files.

Canonical detailed references:

• skills/miracle-harness/SKILL.md
• skills/miracle-replay-reader/SKILL.md
• examples/miracle-loop.toml
• examples/miracle-initial-strategy.py