Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
166 changes: 91 additions & 75 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,33 +6,48 @@ java/Minecraft/.minecraft/
java/Minecraft/.minecraftserver/

# C/CUDA build artifacts (each c/ project also has its own .gitignore)
c/*/build/
c/**/*.o
c/**/*.d
c/magma/magma_demo
c/magma/magma_game
build/
magma/**/*.o
magma/**/*.d
blaze/**/*.o
blaze/**/*.d
magma/magma_demo
magma/magma_game
magma/magma_game_metal
magma/metal/*.air
magma/metal/*.metallib
magma/tests/test_raster_parity_metal
# craster test binaries + previews (sources test_*.c / test_*.sh ARE tracked)
c/magma/game/test_input_map
c/magma/game/test_world_live
c/magma/game/test_player_ctl
c/magma/game/test_hud
c/magma/game/test_entity_render
c/magma/game/test_sky
c/magma/tests/test_mesh_models
c/magma/game/*_preview.ppm
c/magma/game/*_preview.png
magma/game/test_input_map
magma/game/test_world_live
magma/game/test_player_ctl
magma/game/test_hud
magma/game/test_entity_render
magma/game/test_sky
magma/tests/test_mesh_models
magma/game/*_preview.ppm
magma/game/*_preview.png
# craster trace harness: binary + regenerable per-run output (sources under trace/ ARE tracked)
c/magma/trace_game
c/magma/trace/world_dump
c/magma/trace/out/
magma/trace_game
magma/trace/world_dump
magma/trace/out/
# regenerable render candidates (the mc_pose_*.png goldens ARE tracked)
c/magma/raster/verify/mc_capture/magma_pose_*.png
c/magma/raster/verify/mc_capture/magma_frame.png
verify/mc_capture/magma_pose_*.png
verify/mc_capture/magma_frame.png
# locally captured oracle pairs and generated tape reports
verify/ui_entities/goldens/*.png
verify/trace/report/tape_*.md
verify/trace/report/tape_*.gate.json

# logs + run artifacts
*.log
frame*.png
*.npy
/.tmp/
/run/
magma/.tmp/
# accidental nested trace/build output
magma/c/

# python
.venv/
Expand All @@ -43,77 +58,78 @@ __pycache__/

# stray gradle daemon if a build ever runs from repo root
/Minecraft/
c/magma/trace/alloctrack.so
c/magma/magma_game_dbg
c/magma/game/sky_pose_*.png
c/magma/game/sky_pose_*.ppm
c/magma/trace/uvfix/
c/magma/trace/out/
c/magma/game/test_biome_color
c/magma/game/Golden.class
c/magma/tests/test_light_brightness
c/magma/tests/test_fog_radial
c/magma/tests/test_model_oracle
c/magma/tests/*.class
c/magma/trace/trees_viewdistance_*.png
c/magma/trace/biome_scan
c/magma/game/test_play_compose
c/magma/game/test_tick_timer
c/magma/game/test_route_e2e
magma/trace/alloctrack.so
magma/magma_game_dbg
magma/game/sky_pose_*.png
magma/game/sky_pose_*.ppm
magma/trace/uvfix/
magma/trace/out/
magma/game/test_biome_color
magma/game/Golden.class
magma/tests/test_light_brightness
magma/tests/test_fog_radial
magma/tests/test_model_oracle
magma/tests/*.class
magma/trace/trees_viewdistance_*.png
magma/trace/biome_scan
magma/game/test_play_compose
magma/game/test_tick_timer
magma/game/test_route_e2e
# game test binaries have no extension; test_*.c / test_*.sh sources ARE tracked
c/magma/game/test_*
!c/magma/game/test_*.*
c/magma/magma_game_cuda
c/magma/raster/verify/mc_capture/*.bak_*
magma/game/test_*
!magma/game/test_*.*
magma/magma_game_cuda
verify/mc_capture/*.bak_*
java/Minecraft/qrl/
c/magma/raster/verify/mc_capture/gui_candidate
c/magma/raster/verify/tapes/*
verify/mc_capture/gui_candidate
verify/tapes/*
# per-tape accepted-divergence sidecars ARE tracked
!c/magma/raster/verify/tapes/*.known_divergences.json
c/magma/rl/out/*
!verify/tapes/*.known_divergences.json
blaze/rl/out/*
# canonical gate reference data (verify_cpu.py --chain / make_snapshots T0=1)
!c/magma/rl/out/chain_actions_s10.json
!c/magma/rl/out/coal_prefixes.json
!blaze/rl/out/chain_actions_s10.json
!blaze/rl/out/coal_prefixes.json

# ---- regenerated from YOUR Minecraft install (never committed) ----
# scripts/bootstrap_oracle.sh
java/oracle-src/
# scripts/bootstrap_assets.sh
c/magma/assets/atlas_gen.h
c/magma/assets/colormap_gen.h
c/magma/assets/gui_atlas.h
c/magma/assets/hand_atlas.h
c/magma/assets/inventory_ui_atlas.h
c/magma/assets/hud_atlas.h
c/magma/assets/item_atlas.h
c/magma/assets/loading_bg.h
c/magma/assets/mob_atlas.h
c/magma/assets/portal_tex.h
c/magma/assets/sky_atlas.h
c/magma/assets/underwater_tex.h
c/magma/assets/water_frames.h
magma/assets/atlas_gen.h
magma/assets/colormap_gen.h
magma/assets/gui_atlas.h
magma/assets/hand_atlas.h
magma/assets/inventory_ui_atlas.h
magma/assets/hud_atlas.h
magma/assets/item_atlas.h
magma/assets/loading_bg.h
magma/assets/mob_atlas.h
magma/assets/portal_tex.h
magma/assets/sky_atlas.h
magma/assets/underwater_tex.h
magma/assets/water_frames.h


# generated demo outputs (pack lives under c/magma/raster/verify/demo/)
# generated demo outputs (pack lives under verify/demo/)
demos/*
!demos/.gitkeep
c/magma/rl/blaze/*.so
c/magma/rl/blaze/blaze_verify
blaze/env/*.so
blaze/env/blaze_verify
*.nsys-rep

# ---- regenerated from YOUR Minecraft install (never committed) ----
# scripts/bootstrap_oracle.sh
java/oracle-src/
# scripts/bootstrap_assets.sh
c/magma/assets/atlas_gen.h
c/magma/assets/colormap_gen.h
c/magma/assets/gui_atlas.h
c/magma/assets/hand_atlas.h
c/magma/assets/inventory_ui_atlas.h
c/magma/assets/hud_atlas.h
c/magma/assets/item_atlas.h
c/magma/assets/loading_bg.h
c/magma/assets/mob_atlas.h
c/magma/assets/portal_tex.h
c/magma/assets/sky_atlas.h
c/magma/assets/underwater_tex.h
c/magma/assets/water_frames.h
magma/assets/atlas_gen.h
magma/assets/colormap_gen.h
magma/assets/gui_atlas.h
magma/assets/hand_atlas.h
magma/assets/inventory_ui_atlas.h
magma/assets/hud_atlas.h
magma/assets/item_atlas.h
magma/assets/loading_bg.h
magma/assets/mob_atlas.h
magma/assets/portal_tex.h
magma/assets/sky_atlas.h
magma/assets/underwater_tex.h
magma/assets/water_frames.h
103 changes: 80 additions & 23 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,24 +23,29 @@ One-shot clean box: `bash scripts/setup_and_verify.sh` (then `--full` with GPU).

## What this repo is

From-scratch C/CUDA reimplementation of Minecraft 1.11.2 (magma + mc-sim),
From-scratch C/CUDA reimplementation of Minecraft 1.11.2 (magma + blaze),
bit-verified against the real Java game, plus a batched CUDA RL env (blaze).
Product name: **netherite**. Trees:

- `java/` - playable Forge+Malmo/qrl client, launch scripts, oracle-src (bootstrap)
- `c/magma/` - product C game + software rasterizer + RL
- `c/mc-sim/` - simulation kernels (CPU == CUDA)
- `c/render-opt/` - verified render kernels + JNI drop-ins (lab closed)
- `java/` - the oracle: Forge+Malmo/NetheriteMod (mod id qrl) client, launch scripts, oracle-src
(bootstrap), render-opt kernel lab (closed)
- `blaze/` - the simulation: reference CPU + production CUDA tick (CPU == CUDA),
batched RL env (`blaze/env/`) and trainers (`blaze/rl/`)
- `magma/` - the playable fidelity tier: blaze's tick + software rasterizer
- `verify/` - cross-stack harness: tapes, scenarios, gates, nightly sweep

Glossary and naming rationale (incl. the blaze mob collision): `NAMES.md`.

## Where to read (stop when you have enough)

| Need | Open |
|------|------|
| First clone / no oracle-src | `docs/BOOTSTRAP.md` |
| How to play, VNC, qrl, sweep | `docs/RUNBOOK.md` |
| How to play, VNC, NetheriteMod, sweep | `docs/RUNBOOK.md` |
| Ship criteria / gate status | `docs/GATES.md` |
| Fidelity procedure | `c/magma/VERIFY.md` |
| Product contract / open bugs | `c/magma/PRODUCT.md`, `OPEN_DIVERGENCES.md` |
| Is X in the game? cut / pinned / open / unrecoverable | `docs/SCOPE.md` |
| Fidelity procedure | `magma/VERIFY.md` |
| Product contract / open bugs | `magma/PRODUCT.md`, `OPEN_DIVERGENCES.md` (closed forensics: `CLOSED_DIVERGENCES.md`) |
| Architecture for a tree | that tree's `SPEC.md` |
| History / lessons | `docs/DEVLOG.md` |
| Old reports | `docs/archive/` (ignore by default) |
Expand All @@ -61,25 +66,41 @@ bash scripts/demo_pixel_sbs.sh
# or stepwise:
bash scripts/bootstrap_oracle.sh
bash scripts/bootstrap_assets.sh
make -C c/magma game
make -C magma game
bash netherite_sweep.sh --quick

cd java/Minecraft && ./gradlew -g run/gradle build
uv run --no-project python c/mc-sim/oracle/runner.py <name>
uv run --no-project python blaze/oracle/runner.py <name>

# touching magma/cuda/raster_cuda.cu or magma/metal/raster_kernels.metal?
# The six kernels are hash-paired (verify/kernels/parity_manifest.json): edit
# BOTH twins, then prove it on both machines and re-record the manifest:
bash scripts/kernel_parity_gate.sh # anvil: cpu==cuda; macbook: cpu==metal

# wrapper-vs-owr worldgen census pin (CPU; blessed residuals in sidecar):
bash verify/worldgen/wrapper_gate.sh # rc=0 exact match vs known_divergences.json
# bash verify/worldgen/wrapper_diff.sh # diagnostic report + load-order probe
# bash verify/worldgen/wrapper_gate.sh --update # re-bless only with maintainer judgment
```

## Pixel investigation

When a tape frame is wrong, do not hand-roll numpy. `c/magma/raster/verify/trace`:
When a tape frame is wrong, do not hand-roll numpy. The tool lives at
`verify/trace/pxdiff.py` and `--tape` resolves replay output from
`verify/trace/out/tape_<NAME>/`, so run it from `verify/trace`:

```bash
U="uv run --no-project --with numpy,scipy,pillow python"
$U pxdiff.py selftest # trust the tool first
$U pxdiff.py frames --tape <NAME> # rank ticks by unexplained px
$U pxdiff.py clusters --tape <NAME> --tick N # cluster table + CAUSE
$U pxdiff.py zoom --tape <NAME> --tick N --cluster 0 --scale 10 -o /tmp/z.png
$U pxdiff.py probe --tape <NAME> --tick N --cluster 0 # every discriminator
$U pxdiff.py pixels --tape <NAME> --tick N --cluster 0 # exact RGB pairs
cd verify/trace
U() { uv run --no-project --with numpy,scipy,pillow python "$@"; } # bash+zsh
U pxdiff.py selftest # trust the tool first
U pxdiff.py frames --tape <NAME> # rank ticks by unexplained px
U pxdiff.py survey --tape <NAME> --tick N -o DIR # START HERE: top<=5
# clusters as numbered boxes on overview.png + zoom_N.png triptychs +
# survey.json; big unresolved clusters get tile-refined causes.
U pxdiff.py clusters --tape <NAME> --tick N # cluster table + CAUSE
U pxdiff.py zoom --tape <NAME> --tick N --cluster 0 --scale 10 -o /tmp/z.png
U pxdiff.py probe --tape <NAME> --tick N --cluster 0 # every discriminator
U pxdiff.py pixels --tape <NAME> --tick N --cluster 0 # exact RGB pairs
```

Causes: `texel-selection`, `shading-offset`, `registration`, `cutout-sky+/-`,
Expand All @@ -88,6 +109,23 @@ drives the mc_capture / ui_hud / ui_entities gates. `grind.py` ranks a whole
tape by mean/ch; `pixel_gate.py` decides pass/fail. Never report `unresolved`
as a diagnosis, and never claim a cause the tool did not measure.

Reading it right (each of these cost a cold agent real time):

- The `gate` column is pixel_gate's mask label (which budget absorbed the
pixels), not the faulty subsystem: a world-sized cluster classed `particles`
is NOT a particle bug, and `soak_from` in reports means spill from an
over-budget class. The `cause` column is the diagnosis.
- `sel` is exact-match texel selection; real minified surfaces usually carry a
small light delta on top, so trust the `tol4` column / probe field.
- Heed the frame-level notes. Many small clusters agreeing on one shift =
whole-frame registration. A giant `unresolved` cluster with
`structure_corr<=0` and `best_shift (0,0)` = the CAMERA moved: stop pixel
probing and diff the pose (tape jsonl `x/y/z/on_ground` vs
`out/tape_<NAME>/magma_state.jsonl` at that tick) - a sub-block Y error
remaps the whole scene (nether_elytra t=176: 0.93-block landing lag).
- px counts: survey/clusters count connected-component members; probe/pixels
count every differing pixel in the padded rect. Both are correct.

Two things that make a pixel measurement lie, both paid for already:

- **Do not replay tapes in parallel across worktrees** unless you are on
Expand All @@ -101,9 +139,18 @@ Two things that make a pixel measurement lie, both paid for already:
a uv cache or build tree there can fill it, kill its own run, and break every
other shell on the box (a codex run wrote 15 GB of CUDA wheels to
`/tmp/<name>-uv-cache` and died on "Disk quota exceeded"). When launching
delegates, pin `UV_CACHE_DIR=/home/infatoshi/.cache/uv` and
`TMPDIR=/home/infatoshi/dev/nw/.tmp` in their environment and say so in the
delegates, pin `UV_CACHE_DIR=$HOME/.cache/uv` and
`TMPDIR=$HOME/dev/nw/.tmp` in their environment and say so in the
prompt.
- **A retired tape used to measure as a silent PASS over zero frames.**
The recorder bakes an ABSOLUTE golden path into every tick row, so moving
a tape into `tapes/retired/` orphaned all of them;
`oracle_frames_cache` skipped every missing file without a word and the
pixel gate reported `PASS: no unexplained clusters over 0 frames`. Fixed
2026-07-29: goldens now fall back to `<dir of the tape file>/<frames dir>/`,
pxdiff resolves `tapes/retired/` too, and a tape that declares goldens but
resolves none is a FATAL, not a pass. If a gate reports 0 frames checked,
that is a harness failure - never read it as a clean tape.
- **Check what the goldens actually contain before chasing a diff.** A tape
recorded through Malmo has `hideGUI` forced on for the whole mission, so its
goldens have no HUD at all; `capture.hide_gui` in the tape meta is the
Expand All @@ -118,7 +165,17 @@ Two things that make a pixel measurement lie, both paid for already:
from it; tapes recorded before that field keep the old converged seed.
`MAGMA_FOG_C1_INIT=<0..1>` overrides the seed for sweeping it on old tapes.
Do not hardcode a value - it depends on the recording session, not the tape
(see `c/magma/OPEN_DIVERGENCES.md`).
(see `magma/OPEN_DIVERGENCES.md`).
- **The end-crystal healing beam needs the client's `ticksExisted`.**
`RenderDragon.renderCrystalBeams` scrolls `endercrystal_beam` by
`-ticksExisted*0.01` per tick over a 16x256 sheet that is ~2x minified, so a
one-tick phase error randomizes the whole glyph speckle. The recorder now
writes it per entity (dragon field 18, crystal field 12); tapes older than
that are reconstructed as `tick - first_seen + ent_ticks0`, default 7,
overridable with `MAGMA_ENT_TICKS0`. The default is a measured sweep, not a
guess: over the offset's full 100-tick period exactly one value is sharply
better (76.8k differing px vs 109-114k at all 99 others). Re-sweep it rather
than fitting anything else if a new End tape's beam looks like noise.
- **Measure a viewmodel residual against the render, not against a texel.**
Dividing a golden by a raw atlas texel prices in shading the oracle also
applies, and that is how a phantom "1.57x over-bright arm" got filed for a
Expand All @@ -132,7 +189,7 @@ Python: **UV only** (`uv run`, never bare `pip`/`python` for project work).
- Demos (png/mp4): scp to Mac; do not assume local image display.
- Human play: Moonlight/Sunshine `:0` or mcwindow (`docs/RUNBOOK.md`).
- Agent/trace: Xvfb `:1` via `bash java/start_vnc_client.sh` (VNC 5900, pw `redstone`).
- One client owns qrl port **25575** at a time.
- One client owns NetheriteMod port **25575** at a time.

## Gotchas

Expand All @@ -143,7 +200,7 @@ Python: **UV only** (`uv run`, never bare `pip`/`python` for project work).
- No emojis, no em dashes. Minimal diffs. Verify before claiming done.
- A replay that reports `magma_game failed (rc=-11)` and then
`EOFError: No data left in file` is a **SIGSEGV in the first captured frame**,
and the first thing to try is `make -C c/magma clean && make -C c/magma`. Seen
and the first thing to try is `make -C magma clean && make -C magma`. Seen
2026-07-25: an incremental build in the main tree produced a binary that
faulted inside `getenv` at the top of `gm_world_mesh_view` (a corrupted
`environ`, i.e. heap damage). The same commit built clean in a worktree, every
Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ auto-load has a project file; it does not duplicate the full map.
| First-clone bootstrap | `docs/BOOTSTRAP.md` |
| Play / VNC / qrl / sweep | `docs/RUNBOOK.md` |
| Product gates | `docs/GATES.md` |
| What's cut / pinned / open | `docs/SCOPE.md` |
| History | `docs/DEVLOG.md` |
| Fidelity | `c/magma/VERIFY.md` |
| Fidelity | `magma/VERIFY.md` |
| Why a pixel differs | `pxdiff.py` (AGENTS.md "Pixel investigation") |
| Isolated worktree for a delegate | `scripts/agent_worktree.sh NAME` |

Expand Down
Loading