Skip to content

Unify fresh-input solver measurement - #38

Merged
plancherb1 merged 6 commits into
mainfrom
eval/unified-solver-measurement
Sep 1, 2026
Merged

Unify fresh-input solver measurement#38
plancherb1 merged 6 commits into
mainfrom
eval/unified-solver-measurement

Conversation

@plancherb1

@plancherb1 plancherb1 commented Sep 1, 2026

Copy link
Copy Markdown
Member

Summary

This replaces the special-case NVIDIA-thread solver confirmation/veto with one symmetric measurement path for every destructive solver contender.

  • adds an authoritative fresh-input POTRF/TRSV/POSV ladder for native Block/Warp/Thread and NVIDIA Block/Thread
  • sweeps and preserves the exact launch configuration, not just the implementation tier
  • gives every timed solver launch a distinct valid system, with initialization outside timing
  • randomizes contender order in paired rounds and retains every raw sample
  • requires the solver companion whenever ladder data regenerates defaults or figures
  • fails closed on incomplete joins and on selected solver plans above the stated 5% spread threshold
  • records robust median and raw samples for host-library and Riccati paper harnesses
  • removes the obsolete NVIDIA-thread-only validity benchmark and its veto terminology
  • updates tuning, quiet-window, and Jetson documentation

No dispatch defaults or publication timing results change in this PR. The reviewed harness will produce two independent captures per GPU: Capture A selects the exact implementation-plus-launch plan, and Capture B evaluates that frozen plan.

Why

The old workflow timed destructive operations in the general ladder, then treated NVIDIA-thread differently with a separate confirmation gate. It could also transfer an implementation family across devices while silently choosing a target-tuned launch shape. That made the figure hard to explain and mixed tuning with evaluation.

The new workflow treats every supported solver implementation identically and carries the complete selected plan through held-out and cross-architecture evaluation.

Verification

  • pytest -q test/test_tuning_tools.py — 9 passed
  • native bench_solver_ladder.cu compile on sm_120 — passed
  • MathDx-enabled bench_solver_ladder.cu compile on sm_120 — passed
  • paper host-library and Riccati harness builds on sm_120 — passed
  • serial MathDx prebuild on Jetson AGX Orin sm_87 — passed
  • Orin solver-ladder smoke run: 555 results, 0 skips

Publication sweeps are intentionally deferred until this measurement code is reviewed, all hosts use the final clean commit, Jetson clocks are pinned, and the GPU is quiet.

Companion paper PR

The paper-side held-out joins, manifest, runbook, and provisional labeling are in plancherb1/glass-paper#4.

@plancherb1

Copy link
Copy Markdown
Member Author

Jetson bring-up found two test-infrastructure constraints worth separating from this PR's measurement code:

  1. Xavier's stock Python is 3.8. The tuning entry point initially failed while evaluating list[str]; commit a47a651 adds postponed annotation evaluation, and the tuning contracts plus both ladder binaries now build on Xavier/CUDA 11.4.
  2. Jetson does not provide nvidia-smi, so the generic pytest harness takes its documented sm_75 fallback unless CUDA_ARCH is explicit. Using CUDA_ARCH=sm_87 on Orin and sm_72 on Xavier fixes the resulting no kernel image error. The unsafe fallback should become a fail-fast follow-up.
  3. Large-output tests on both boards hit the CUDA device-printf FIFO: representative failures return exactly 2,048 floats even when 2,500–11,448 are expected. The cached failures are confined to large-output cases (thread, a subset of robotics, and one reduced_blas case), while smaller outputs pass. A follow-up should make test/cuda/helpers.cuh::print_device_vec copy to host and print there instead of using device printf.

These are harness transport/configuration issues, not evidence of a numerical mismatch. The authoritative local receipt in 59d6ab3 reruns all eight shards fresh on sm_120 and passes 4,738 tests; PR CI verifies the signature and fingerprints. The new error-checking solver ladder also compiles and runs on both Jetsons at their explicit native architectures.

@plancherb1

plancherb1 commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

Overnight Jetson execution exposed and fixed a long-run transport deadlock before any paper data was promoted.

The fresh-input solver emits all raw paired-round samples. run_isolated() launched it with stdout=PIPE, then watch_process() polled for process exit before communicate() drained that pipe. On Orin the child eventually blocked in pipe_write with GR3D idle. Commit 470979b replaces the unread pipe with a seekable temporary capture, preserving all-or-nothing output while allowing the watcher to run, and adds a resume path: --from-ladder <mega> now reuses a completed non-destructive capture and records only the missing solver companion.

Regression coverage:

  • a contract asserts timed output uses a seekable capture rather than PIPE;
  • a contract asserts --from-ladder without --from-solver-ladder resumes the solver-only path;
  • both run in CPU CI and the signed tiers shard.

Commit 81d5f2e carries a newly generated, release-policy-verified receipt: all 8 shards fresh, 4,740/4,740 tests passed, signature/fingerprint/ancestry valid, and 21/21 required obligations. Orin is now resuming from its persisted 6/6 mega file on the fix. Tonight's unpinned/mixed-orchestration captures remain explicitly exploratory and will not be presented as final paper evidence.

@plancherb1
plancherb1 merged commit 9814536 into main Sep 1, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant