Skip to content

fix(testing)!: a child reports on its own channel, not standard output - #70

Merged
mahdi-shafiei merged 1 commit into
mainfrom
fix/child-report-file
Oct 30, 2026
Merged

mahdi-shafiei merged 1 commit into
mainfrom
fix/child-report-file

Conversation

@mahdi-shafiei

Copy link
Copy Markdown
Contributor

Summary

A run_python child sends its result on a file only it knows, so no library output can be read as the result.

 child                                   parent (run_python)
-print(json.dumps(value))   -> stdout -> ChildResult.last_json_as(shape)   # last stdout line
+report(value)              -> $SUBSTRAX_CHILD_REPORT file
+                                       ChildResult.report_as(shape) / report_json()
 libraries (Gloo, C++)      -> stdout    stays in ChildResult.stdout
  • substrax.testing.report(value) writes once, to a staging file moved into place, and refuses a second report or a process run_python did not start.
  • ChildResult.report holds the text (None when the child sent none). report_as(shape) keeps the strict pydantic validation; report_json() decodes it unshaped.
  • run_example reads main()'s summary through the same channel.
  • last_json and last_json_as are removed. Every child in the tests, the README and docs/api/testing.md reports instead; import probes measure sys.modules before they import report.

Evidence

  • Before: datarax run 40894143757 (2026-10-29, Python 3.13 shard 4): test_multi_process_placement.py::TestRefusedAcrossProcesses::test_a_configuration_differing_between_processes_is_refused_in_every_process failed with json_invalid, input_value='[Gloo] Rank 1 is connect...'. Gloo's line arrived after the child's JSON.
  • After: test_a_report_survives_output_written_after_it has the child report, then print a Gloo-shaped line and write to file descriptor 1; the report reads back and stdout's last line is the descriptor write. Red first: the test module failed to import report.
  • Local, one directory per process: tests/testing 184 passed, tests/runtime 124, tests/checkpoint 96, tests/rng 24, tests/artifacts 16, tests/tracking 60. The 2 skips are GPU tests on a CPU lane.
  • Coverage 100% on child_report.py, child_process.py, examples.py and _example_main.py. ruff, format, pyright (0 errors) and every pre-commit hook pass.

Merge Danger

Door: two-way

Blast Radius: testing helpers

A breaking change to substrax.testing, which only test suites import. After the release and re-pin, datarax moves 7 parser calls in 5 files and the children of 13 run_processes callers, and artifex moves 2 parser calls and about 55 run_repo_json children. No other sibling calls last_json or last_json_as. One choice beyond the decision record: a second report in one child raises instead of replacing the first.

ChildResult.last_json and last_json_as parsed the last line of a child's standard output,
which the child shares with every library it loads. datarax's scheduled CI failed when Gloo
wrote "[Gloo] Rank 1 is connected to 1 peer ranks..." after a multi-process test's report,
so pydantic read Gloo's line as the report (run 40894143757).

run_python now creates a report file per run and names it in SUBSTRAX_CHILD_REPORT. The child
sends its value with substrax.testing.report, once, written and moved into place so the
parent reads all of it or none; ChildResult.report holds the text, report_as(shape) validates
it strictly and report_json() decodes it unshaped. Nothing printed to standard output or file
descriptor 1 reaches it. run_example reads main()'s summary the same way. last_json and
last_json_as are removed, and every child in the tests, README and API docs reports instead;
import probes measure before they import report.
@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 substrax | 🛠️ Build #35513567 | 📁 Comparing 76b7d63 against latest (79a37e8)

  🔍 Preview build  

2 files changed
± index.html
± api/testing/index.html

@mahdi-shafiei
mahdi-shafiei marked this pull request as ready for review October 30, 2026 16:01
@mahdi-shafiei
mahdi-shafiei merged commit adb9ecb into main Oct 30, 2026
12 checks passed
@mahdi-shafiei
mahdi-shafiei deleted the fix/child-report-file branch October 30, 2026 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant