Repository navigation
fix(testing)!: a child reports on its own channel, not standard output - #70
Merged
Merged
Conversation
ChildResult.last_json and last_json_as parsed the last line of a child's standard output, which the child shares with every library it loads. datarax's scheduled CI failed when Gloo wrote "[Gloo] Rank 1 is connected to 1 peer ranks..." after a multi-process test's report, so pydantic read Gloo's line as the report (run 40894143757). run_python now creates a report file per run and names it in SUBSTRAX_CHILD_REPORT. The child sends its value with substrax.testing.report, once, written and moved into place so the parent reads all of it or none; ChildResult.report holds the text, report_as(shape) validates it strictly and report_json() decodes it unshaped. Nothing printed to standard output or file descriptor 1 reaches it. run_example reads main()'s summary the same way. last_json and last_json_as are removed, and every child in the tests, README and API docs reports instead; import probes measure before they import report.
mahdi-shafiei
marked this pull request as ready for review
October 30, 2026 16:01
This was referenced Oct 30, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
run_pythonchild sends its result on a file only it knows, so no library output can be read as the result.substrax.testing.report(value)writes once, to a staging file moved into place, and refuses a second report or a processrun_pythondid not start.ChildResult.reportholds the text (Nonewhen the child sent none).report_as(shape)keeps the strict pydantic validation;report_json()decodes it unshaped.run_examplereadsmain()'s summary through the same channel.last_jsonandlast_json_asare removed. Every child in the tests, the README anddocs/api/testing.mdreports instead; import probes measuresys.modulesbefore they importreport.Evidence
test_multi_process_placement.py::TestRefusedAcrossProcesses::test_a_configuration_differing_between_processes_is_refused_in_every_processfailed withjson_invalid,input_value='[Gloo] Rank 1 is connect...'. Gloo's line arrived after the child's JSON.test_a_report_survives_output_written_after_ithas the child report, then print a Gloo-shaped line and write to file descriptor 1; the report reads back and stdout's last line is the descriptor write. Red first: the test module failed to importreport.tests/testing184 passed,tests/runtime124,tests/checkpoint96,tests/rng24,tests/artifacts16,tests/tracking60. The 2 skips are GPU tests on a CPU lane.child_report.py,child_process.py,examples.pyand_example_main.py. ruff, format, pyright (0 errors) and every pre-commit hook pass.Merge Danger
Door: two-way
Blast Radius: testing helpers
A breaking change to
substrax.testing, which only test suites import. After the release and re-pin, datarax moves 7 parser calls in 5 files and the children of 13run_processescallers, and artifex moves 2 parser calls and about 55run_repo_jsonchildren. No other sibling callslast_jsonorlast_json_as. One choice beyond the decision record: a secondreportin one child raises instead of replacing the first.