Repository navigation
compute: process-tree cleanup, run handles keep their settings, SkyPilot SDK and copy transport - #64
Merged
Conversation
…celled On a timeout the worker killed the task's process alone, so a descendant it had started kept running with the task's log pipes open, and the worker waited on them. Each task now runs in its own session, and the worker kills that process group when the task exits, times out or is cancelled. The worker records SIGTERM and SIGINT and acts on them between polls of the task, so a cancellation finishes the cleanup before the worker exits, including for a task that ignores SIGTERM. The local backend's cancel signals the worker, which now reaps the task group it no longer shares.
substrax-compute rebuilt a stored run's backend from the project's configuration as it is now, so editing or removing a backend's settings redirected or broke every run submitted before the edit. RunHandle gains a required settings field, which every backend fills through new_run(spec, project, settings=...) with its validated settings, its state directory resolved to an absolute path; the CLI rebuilds the backend from it alone. Handles are version 2, and a version-1 handle is refused: it holds no settings to rebuild from. RunStore.read_all() replaces handles(): it returns the readable handles and each unreadable record with the reason, so one old or damaged record no longer hides every run. substrax-compute status lists what it can read, names each record it cannot, and exits 1. BREAKING CHANGE: new_run takes settings and RunHandle requires them; version-1 handles are refused; RunStore.handles() is now read_all().
The skypilot backend drove the sky command and parsed what it printed: `sky queue -o json` mixed notices into its output, a job status it did not know read as a failed job, and a launch interrupted before it returned left no record of the request SkyPilot was still provisioning. Every operation now runs substrax/compute/sky_sdk.py in SkyPilot's own interpreter (sky_python, by default the python beside the sky command), which calls the public SDK and returns its native records as JSON. Before it waits, a launch writes the run's intent and SkyPilot's request id beside the task file. cancel leaves a finished run alone; otherwise it cancels that request and the job and tears the cluster down, attempting every step and raising an ExceptionGroup of the failures, which substrax-compute reports with exit code 1. teardown() removes a cluster on its own. An unknown job status raises; FAILED, FAILED_SETUP and FAILED_DRIVER read as failed. How a run's files move is a transport's, in skypilot_transports: the bucket transport holds what the backend did with buckets, so the backend no longer branches on it. BREAKING CHANGE: the backend needs SkyPilot's interpreter to import sky; the `sky` command alone is not used to run operations.
… transport Providers such as RunPod and Vast cannot mount the buckets the skypilot backend relied on, so it could not run there. The copy transport has SkyPilot copy each mount's local source onto the cluster, read-only; a writable mount is refused, since a copy cannot be written back. The worker runs under bubblewrap with the system, the project and the inputs read-only, and network_isolation also takes its network away. fetch copies the outputs over rsync and verifies them against manifests of the remote directory (sky_transfer) taken before and after; a difference keeps the cluster up. The cluster stays up until teardown() or its lifetime_minutes timer, which starts after setup, and status reads a verified copy once the cluster's queue is gone. Each transport's settings are a table of their own, and the backend configures exactly one: [...skypilot.bucket] (outputs_bucket, buckets) or [...skypilot.copy] (sources, disk_size, lifetime_minutes, network_isolation). setup_commands prepares an image before the locked environment is synced. Bucket transport refuses a provider that cannot mount buckets. BREAKING CHANGE: outputs_bucket and buckets move under [tool.substrax.compute.backends.skypilot.bucket].
mahdi-shafiei
marked this pull request as ready for review
October 28, 2026 23:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compute: a task's process tree is stopped with it, a run keeps the settings it was submitted with, SkyPilot runs through its public SDK, and SkyPilot gains a copy transport for providers that cannot mount buckets (RunPod, Vast). Four commits, each passing the compute suite, every pre-commit hook and the strict docs build on its own.
1.
fix(compute): stop a task's whole process tree. Each task runs in its own session; the worker kills that group when the task exits, times out or is cancelled, and turns SIGTERM and SIGINT into that cleanup. A local cancel now stops the task tree, a task that ignores SIGTERM included (test_local_cancel_stops_task_and_descendant).2.
feat(compute)!: a run handle keeps its settings.RunHandle.settingsis required and filled bynew_run(spec, project, settings=...); the CLI rebuilds a run's backend from it, never from the project's current configuration. Handles are version 2; a version-1 handle is refused.RunStore.read_all()replaceshandles()and returns each unreadable record with its reason, sosubstrax-compute statuslists every run it can read, names the rest, and exits 1.3.
feat(compute)!: SkyPilot through its public SDK. Every operation runssky_sdk.pyin SkyPilot's own interpreter and reads native records. A launch records its intent and request id before waiting.cancelleaves a finished run alone; otherwise it cancels the request and the job and tears the cluster down, raising anExceptionGroupof the failures (CLI exit 1). An unknown job status raises;FAILED_DRIVERreads as failed. Bucket handling moves into a transport (skypilot_transports).4.
feat(compute)!: copy transport, and one settings table per transport.[...skypilot.copy]copies read-only local inputs, runs the worker under bubblewrap (network_isolationopt-in), and fetches outputs over rsync verified against before-and-after manifests.outputs_bucketandbucketsmove under[...skypilot.bucket]; exactly one transport is configured.Breaking changes (no stack repository uses any of them on its main):
new_runandRunHandlerequire settings; version-1 handles are refused;RunStore.handles()isread_all(); the SkyPilot backend needs SkyPilot's interpreter;outputs_bucketandbucketsmove under.bucket. The CHANGELOG says what a user does about each.Verification, on each commit's exact tree, exported from the index:
tests/compute: 121, 127, 143 and 163 passed, none skipped (the bubblewrap isolation test runs).mkdocs build --strictpasses on commits 3 and 4.Not checked: a real provider run on any cloud, and copy transport on RunPod or Vast.