[Feature] Supporting dstack-inside-dstack for runs#4023
Merged
Conversation
r4victor
reviewed
Jul 13, 2026
un-def
reviewed
Jul 13, 2026
un-def
reviewed
Jul 13, 2026
Removing the job server connection on PROVISIONING/PULLING iterations reset the failure time tracked by the pool, so retry_timed_out() could never fire and jobs with a failing tunnel were never terminated by the server. Remove the connection only when the job leaves RUNNING; jobs_terminating covers the final cleanup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DSTACK_TOKEN alone now overrides the token of the project configured in config.yml. DSTACK_SERVER_URL still takes effect only together with DSTACK_TOKEN, and a missing token or project now produces an explicit error instead of a silent fallback when no config is available. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Enabling dstack access inside a run now reads like Docker-in-Docker (`docker: true`): `dstack: true`. No migration needed: the field is stored in run spec JSON only; pre-release DBs with the old field can be reset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reverse forward targeted 127.0.0.1 while the server may be bound to a specific address via DSTACK_SERVER_HOST, making the forward target unreachable. Derive the target from the bind address and include it in the error when the server is not reachable through the tunnel. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jvstme
reviewed
Jul 13, 2026
_get_lock and remove_all only run await-free expressions, so the extra lock adds nothing under the single-threaded event loop. Make _get_lock synchronous and drop the lock. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The persistent server connection counts as SSH activity, so a dev environment with both dstack access and inactivity_duration would never be detected as inactive. Reject the combination at configuration time instead of silently breaking inactivity_duration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With multiple server replicas, each replica used to overwrite the job's /run/dstack/server.sock on every open, so ownership churned between replicas (they repeatedly stole the socket from each other) and every overwrite briefly broke access. Probe the socket first and (re)create the reverse forward only when it is missing or unreachable; otherwise adopt the existing one. This keeps a single stable owner. If the owner replica dies, another takes over on its next health check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allow `dstack: true` on service configurations, mirroring tasks and dev environments. Each replica is a separate job that gets its own `/run/dstack/server.sock`, so multiple replicas behave like multi-node job replicas. Exclude the field from run specs sent to older servers, same as for tasks and dev environments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Let operators forbid `dstack: true` in run configurations, mirroring DSTACK_FORBID_SERVICES_WITHOUT_GATEWAY. When set, submitting a run with dstack access enabled is rejected server-side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a 'dstack in dstack' Protips section, and note in each dstack-access section that runs can also submit runs with dstack apply and attach to them with dstack attach, not just inspect them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
un-def
reviewed
Jul 14, 2026
jvstme
reviewed
Jul 14, 2026
The dstack field applied to exactly the run configuration types that extend BaseRunConfiguration, so define it there directly and drop the ConfigurationWithDstackParams mixin. The run spec exclude check then no longer needs the isinstance guard, since every run configuration has the field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jvstme
approved these changes
Jul 15, 2026
jvstme
left a comment
Collaborator
There was a problem hiding this comment.
The run configuration option and the connection pool implementation look good to me. Haven't looked at the rest
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Runs may need to call dstack themselves, for example to inspect runs, submit
another run, or attach to it. This currently requires the dstack server to
accept direct connections from the run.
UX
Add
dstack: trueto a run configuration.With
dstack: true, dstack exposes the server managing the run through/run/dstack/server.sockinside each job. It setsDSTACK_SERVER_URLto thecorresponding
http+unixURL andDSTACK_PROJECTto the current project,allowing the CLI and API clients to use that connection.
Authentication remains explicit. dstack does not inject
DSTACK_TOKEN; it canbe passed through the existing environment or secret mechanisms.
The CLI also accepts
DSTACK_SERVER_URL,DSTACK_PROJECT, andDSTACK_TOKENas a complete configuration without requiring
~/.dstack/config.yml.dstackcannot be combined withinactivity_durationon dev environments,since the persistent server connection counts as activity.
Implementation
Before starting commands, the server opens an SSH connection to each job and
creates a remote Unix-socket forward. Connections to
/run/dstack/server.sockinside the job are forwarded to the HTTP port of theserver managing the run.
DSTACK_SERVER_URLpoints clients to this socket.This uses the existing job SSH path, so the server does not need a public URL
or gateway. Health checks use the same socket path end to end; if the SSH
connection or socket becomes stale, the server closes and recreates the
tunnel. The tunnel is removed when the job terminates.
With multiple server replicas, a replica probes the socket and adopts an
existing healthy one instead of overwriting it, so jobs keep a single stable
server owner.