Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ It owns:

- Host records and lifecycle state in Postgres
- Inline provisioning in `POST /hosts`
- Provider VM creation and deletion (exe.dev, AWS, Hetzner, local Docker)
- Provider VM creation and deletion (exe.dev, AWS, Hetzner, Exoscale,
local Docker, Docker Sandboxes)
- Tailscale auth key creation, device discovery, and cleanup
- SSH host key scanning and `known_hosts` material
- Account-bound exe.dev HTTP proxy resources
Expand Down
112 changes: 104 additions & 8 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,18 @@ to keep the credential-holding control plane off other interfaces.

## Choose a provider

Four remote providers are supported and verified end to end: `exe`
(exe.dev), `aws` (EC2), `hetzner` (Hetzner Cloud), and `exoscale`
(Exoscale). A fifth, `docker`,
runs sandboxes as local containers and needs no external account — see
[Local sandboxes with Docker](#local-sandboxes-with-docker). `DEFAULT_HOST_PROVIDER`
selects which one serves `POST /hosts` (default `exe`). Set the matching
provider variables below. The image ships with all provider extras
installed.
| Provider | Sandboxes | Where |
| --- | --- | --- |
| `exe` | exe.dev VMs | Remote |
| `aws` | EC2 instances | Remote |
| `hetzner` | Hetzner Cloud VMs | Remote |
| `exoscale` | Exoscale VMs | Remote |
| `docker` | Containers ([Local sandboxes with Docker](#local-sandboxes-with-docker)) | Local, no external account |
| `docker-sbx` | microVMs ([Local microVMs with Docker Sandboxes](#local-microvms-with-docker-sandboxes)) | Local |

`DEFAULT_HOST_PROVIDER` selects the provider for `POST /hosts` (default
`exe`). Set the matching provider variables below. The image contains
all provider extras.

## Local sandboxes with Docker

Expand Down Expand Up @@ -100,6 +104,81 @@ same socket mount and socket-GID supplemental group. `DOCKER_HOST` remains
available when the daemon is remote or rootless instead of exposed through
`/var/run/docker.sock`.

## Local microVMs with Docker Sandboxes

The `docker-sbx` provider runs each sandbox as a
[Docker Sandboxes](https://docs.docker.com/ai/sandboxes/) microVM. Each
microVM has its own kernel, its own filesystem, and its own Docker
daemon. The sandboxd network policy controls the egress. This provider
is local to the drukbox machine, the same as the `docker` provider. It
does not support Tailscale.

Prepare the host fully before drukbox starts. drukbox only connects to
the host:

1. Install Docker Engine and `docker-sbx`. Ubuntu 24.04+ with KVM is
necessary: `/dev/kvm` must exist, and the service user must be in the
`kvm` group.
2. Sign in one time with `sbx login`. Headless hosts use a device-code
flow.
3. Start the daemon: `sbx daemon start -d --policy balanced`.

Docker documents `sbx` as a tool for the daemon owner's own user on the
host. Thus the simplest deployment runs drukbox directly on the host, as
the same user:

```bash
uv run uvicorn api.app:app --host 127.0.0.1 --port 8780
```

In this mode, no mounts and no extra variables are necessary. The CLI
finds the daemon socket automatically, and the `127.0.0.1` default for
`DOCKER_SBX_ADVERTISE_HOST` is correct.

drukbox can also run as a container adjacent to the daemon. Docker does
not document this mode; drukbox uses the CLI's own daemon-endpoint
variable, `DOCKER_SANDBOXES_API`. Mount the daemon socket, the `sbx`
binary of the host (then the CLI version and the daemon version always
agree), the CLI auth store, and the workspace root:

```bash
docker run --rm --network host \
--mount type=bind,src=$HOME/.local/state/sandboxes/sandboxes/sandboxd/sandboxd.sock,dst=/run/sandboxd.sock \
--mount type=bind,src=$(command -v sbx),dst=/usr/local/bin/sbx,readonly \
--mount type=bind,src=$HOME/.config/com.docker.sandboxes,dst=/root/.config/com.docker.sandboxes,readonly \
--mount type=bind,src=$HOME/.drukbox/sbx-workspaces,dst=$HOME/.drukbox/sbx-workspaces \
--env DOCKER_SANDBOXES_API=unix:///run/sandboxd.sock \
--env DOCKER_SBX_WORKSPACE_ROOT=$HOME/.drukbox/sbx-workspaces \
--env DOCKER_SBX_ADVERTISE_HOST=172.17.0.1 \
--env-file drukbox.env \
ghcr.io/czpython/drukbox:latest
```

The daemon reads workspace paths on its own filesystem. Thus the
workspace mount must have the same path on the host and in the
container. The janitor and pool containers need the same mounts and
variables. `DOCKER_SBX_ADVERTISE_HOST` is the Docker bridge address
here, because the sandbox SSH ports must be open to the drukbox
container, not only to the host loopback interface.

Only this machine can connect to the sandboxes. The key for each host
is the auth boundary. Sandboxes have no `SERVICE_LABEL` tag, because
`sbx create` has no label option.

The template image (`DOCKER_SBX_DEFAULT_IMAGE`, default
`ghcr.io/czpython/drukbox/sbx-sandbox:latest`) must start sshd without
environment variables. `sbx create` sends none. drukbox injects the key
for each host through the exec channel after the start. Build
[images/sbx/](../images/sbx/) to change the template. The
`images/local/` entrypoint needs boot-time environment variables and
cannot start as a sandbox template.

A sandbox creation takes approximately 20 seconds with a warm template
cache, and more than 30 seconds at the first pull. Thus a warm pool
(`POOL_SIZES`) is useful. Each sandbox gets the explicit
`DOCKER_SBX_CPUS` and `DOCKER_SBX_MEMORY` sizes. Without them,
the daemon gives one sandbox all host CPUs and half of the host memory.

## Choose a networking mode

`TAILSCALE_ENABLED=false` (default): callers reach sandboxes over the
Expand Down Expand Up @@ -258,3 +337,20 @@ rootless daemon. Drukbox mints a per-VM ed25519 key and publishes sshd on a
random `127.0.0.1` port. See
[Local sandboxes with Docker](#local-sandboxes-with-docker) for the
container command and the trust caveat.

Docker Sandboxes provider:

| Variable | Default | Purpose |
| --- | --- | --- |
| `DOCKER_SBX_DEFAULT_IMAGE` | `ghcr.io/czpython/drukbox/sbx-sandbox:latest` | Template image that contains sshd and starts without environment variables. Build `images/sbx/Dockerfile` to change it. |
| `DOCKER_SBX_SSH_USERNAME` | `root` | User in the sandbox for caller SSH access. |
| `DOCKER_SBX_BOOTSTRAP_SSH_TIMEOUT_SECONDS` | `30.0` | Time limit for the ssh-keyscan tries on a new sandbox. |
| `DOCKER_SBX_ADVERTISE_HOST` | `127.0.0.1` | Host address for the published SSH ports. Use the Docker bridge address when drukbox runs in a container. |
| `DOCKER_SBX_CPUS` | `2` | Number of CPUs for each sandbox. |
| `DOCKER_SBX_MEMORY` | `2g` | Memory for each sandbox, in binary units. |
| `DOCKER_SBX_WORKSPACE_ROOT` | `~/.drukbox/sbx-workspaces` | Directory with one temporary workspace for each sandbox. The path must be the same for drukbox and for the daemon. |

The published image does not contain the `sbx` CLI. Mount the binary and
the auth store of the host, as
[Local microVMs with Docker Sandboxes](#local-microvms-with-docker-sandboxes)
shows. Set `DOCKER_SANDBOXES_API` to the mounted daemon socket.
5 changes: 3 additions & 2 deletions docs/networking.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ shaped the way it is. For turning these modes on, read
## Two modes

`TAILSCALE_ENABLED` selects between two networking models. A provider
whose hosts cannot join a tailnet (docker — local containers) always
takes the external path, whatever the mode. The API response carries
whose hosts cannot join a tailnet (docker — local containers,
docker-sbx — local microVMs) always takes the external path,
whatever the mode. The API response carries
both addresses; which is populated depends on the mode and the
provider:

Expand Down
10 changes: 5 additions & 5 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@ no rate limiting (see [Resource exhaustion](#resource-exhaustion)).
How a caller reaches a sandbox, and the tradeoffs of each path, are
covered in [Networking](networking.md). The security-relevant summary:

- **Per-VM keys.** On AWS (Tailscale off) and Hetzner, drukbox mints a
fresh ed25519 keypair per VM, returns the private half **once** in
the create response, and never persists it — a later
`GET /hosts/{id}` returns `private_key: null`. The key is the auth
boundary; password auth is never enabled.
- **Per-VM keys.** On AWS (Tailscale off), Hetzner, docker, and
docker-sbx, drukbox mints a fresh ed25519 keypair per VM, returns
the private half **once** in the create response, and never persists
it — a later `GET /hosts/{id}` returns `private_key: null`. The key
is the auth boundary; password auth is never enabled.
- **AWS ingress fail-open.** The managed `drukbox-managed` security
group opens SSH to the detected egress `/32`, or to whatever
`AWS_SSH_CIDRS` specifies. If egress detection fails and no CIDRs are
Expand Down
20 changes: 20 additions & 0 deletions images/sbx/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Sandbox template for the `docker-sandbox` provider: Ubuntu + sshd.
#
# docker build -t drukbox/sbx-sandbox:latest images/sbx/
#
# This template does not need environment variables at start, because
# `sbx create` cannot send them. drukbox injects the SSH key through the exec
# channel when the sandbox runs. The images/local/ entrypoint needs boot-time
# environment variables. Thus it cannot operate as a sandbox template.
FROM ubuntu:24.04

RUN apt-get update \
&& apt-get install -y --no-install-recommends openssh-server ca-certificates sudo \
&& rm -rf /var/lib/apt/lists/* \
&& mkdir -p /run/sshd

COPY entrypoint.sh /usr/local/bin/drukbox-entrypoint
RUN chmod +x /usr/local/bin/drukbox-entrypoint

EXPOSE 22
ENTRYPOINT ["/usr/local/bin/drukbox-entrypoint"]
16 changes: 16 additions & 0 deletions images/sbx/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
# Boot entrypoint for the drukbox Docker Sandboxes template. The script starts
# sshd. drukbox injects the SSH key through the exec channel after the start.
# sshd reads authorized_keys for each authentication. A restart is not
# necessary after the key injection.
#
# Do not make an existing authorized_keys file empty. The entrypoint runs
# again at each sandbox restart, after the key injection.
set -euo pipefail

install -d -m 700 /root/.ssh
[ -f /root/.ssh/authorized_keys ] || install -m 600 /dev/null /root/.ssh/authorized_keys

ssh-keygen -A

exec /usr/sbin/sshd -D -e
1 change: 1 addition & 0 deletions src/providers/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import providers.aws
import providers.docker
import providers.docker_sbx
import providers.exe
import providers.exoscale
import providers.hetzner # noqa: F401
4 changes: 4 additions & 0 deletions src/providers/docker_sbx/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from providers.docker_sbx.provider import DockerSbxProvider
from providers.registry import register_vm_provider

register_vm_provider(DockerSbxProvider)
133 changes: 133 additions & 0 deletions src/providers/docker_sbx/api.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
import asyncio
import json
import re

from .exceptions import DockerSbxNotFoundError, DockerSbxTransportError

# The first `sbx create` can pull a large template. The time limit is large
# because it must stop only a blocked daemon, not a slow pull.
_SBX_TIMEOUT_SECONDS = 600.0

# Only the CLI message for a missing sandbox is a not-found error. Messages
# such as "credentials not found" must stay transport errors. If not,
# delete_vm can identify a live sandbox as removed.
_SANDBOX_NOT_FOUND_RE = re.compile(r"sandbox '[^']*' not found")


class SbxCLI:
"""Thin async wrapper for the local ``sbx`` command-line interface.

Each method starts ``sbx`` with ``create_subprocess_exec``. Thus the
subprocess boundary stays in one place. The CLI selects the daemon: the
user socket by default, or the socket that ``DOCKER_SANDBOXES_API`` gives
when drukbox runs in a container.
"""

async def create_sandbox(
self,
*,
name: str,
template: str,
workspace: str,
cpus: int,
memory: str,
) -> None:
# The `shell` agent makes the sandbox start the template entrypoint,
# not an AI agent. The sizes are always explicit. Without them, the
# daemon gives one sandbox all host CPUs and half of the host memory.
await self._run(
"create",
"--name",
name,
"--template",
template,
"--cpus",
str(cpus),
"--memory",
memory,
"--quiet",
"shell",
workspace,
)

async def run_bootstrap(self, name: str, script: str) -> None:
# The script contains caller environment values. All processes can
# read argv through /proc. Thus the script goes through stdin, not
# argv. `bash -s` reads the program from stdin.
await self._run(
"exec",
"--interactive",
"--user",
"root",
name,
"bash",
"-s",
stdin=script,
)

async def publish_ssh_port(self, name: str, *, host_ip: str) -> int:
# An empty host port tells the daemon to select a free port. Thus
# sandboxes do not compete for port numbers. An IPv6 address must have
# brackets in the port specification.
spec_host = f"[{host_ip}]" if ":" in host_ip else host_ip
output = await self._run("ports", name, "--publish", f"{spec_host}::22")
# Each binding shows as "<host_ip>:<port> -> 22/tcp". The output can
# have one line for each address family. The lines share the host port.
for line in output.splitlines():
binding, arrow, target = line.partition("->")
if arrow and target.strip().startswith("22/"):
try:
return int(binding.strip().rsplit(":", 1)[1])
except (IndexError, ValueError) as error:
raise DockerSbxTransportError(
f"sandbox {name!r} published an unparsable SSH port: {line.strip()!r}"
) from error
raise DockerSbxTransportError(f"sandbox {name!r} published no SSH port")

async def remove_sandbox(self, name: str) -> None:
# The --force flag stops the confirmation prompt. It also removes a
# sandbox that has an open SSH session.
await self._run("rm", "--force", name)

async def sandbox_count(self) -> int:
output = await self._run("ls", "--json")
try:
payload = json.loads(output)
# Go writes an empty list as null. An unused daemon shows null.
return len(payload["sandboxes"] or [])
except (json.JSONDecodeError, KeyError, TypeError) as error:
raise DockerSbxTransportError(
f"sbx ls returned an unreadable sandbox list: {output.strip()!r}"
) from error

async def _run(self, *args: str, stdin: str | None = None) -> str:
try:
process = await asyncio.create_subprocess_exec(
"sbx",
*args,
stdin=asyncio.subprocess.PIPE if stdin else asyncio.subprocess.DEVNULL,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
except OSError as error:
# The binary can be missing (FileNotFoundError) or not executable
# (PermissionError). Translate each OSError type. A raw OSError
# must not go out of the provider boundary.
raise DockerSbxTransportError(f"sbx CLI could not be started: {error}") from error
try:
stdout, stderr = await asyncio.wait_for(
process.communicate(stdin.encode() if stdin else None),
timeout=_SBX_TIMEOUT_SECONDS,
)
except TimeoutError as error:
process.kill()
await process.wait()
raise DockerSbxTransportError(
f"sbx {args[0]} did not finish within {_SBX_TIMEOUT_SECONDS:.0f}s"
) from error
if process.returncode != 0:
detail = stderr.decode().strip() or f"sbx {args[0]} exited {process.returncode}"
if _SANDBOX_NOT_FOUND_RE.search(detail):
raise DockerSbxNotFoundError(detail)
raise DockerSbxTransportError(detail)
return stdout.decode()
10 changes: 10 additions & 0 deletions src/providers/docker_sbx/exceptions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
class DockerSbxProviderError(RuntimeError):
"""Base error for the Docker Sandboxes provider."""


class DockerSbxNotFoundError(DockerSbxProviderError):
"""The sandbox was not found."""


class DockerSbxTransportError(DockerSbxProviderError):
"""The sbx command failed because of a transport problem."""
Loading