Failed activation leaves a spinning `switch-to-configuration` holding the lock, poisoning every subsequent deploy

[Laski]

Today while trying to update from 25.11 to 26.05, both the activation and the rollback failed multiple times, leaving the VM in an inconsistent state. I asked an LLM model (Fable 5, low effort) to help me diagnose and fix, and it may have found a deadlock between deploy-rs and nixos. This is the report I asked it to write.

I'm far from a Rust and NixOS expert but I'd be happy to ask the same LLM to fork this repo and try to fix the issue if someone confirms it and tells me that this project accept LLM-generated code (with human review of course).

------- Start of LLM-generated content -------

Summary

When an activation restarts sshd/dbus-broker/systemd-logind (e.g. a NixOS release
upgrade that bumps systemd), the SSH channel deploy-rs is running the activation over dies
mid-switch. Neither the activation nor the magic rollback completes. What makes this more
than a transient failure is the wreckage it leaves behind:

• an orphaned switch-to-configuration switch process spinning at 99.9% CPU, and
• /run/nixos/switch-to-configuration.lock still held by it.

Every later deploy then fails with Could not acquire lock, and leaves another spinning
process of its own. The failure is self-amplifying: each attempt to recover makes it worse.
On our host this turned one interrupted deploy into 4 days of a degraded system with a
core pinned at 100%, undetected until we went looking.

The magic rollback is supposed to be the safety net for exactly this scenario ("connection
lost during activation"), but it cannot run — it needs the same lock the wedged process holds:

⭐ ℹ️ [activate] [INFO] Attempting to re-activate the last generation
Could not acquire lock
⭐ ❌ [activate] [ERROR] There was an error de-activating after an error was encountered:
Nix reactivate last generation command resulted in a bad exit status: exit status: 11

Environment

• deploy-rs 0.1.0
• NixOS 25.11 → 26.05 upgrade (systemd 258 → 260), aarch64-linux, Oracle Cloud VM
• Invocation: deploy .#host --skip-checks -- --max-jobs 0 --no-fallback
  (also reproduced with --magic-rollback false)

Steps to reproduce

1. Deploy a closure whose activation restarts sshd.service, dbus-broker.service and
   systemd-logind.service — a release upgrade that bumps systemd does this reliably.
2. Watch the activation get as far as restarting those units, then die:

restarting the following units: nix-daemon.service, sshd.service, systemd-journald.service,
systemd-networkd.service, systemd-resolved.service, systemd-udevd.service, tailscaled.service
Failed to reload dbus-broker.service
Read from remote host: Connection reset by peer
client_loop: send disconnect: Broken pipe
🚀 ❌ [deploy] [ERROR] Activating over SSH resulted in a bad exit code: Some(255)

3. On the target, the orphan is still there and still burning CPU:

$ ps -o pid,etime,stat,pcpu,cmd -C switch-to-configuration
    PID     ELAPSED STAT %CPU CMD
   6451  09:56:54   R    99.9 /nix/var/nix/profiles/system/bin/switch-to-configuration switch

4. Run deploy again → Could not acquire lock, and a second spinning process appears.

Expected

A failed or interrupted activation should not leave the target in a state where recovery is
impossible without manual intervention. Concretely:

• the activation process should be cleaned up (or reparented and supervised) when the SSH
  channel dies, rather than orphaned in a busy loop;
• Could not acquire lock should be reported as an actionable error naming the PID holding
  /run/nixos/switch-to-configuration.lock, not as an opaque failure of the rollback path.

Suggested fix

Run the activation detached from the SSH session, as nixos-rebuild --target-host does,
so that restarting sshd/logind cannot kill it:

systemd-run --unit=deploy-rs-activate --collect --service-type=oneshot \
  <closure>/bin/switch-to-configuration switch

deploy-rs can then poll the unit's result over a fresh connection instead of depending on the
one channel the activation is actively tearing down. This is what unblocked our host after
several failed deploy runs — the activation completed on the first try, and
switch-to-configuration blocked politely in do_sys_poll at 0% CPU instead of spinning.

Note on --magic-rollback false

Disabling magic rollback does not avoid this. The orphaned process and the held lock come from
the activation dying, not from the confirmation waiter, so --magic-rollback false fails the
same way — it just removes the (already non-functional) rollback attempt from the output.

Possibly related

• Password-based ssh authentication does not work #107 — hangs at "Waiting for confirmation event", but attributed to password-prompt
  handling rather than the activation tearing down its own transport.
• Deploy-rs hangs up and doesn't do anything #294 — hang without output, during the build phase rather than activation.

Neither mentions the orphaned spinning process or the lock it holds, which is the part that
makes this unrecoverable without logging in and killing PIDs by hand.