[sashyo]

Devops-GW stopped serving today and needed a manual container restart, which produced TCP connection timeout in the browser.

What happened

04:23:20  SSH session through Devops-GW to TideStun:22 — worked
04:23:31  WebRTC peer closed        ← last log line from the gateway
04:23:32  signal server restarted
          … 40 minutes of complete silence …

The container was Running the whole time with 0 restarts. It simply stopped participating, and the signal server routed to its stale address.

The retry loop logs Connecting to … and Reconnecting (delay: Nms) on every attempt. Forty minutes produced none of those lines, so the loop was never re-entered — it was stuck inside connect_and_run.

Cause

The main loop awaited ws_stream.next() with no deadline. If the peer disappears without a clean close reaching us — a half-open socket — that future never resolves. The loop never returns Disconnected, so the reconnect logic below it is unreachable.

There is a ping task, but when its send fails it only breaks itself. It has no channel back to the main loop, so its death is silent.

The consequence is worse than one gateway: any signal server restart can leave gateways permanently hung, each needing a manual restart. Both of today's incidents fit that shape.

Fix

A liveness deadline in the select loop. Any frame — pings and pongs included — marks the link alive; 45s of total silence (three ping intervals, impossible on a healthy link since both ends ping every 15s) ends the connection so the existing exponential backoff takes over.

Two things that made it worse, also fixed

The signal server had no supervisor. deploy.sh launched it with nohup, so it dies on every reboot and stays dead — which is what took it down earlier today, and it surfaces in the console as "Contact your administrator to get server access" rather than as an outage. Now installs a systemd unit with Restart=always and StartLimitIntervalSec=0, falling back to nohup where systemd isn't available. It also kills any leftover nohup instance before starting, so the port isn't already taken.

Access tokens were being written to the logs. From the live log:

[SSH-Proxy] Connecting to gateway: ws://4.198.216.16:7891/ws/ssh?token=eyJhbGciOiJFZERTQSI…

A complete, decodable access token in a log file on disk, carrying the user's roles, email and vuid — once per SSH session. DPoP-binding limits replay but this should not be there at all. Credential-bearing query parameters are now redacted on both the signal server and the gateway, keeping host, port and serverId so the line stays useful.

Testing

55 gateway tests and 34 signal server tests pass, 9 new on redaction (token stripped, every secret parameter, case-insensitive, clean URLs untouched, malformed pairs). deploy.sh passes bash -n.

The hang fix itself is not covered by a test — reproducing it needs a half-open socket, which is awkward to fake in a unit test. The reasoning is solid and the change is small, but the real confirmation is a deliberate signal server restart with a gateway attached: it should log no traffic for 45s, then Reconnecting, then re-register on its own. Worth doing on Staging-GW before trusting it everywhere.

The systemd unit has not been run — it needs systemctl on the actual host. Worth watching the first deploy.sh closely, particularly that EnvironmentFile parses the existing .env (it is KEY=value throughout, which systemd accepts, but values containing spaces or quotes would need review).