Skip to content

Commit fc7ea96

Browse files
committed
docs: record the rollout's defects, and plan the network-blip work
DEFERRED.md gains five entries alongside the attended-status one, each with the code that proves it and a proposed fix with the rejected alternative: - a lost lease leaks a browser window, because the closure list a device syncs excludes leases with release_at set — which is exactly the set just released - reconnect backoff uses setTimeout on both paths, which does not survive service-worker eviction; recorded as SUSPECTED, since the 30s backstop alarm should still have bounded recovery and did not - ninety seconds without a heartbeat destroys every session on a device, a tolerance thinner than the recovery path it has to accommodate - a leaked window is invisible to API monitoring, so the soak's no-capacity-leak gate measures less than its name claims - Chrome's debugger banner spans profiles, so an operator can detach the canary from a window they have no reason to connect to it The plan separates the two live mechanisms rather than assuming either. Test A blocks egress by IP with DNS untouched; Test B toggles tailscale and reconfigures DNS as the real incident did. Recovery latency is measured server-side from lastSeenAt, so the monitor does not need to survive the outage it is measuring. The signature to look for is a SHORT outage producing a LONG recovery — six seconds cost seven minutes on 2026-07-29. Fixes are deliberately not pre-committed. The backgrounding hypothesis is carried as open, with an explicit note that the run which appeared to kill it was invalid.
1 parent 76eaf68 commit fc7ea96

2 files changed

Lines changed: 304 additions & 0 deletions

File tree

‎DEFERRED.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,3 +173,139 @@ alone changes nothing.
173173
state.
174174
- Quality gate: clean-code, architecture, and QA lanes, since this is a protocol
175175
change with cross-repo consumers.
176+
177+
---
178+
179+
## A lost lease leaks a browser window
180+
181+
**Baseline:** branch `dev`, commit `76eaf68`.
182+
**Found:** 2026-07-29, Phase 3b soak run 1.
183+
**Severity:** every device-loss incident strands a Chrome window that only a human will close. Compounds without bound.
184+
185+
### What was observed
186+
187+
The soak's device missed its heartbeat past `DEVICE_LOST_MS`, so the coordinator marked its lease `lost`. Hours later the server reported `used 1/2` and the side panel reported `1/2` — both correct — while **two** `example.com` windows were open on screen. Tab `2134210655` belonged to the destroyed lease and was never closed.
188+
189+
### Why it happens
190+
191+
Declaring a device lost sets `status = 'lost'` **and `release_at`** on its leases. But the closure list a device receives on sync, in `apps/backend/src/tenant-coordinator.ts:300`, is:
192+
193+
```sql
194+
SELECT * FROM lease
195+
WHERE device_id = ? AND status IN ('closing','expired') AND release_at IS NULL
196+
ORDER BY created_at
197+
```
198+
199+
`release_at IS NULL` excludes exactly the leases that were just released. The server frees the slot and considers itself done; the extension is never told to close the tab, so it doesn't.
200+
201+
### Proposed fix
202+
203+
Close the loop on reconnect rather than widening the query blindly — a `lost` lease has `release_at` set precisely because the server has finished with it, and other logic depends on that.
204+
205+
**Recommended:** on device sync, send a separate *orphan* list — leases with `release_at` set whose device is now back — instructing the extension to close those tabs and forget the assignments. It is advisory, needs no acknowledgement, and is safe to repeat.
206+
207+
*Rejected:* dropping `release_at IS NULL` from the closure query. Closures are part of the release handshake; feeding already-released leases into it would have the extension re-acknowledge closures the server has settled.
208+
209+
**Also acceptable and complementary:** have the extension, on reconnect, close any controlled tab whose assignment the server does not acknowledge. That defends against the general case rather than this one path.
210+
211+
### Verification
212+
213+
- Coordinator test: a device that reconnects after its leases were lost receives the orphan list.
214+
- Extension test: receiving it closes exactly those tabs and leaves untracked user tabs alone.
215+
- Manually: force a device loss, reconnect, confirm no window survives and the count matches the side panel.
216+
217+
---
218+
219+
## Reconnect backoff does not survive service-worker eviction
220+
221+
**Baseline:** branch `dev`, commit `76eaf68`.
222+
**Found:** 2026-07-29, inferred from soak run 1 timings. **Mechanism not yet proven** — see the test plan in `docs/plan-network-blip-resilience.md`.
223+
**Severity:** suspected cause of a six-second network blip costing every session on a device.
224+
225+
### Evidence
226+
227+
Both reconnect paths schedule with `setTimeout`:
228+
229+
- `apps/extension/src/core/ws-client.ts:134` — `this.reconnectTimer = setTimeout(...)`, backoff 500 ms doubling to a 30 s cap (`:14`, `:15`).
230+
- `apps/extension/src/core/profile-client.ts:716` — `this.retryTimer = setTimeout(...)`, same shape (`:28`, `:29`). This is the **device control socket**, the one carrying the heartbeat.
231+
232+
An MV3 service worker is evicted when idle, and a `setTimeout` dies with it. Losing the socket removes the very activity that was keeping the worker alive, so the pending retry is discarded at the moment it is most needed. Recovery then depends on the 30-second `ws-backstop` alarm (`apps/extension/src/entrypoints/background.ts:99`).
233+
234+
That should still bound recovery near 30 seconds, which is why this is **suspected rather than established**: run 1's device stayed offline for minutes, not tens of seconds. Either the alarm was delayed, or reconnect attempts were firing and failing for another reason — DNS, in the incident that produced this. The test plan is designed to tell those apart.
235+
236+
### Proposed fix (contingent on the test)
237+
238+
Do not fix before the mechanism is confirmed; the wrong fix here is easy to justify and useless. If reconnection proves to be timer-death:
239+
240+
- drive retries from `chrome.alarms` rather than `setTimeout`, accepting the 30-second floor, and keep `setTimeout` only for sub-30-second attempts while the worker is known alive;
241+
- treat the backstop alarm as the authority for "should I be connected", which it nearly is already.
242+
243+
### Verification
244+
245+
Reproduce per the plan, capture whether the worker was evicted, whether the alarm fired, and whether reconnect attempts occurred at all. Only then choose the fix.
246+
247+
---
248+
249+
## Ninety seconds without a heartbeat destroys every session on a device
250+
251+
**Baseline:** branch `dev`, commit `76eaf68`.
252+
**Found:** 2026-07-29, Phase 3b soak run 1.
253+
**Severity:** design question for unattended operation, not a coding error.
254+
255+
### The numbers
256+
257+
`apps/backend/src/tenant-coordinator.ts:10-11`:
258+
259+
```ts
260+
const DEVICE_OFFLINE_MS = 75_000; // reported offline
261+
const DEVICE_LOST_MS = 90_000; // every lease on the device -> 'lost'
262+
```
263+
264+
Ninety seconds is roughly four missed heartbeats (`HEARTBEAT_MS = 22_000`). Crossing it sets `status = 'lost'` and `release_at` on every lease the device holds — terminal, `410` to the consumer, work discarded, and a leaked window per the entry above.
265+
266+
### Why it is a problem
267+
268+
The tolerance is thinner than the recovery path it must accommodate. Recovery is gated on a 30-second alarm, leaving at most three alarm cycles of slack, and only if every one succeeds. Meanwhile the events that cause a gap this size are utterly routine: a VPN connecting, a Wi-Fi roam, a DNS change. One was enough on 2026-07-29 — and 1Password's socket, hit by the same event, was back in six seconds.
269+
270+
For an attended session an operator notices and retries. An unattended fleet silently discards in-flight work.
271+
272+
### Proposed fix
273+
274+
**Recommended:** separate *capacity reclamation* from *lease destruction*. Freeing a device slot quickly is legitimate; destroying the consumer's session is a different decision that does not need the same deadline. Add a suspended state where the lease stops counting against capacity but remains adoptable if the device returns within a longer window, and destroy only past that.
275+
276+
*Rejected:* simply raising `DEVICE_LOST_MS`. It trades one arbitrary number for another and delays capacity reclamation for every genuine loss to buy tolerance for transient ones.
277+
278+
Consider alongside the reconnect entry above — a fix there reduces how often this threshold is reached, but does not make the threshold right.
279+
280+
### Out of scope
281+
282+
`DEVICE_OFFLINE_MS`. Reporting a device offline after 75 seconds is accurate and harmless; it destroys nothing.
283+
284+
---
285+
286+
## A leaked browser window is invisible to API monitoring
287+
288+
**Baseline:** branch `dev`, commit `76eaf68`.
289+
**Found:** 2026-07-29 — the window leak above surfaced only because an operator looked at their own screen.
290+
**Severity:** an acceptance gate that measures less than its name claims.
291+
292+
The Phase 3b soak checks `capacity_leak` as `deviceUsed > 1`, which is server-side accounting. In the observed leak that accounting was **correct** — the server had released the slot. The leak was entirely client-side, and no API-driven check can see it: nothing exposes how many tabs or windows the extension actually holds versus how many the server believes it holds.
293+
294+
Any soak or monitor built on the public API therefore cannot enforce "no capacity leak" as written; it can only enforce "no server-side capacity leak".
295+
296+
**Proposed fix:** have the device report its observed controlled-tab count in the heartbeat, and have the coordinator flag a divergence from its own lease count. That makes the leak detectable remotely and turns the gate into what it claims to be. Until then, treat the gate as partial and confirm tab counts visually during acceptance.
297+
298+
---
299+
300+
## Chrome's debugger banner appears across profiles
301+
302+
**Baseline:** branch `dev`, commit `76eaf68`. **Chrome behaviour, not a defect in this repo.**
303+
**Severity:** an operator can detach the canary's debugger by accident, from a window they have no reason to associate with it.
304+
305+
Attaching `chrome.debugger` shows the "controlled by automated test software" banner on windows belonging to **other Chrome profiles**, not only the controlled one. Confirmed by the canary operator as long-standing behaviour on this machine.
306+
307+
`apps/extension/RUNBOOK.md` instructs operators not to click that banner's detach control, but the instruction reads as being about the controlled window. Someone seeing the banner over an unrelated profile may reasonably dismiss it there — detaching the canary and, per the reconnect entries above, potentially destroying its leases.
308+
309+
It also means the banner is **not** a per-tab indicator of whether a given tab is under CDP control, so it cannot be used as a diagnostic. That mistake was made once already during this rollout.
310+
311+
**Proposed fix:** documentation only. State in the extension runbook that the banner is process-wide, that dismissing it anywhere detaches the canary, and that it must not be read as evidence about a particular tab.
Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
<!-- Content type: How-to -->
2+
3+
# Plan: diagnose and fix network-blip session loss
4+
5+
**Baseline:** branch `dev`, commit `76eaf68`. Understudy version
6+
`e0673967-0ca5-4cca-81ee-50d4088cec33`. Extension `0.1.1` from
7+
`5cc969024b32380f1a3e14ff6f72e8d0c808b086`. Canary device
8+
`aaf119f2-a85f-46b4-8b53-8e92196d6275`, credential version 2.
9+
10+
Line numbers are as-of that commit and must be re-confirmed before editing.
11+
12+
## What happened, and what is actually established
13+
14+
Phase 3b soak run 1 died 40 minutes in. `docs/unattended-production-rollout.md`
15+
holds the full record; the essentials:
16+
17+
| Fact | How it is known |
18+
|---|---|
19+
| Tailscale activated its interface and **reconfigured DNS** at `06:02:16Z` | `journalctl`, inside the failure window |
20+
| A second long-lived socket (1Password's notifier) dropped at `06:03:00Z` and recovered at `06:03:06Z` | same journal — **six seconds** |
21+
| Understudy's device was still `offline` at `06:09:49Z`, lease already `lost` | soak sample |
22+
| The machine never slept and Chrome never restarted | four-week uptime, zero suspend records, Chrome process older than the soak |
23+
24+
**The problem is not the outage. It is the recovery.** A six-second
25+
interruption that another application shrugged off cost this device every
26+
session it held, terminally, plus a leaked window.
27+
28+
## What is NOT established
29+
30+
Be careful here — two plausible mechanisms have already been written down and
31+
one has already been wrongly declared dead.
32+
33+
1. **Timer death.** Both reconnect paths use `setTimeout`
34+
(`apps/extension/src/core/ws-client.ts:134`,
35+
`apps/extension/src/core/profile-client.ts:716`), which does not survive
36+
service-worker eviction. But the `ws-backstop` alarm
37+
(`apps/extension/src/entrypoints/background.ts:99`) fires every 30 seconds
38+
and calls `ensureConnection`, which *should* bound recovery near 30 seconds.
39+
It did not. So either the alarm was delayed, or attempts were firing and
40+
failing.
41+
2. **DNS.** The trigger was a DNS reconfiguration, not a plain link drop. If
42+
resolution for `understudy.proofof.tech` stayed broken or poisoned inside
43+
Chrome for minutes, reconnect attempts would fail no matter how promptly they
44+
were scheduled. 1Password may well have been holding an established
45+
connection or a cached address, which would explain its six-second recovery
46+
without contradicting this.
47+
3. **Backgrounded Chrome / MV3 eviction.** Still open. An 18-minute attempt to
48+
test it was **invalid** — the operator was switching windows throughout, so
49+
the condition never held. Do not cite that run as evidence in either
50+
direction.
51+
52+
The tests below are designed to separate these, not to confirm a favourite.
53+
54+
## Sequencing
55+
56+
**These tests destroy any running soak.** Breaking connectivity is exactly the
57+
condition that kills a lease. Either let the current soak finish and record its
58+
result first, or accept restarting it — do not run both and then argue about
59+
which caused what. Phase 3b's clock restarts either way.
60+
61+
## Instrumentation
62+
63+
The monitor cannot live on the machine whose network is being broken. It does
64+
not need to: **the recovery latency is measurable from the server side alone.**
65+
66+
- `t_break` and `t_restore` are chosen by the operator, so both are known exactly.
67+
- `GET /v1/devices` returns `lastSeenAt`. Polls during the outage fail; that is
68+
fine. The first poll after `t_restore` whose `lastSeenAt` is fresh (< 30 s old)
69+
gives `t_device_back`.
70+
- **`recovery_latency = t_device_back − t_restore`** is the number this plan
71+
exists to produce. Run 1 implies minutes. Anything above a few seconds is the
72+
defect.
73+
74+
Capture at the same time, because they discriminate between mechanisms and only
75+
an operator at the machine can see them:
76+
77+
- `chrome://extensions` — does the card read `service worker` or
78+
`service worker (Inactive)` during the outage? Answers eviction directly.
79+
- The service-worker console — are reconnect attempts logged during the outage,
80+
and do they fail with a DNS error or a connection error? Answers 1 versus 2.
81+
- The side panel's local event log, which survives longer than the console.
82+
- Tab and window count afterwards, for the leak.
83+
84+
## Test A — pure connectivity loss, DNS untouched
85+
86+
Isolates "does reconnection work at all" from anything DNS-specific. Block
87+
egress to the service without disturbing the resolver:
88+
89+
```bash
90+
# resolve first so the rule pins addresses, not names
91+
getent ahosts understudy.proofof.tech
92+
sudo iptables -I OUTPUT -d <each-resolved-ip> -j REJECT
93+
# ... wait the test duration ...
94+
sudo iptables -D OUTPUT -d <each-resolved-ip> -j REJECT
95+
```
96+
97+
Run at **10 s, 30 s, 60 s, 120 s**, with a live session each time, recording
98+
`recovery_latency` and whether the lease survived.
99+
100+
Expected if healthy: recovery within a few seconds of restore at every
101+
duration; the lease survives 10 s and 30 s, and is `lost` at 120 s by policy
102+
(`DEVICE_LOST_MS = 90_000`).
103+
104+
The interesting failure is a **short** outage producing a **long** recovery.
105+
That is run 1's signature and points at mechanism 1.
106+
107+
## Test B — replicate the actual trigger
108+
109+
```bash
110+
sudo tailscale down # observe
111+
sudo tailscale up # observe
112+
```
113+
114+
Same measurements. This reconfigures DNS, which Test A deliberately does not.
115+
116+
- Test A clean, Test B slow → the problem is DNS-specific, and the fix is about
117+
resolution and connection re-establishment, not timers.
118+
- Both slow → reconnection is broken generally; timers or alarm delivery.
119+
- Both clean at every duration → the mechanism is not reproducible this way, and
120+
the next suspect is sustained backgrounding, tested properly this time: machine
121+
genuinely untouched, Chrome fully occluded, sampled from elsewhere.
122+
123+
## Fixes, chosen after the tests and not before
124+
125+
Deliberately not pre-committed — the wrong fix here is easy to justify and
126+
useless, and two mechanisms are still live.
127+
128+
**If timer death (1):** drive retries from `chrome.alarms` rather than
129+
`setTimeout`, keeping `setTimeout` only for sub-30-second attempts while the
130+
worker is known alive, and treat the backstop alarm as the authority on whether
131+
a connection should exist — which it nearly is already.
132+
133+
**If DNS (2):** the socket must be re-established in a way that re-resolves
134+
rather than reusing a poisoned or cached answer, and repeated resolution failure
135+
must not push the backoff to its 30-second cap and stay there.
136+
137+
**Independent of both, and worth doing regardless** — these are already recorded
138+
in `DEFERRED.md` and are what convert a blip into destroyed work:
139+
140+
- **Ninety seconds destroys every session on the device.** Separate capacity
141+
reclamation from lease destruction: free the slot promptly, but keep the lease
142+
adoptable for a longer window and destroy only past that.
143+
- **A lost lease leaks a browser window.** Send an orphan list on device sync so
144+
the extension closes tabs for leases the server has already released.
145+
146+
Fixing recovery lowers how often the cliff is reached. It does not make the
147+
cliff the right shape.
148+
149+
## Verification
150+
151+
- Test A at 10 s and 30 s: lease survives, `recovery_latency` within a few
152+
seconds, no leaked window.
153+
- Test A at 120 s: lease is `lost` **by policy**, and — once the leak fix lands —
154+
no window survives.
155+
- Test B: indistinguishable from Test A at the same durations.
156+
- A soak that spans at least one real network transition without losing its
157+
session.
158+
- Quality gate on any code change: clean-code, architecture, and QA lanes.
159+
160+
## Out of scope
161+
162+
- `DEVICE_OFFLINE_MS`. Reporting a device offline at 75 seconds is accurate and
163+
destroys nothing.
164+
- The heartbeat interval. 22 seconds is not implicated; the heartbeat was fine
165+
until the socket went away.
166+
- Metamind. Its sweep already handles a terminal understudy session correctly —
167+
it settles the lease rather than retrying forever. Nothing here needs a
168+
consumer change.

0 commit comments

Comments
 (0)