|
| 1 | +<!-- Content type: How-to --> |
| 2 | + |
| 3 | +# Plan: diagnose and fix network-blip session loss |
| 4 | + |
| 5 | +**Baseline:** branch `dev`, commit `76eaf68`. Understudy version |
| 6 | +`e0673967-0ca5-4cca-81ee-50d4088cec33`. Extension `0.1.1` from |
| 7 | +`5cc969024b32380f1a3e14ff6f72e8d0c808b086`. Canary device |
| 8 | +`aaf119f2-a85f-46b4-8b53-8e92196d6275`, credential version 2. |
| 9 | + |
| 10 | +Line numbers are as-of that commit and must be re-confirmed before editing. |
| 11 | + |
| 12 | +## What happened, and what is actually established |
| 13 | + |
| 14 | +Phase 3b soak run 1 died 40 minutes in. `docs/unattended-production-rollout.md` |
| 15 | +holds the full record; the essentials: |
| 16 | + |
| 17 | +| Fact | How it is known | |
| 18 | +|---|---| |
| 19 | +| Tailscale activated its interface and **reconfigured DNS** at `06:02:16Z` | `journalctl`, inside the failure window | |
| 20 | +| A second long-lived socket (1Password's notifier) dropped at `06:03:00Z` and recovered at `06:03:06Z` | same journal — **six seconds** | |
| 21 | +| Understudy's device was still `offline` at `06:09:49Z`, lease already `lost` | soak sample | |
| 22 | +| The machine never slept and Chrome never restarted | four-week uptime, zero suspend records, Chrome process older than the soak | |
| 23 | + |
| 24 | +**The problem is not the outage. It is the recovery.** A six-second |
| 25 | +interruption that another application shrugged off cost this device every |
| 26 | +session it held, terminally, plus a leaked window. |
| 27 | + |
| 28 | +## What is NOT established |
| 29 | + |
| 30 | +Be careful here — two plausible mechanisms have already been written down and |
| 31 | +one has already been wrongly declared dead. |
| 32 | + |
| 33 | +1. **Timer death.** Both reconnect paths use `setTimeout` |
| 34 | + (`apps/extension/src/core/ws-client.ts:134`, |
| 35 | + `apps/extension/src/core/profile-client.ts:716`), which does not survive |
| 36 | + service-worker eviction. But the `ws-backstop` alarm |
| 37 | + (`apps/extension/src/entrypoints/background.ts:99`) fires every 30 seconds |
| 38 | + and calls `ensureConnection`, which *should* bound recovery near 30 seconds. |
| 39 | + It did not. So either the alarm was delayed, or attempts were firing and |
| 40 | + failing. |
| 41 | +2. **DNS.** The trigger was a DNS reconfiguration, not a plain link drop. If |
| 42 | + resolution for `understudy.proofof.tech` stayed broken or poisoned inside |
| 43 | + Chrome for minutes, reconnect attempts would fail no matter how promptly they |
| 44 | + were scheduled. 1Password may well have been holding an established |
| 45 | + connection or a cached address, which would explain its six-second recovery |
| 46 | + without contradicting this. |
| 47 | +3. **Backgrounded Chrome / MV3 eviction.** Still open. An 18-minute attempt to |
| 48 | + test it was **invalid** — the operator was switching windows throughout, so |
| 49 | + the condition never held. Do not cite that run as evidence in either |
| 50 | + direction. |
| 51 | + |
| 52 | +The tests below are designed to separate these, not to confirm a favourite. |
| 53 | + |
| 54 | +## Sequencing |
| 55 | + |
| 56 | +**These tests destroy any running soak.** Breaking connectivity is exactly the |
| 57 | +condition that kills a lease. Either let the current soak finish and record its |
| 58 | +result first, or accept restarting it — do not run both and then argue about |
| 59 | +which caused what. Phase 3b's clock restarts either way. |
| 60 | + |
| 61 | +## Instrumentation |
| 62 | + |
| 63 | +The monitor cannot live on the machine whose network is being broken. It does |
| 64 | +not need to: **the recovery latency is measurable from the server side alone.** |
| 65 | + |
| 66 | +- `t_break` and `t_restore` are chosen by the operator, so both are known exactly. |
| 67 | +- `GET /v1/devices` returns `lastSeenAt`. Polls during the outage fail; that is |
| 68 | + fine. The first poll after `t_restore` whose `lastSeenAt` is fresh (< 30 s old) |
| 69 | + gives `t_device_back`. |
| 70 | +- **`recovery_latency = t_device_back − t_restore`** is the number this plan |
| 71 | + exists to produce. Run 1 implies minutes. Anything above a few seconds is the |
| 72 | + defect. |
| 73 | + |
| 74 | +Capture at the same time, because they discriminate between mechanisms and only |
| 75 | +an operator at the machine can see them: |
| 76 | + |
| 77 | +- `chrome://extensions` — does the card read `service worker` or |
| 78 | + `service worker (Inactive)` during the outage? Answers eviction directly. |
| 79 | +- The service-worker console — are reconnect attempts logged during the outage, |
| 80 | + and do they fail with a DNS error or a connection error? Answers 1 versus 2. |
| 81 | +- The side panel's local event log, which survives longer than the console. |
| 82 | +- Tab and window count afterwards, for the leak. |
| 83 | + |
| 84 | +## Test A — pure connectivity loss, DNS untouched |
| 85 | + |
| 86 | +Isolates "does reconnection work at all" from anything DNS-specific. Block |
| 87 | +egress to the service without disturbing the resolver: |
| 88 | + |
| 89 | +```bash |
| 90 | +# resolve first so the rule pins addresses, not names |
| 91 | +getent ahosts understudy.proofof.tech |
| 92 | +sudo iptables -I OUTPUT -d <each-resolved-ip> -j REJECT |
| 93 | +# ... wait the test duration ... |
| 94 | +sudo iptables -D OUTPUT -d <each-resolved-ip> -j REJECT |
| 95 | +``` |
| 96 | + |
| 97 | +Run at **10 s, 30 s, 60 s, 120 s**, with a live session each time, recording |
| 98 | +`recovery_latency` and whether the lease survived. |
| 99 | + |
| 100 | +Expected if healthy: recovery within a few seconds of restore at every |
| 101 | +duration; the lease survives 10 s and 30 s, and is `lost` at 120 s by policy |
| 102 | +(`DEVICE_LOST_MS = 90_000`). |
| 103 | + |
| 104 | +The interesting failure is a **short** outage producing a **long** recovery. |
| 105 | +That is run 1's signature and points at mechanism 1. |
| 106 | + |
| 107 | +## Test B — replicate the actual trigger |
| 108 | + |
| 109 | +```bash |
| 110 | +sudo tailscale down # observe |
| 111 | +sudo tailscale up # observe |
| 112 | +``` |
| 113 | + |
| 114 | +Same measurements. This reconfigures DNS, which Test A deliberately does not. |
| 115 | + |
| 116 | +- Test A clean, Test B slow → the problem is DNS-specific, and the fix is about |
| 117 | + resolution and connection re-establishment, not timers. |
| 118 | +- Both slow → reconnection is broken generally; timers or alarm delivery. |
| 119 | +- Both clean at every duration → the mechanism is not reproducible this way, and |
| 120 | + the next suspect is sustained backgrounding, tested properly this time: machine |
| 121 | + genuinely untouched, Chrome fully occluded, sampled from elsewhere. |
| 122 | + |
| 123 | +## Fixes, chosen after the tests and not before |
| 124 | + |
| 125 | +Deliberately not pre-committed — the wrong fix here is easy to justify and |
| 126 | +useless, and two mechanisms are still live. |
| 127 | + |
| 128 | +**If timer death (1):** drive retries from `chrome.alarms` rather than |
| 129 | +`setTimeout`, keeping `setTimeout` only for sub-30-second attempts while the |
| 130 | +worker is known alive, and treat the backstop alarm as the authority on whether |
| 131 | +a connection should exist — which it nearly is already. |
| 132 | + |
| 133 | +**If DNS (2):** the socket must be re-established in a way that re-resolves |
| 134 | +rather than reusing a poisoned or cached answer, and repeated resolution failure |
| 135 | +must not push the backoff to its 30-second cap and stay there. |
| 136 | + |
| 137 | +**Independent of both, and worth doing regardless** — these are already recorded |
| 138 | +in `DEFERRED.md` and are what convert a blip into destroyed work: |
| 139 | + |
| 140 | +- **Ninety seconds destroys every session on the device.** Separate capacity |
| 141 | + reclamation from lease destruction: free the slot promptly, but keep the lease |
| 142 | + adoptable for a longer window and destroy only past that. |
| 143 | +- **A lost lease leaks a browser window.** Send an orphan list on device sync so |
| 144 | + the extension closes tabs for leases the server has already released. |
| 145 | + |
| 146 | +Fixing recovery lowers how often the cliff is reached. It does not make the |
| 147 | +cliff the right shape. |
| 148 | + |
| 149 | +## Verification |
| 150 | + |
| 151 | +- Test A at 10 s and 30 s: lease survives, `recovery_latency` within a few |
| 152 | + seconds, no leaked window. |
| 153 | +- Test A at 120 s: lease is `lost` **by policy**, and — once the leak fix lands — |
| 154 | + no window survives. |
| 155 | +- Test B: indistinguishable from Test A at the same durations. |
| 156 | +- A soak that spans at least one real network transition without losing its |
| 157 | + session. |
| 158 | +- Quality gate on any code change: clean-code, architecture, and QA lanes. |
| 159 | + |
| 160 | +## Out of scope |
| 161 | + |
| 162 | +- `DEVICE_OFFLINE_MS`. Reporting a device offline at 75 seconds is accurate and |
| 163 | + destroys nothing. |
| 164 | +- The heartbeat interval. 22 seconds is not implicated; the heartbeat was fine |
| 165 | + until the socket went away. |
| 166 | +- Metamind. Its sweep already handles a terminal understudy session correctly — |
| 167 | + it settles the lease rather than retrying forever. Nothing here needs a |
| 168 | + consumer change. |
0 commit comments