Write requests are silent for their whole duration, so Fly's 120s idle timeout closes any write that uses the node's own transfer and git budgets

[beardthelion]

All three Fly apps set idle_timeout = 120 (infra/fly/fly.toml:39, infra/fly/gitlawb-node-2.fly.toml:43, infra/fly/gitlawb-node-3.fly.toml:43). Fly's idle timeout is an activity timer, not a request-duration cap: any byte in either direction resets it, and a request whose body was fully received but which produces no response bytes is closed once the timer elapses.

The repo write path produces nothing until it has finished everything. git_receive_pack takes body: Bytes, so axum buffers the request before the handler runs, and the only Response the function yields is the final Ok(result) at crates/gitlawb-node/src/api/repos.rs:1383. smart_http::receive_pack builds that response with Body::from(output) over a fully materialized buffer from wait_with_output() (crates/gitlawb-node/src/git/smart_http.rs:74-87). Nothing streams, including git's own progress output, which is collected rather than forwarded.

So the silent span covers the entire operation:

• up to 90s waiting for the advisory lock (LOCK_ACQUIRE_DEADLINE, crates/gitlawb-node/src/git/repo_store.rs:907)
• up to GITLAWB_LOCK_HELD_TRANSFER_TIMEOUT_SECS (default 300) for the acquire-side object-storage refresh
• up to GITLAWB_GIT_SERVICE_TIMEOUT_SECS (default 600) for git itself
• up to another 300s for the release-side upload, which runs before the handler returns

Any write quiet for more than 120 seconds has its client connection closed by the edge. The practical consequence is that the node's own budgets are unreachable in production: a transfer permitted 300s can never use more than 120, and the 600s git budget cannot be spent at all. An operator raising either knob gets no additional headroom, which makes both settings misleading about what the deployment will actually allow.

This is the reconciliation that crates/gitlawb-node/src/git/repo_store.rs:770-780 refers to when it says the 90s lock wait should not be read as a promise to return inside the 120s proxy timeout and that reconciling the two is "tracked separately". Nothing was tracking it; this issue is that tracking.

What this is not

Worth stating explicitly, because each of these was checked and ruled out:

• Not a lock leak. When the edge drops the connection, the handler future is dropped and RepoWriteGuard::Drop closes the lock-owning session, which frees the advisory lock and its pool slot (repo_store.rs:855-893). LockProbe::Drop covers a disconnect during the acquire spin (repo_store.rs:712-731).
• Not a retry storm. Nothing retries a dropped push. git-remote-gitlawb issues a single .send() for the receive-pack POST (crates/git-remote-gitlawb/src/main.rs:315-318) and surfaces the error; git itself does not re-issue a push whose connection died.
• Not the write-concurrency exhaustion in A write guard holds a lock-pool slot for up to 600s, so ~32 concurrent pushes deny every write on the node #282, which is a separate mechanism.
• Not a regression. Before the advisory-lock work in fix(node): release the advisory lock on the session that took it (#279) #285 these transfers were unbounded, so the silent window had no ceiling at all. Bounding it at 300s per span is what makes the number quotable; the mismatch with the edge predates it.

Directions

Fly's documented options for work that outlives the idle timer are to emit keepalive bytes so the timer keeps resetting, or to return early and let the client poll. Raising idle_timeout is the obvious third option but is constrained by the reason it was set to 120 in the first place, recorded inline at infra/fly/fly.toml:39: long idle windows let hung clients pin Fly's connection slots against hard_limit, which was a root factor in the 2026-06-12 outage.

Whichever direction is taken, the timeouts should be settled as a coherent set rather than individually, and the config help for the two knobs should say what the deployment actually permits.

One open question worth resolving first: Fly published a note that idle-timeout-based closing was removed for TCP connections, and it is not clear from their docs whether that covers the http_service path these apps use. If it does, the ceiling may be higher than 120s in practice and the shape of the fix changes.