Skip to content

fix: refresh runtime-proxy credentials before pruning a session on 404/401 - #123

Open
zoutei wants to merge 1 commit into
googlecolab:mainfrom
zoutei:fix/proxy-token-expiry
Open

zoutei wants to merge 1 commit into
googlecolab:mainfrom
zoutei:fix/proxy-token-expiry

Conversation

@zoutei

@zoutei zoutei commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

The runtime-proxy token is short-lived, roughly 60 minutes (the server reports a 3600-second lifetime via tokenExpiresInSeconds), and expires well before an assignment is actually torn down. An expired token produces the exact same 404/401 that a genuinely terminated runtime would. exec, repl, console, and run treated any 404/401 as proof the session was lost and deleted the local binding outright, even though list_assignments() keeps listing the assignment and keeps minting it a fresh proxy token long after the previously-stored token stops working. A live, still-billing runtime could be destroyed by its own client roughly once per hour on any session kept open longer than that.

Fixes #106.

What changed

State.prune_or_recover_session() replaces the unconditional prune on this error path. Before deleting anything it asks the server. If the endpoint is still listed, it adopts the fresh token and url and keeps the local session; this invocation still fails, but the next one succeeds with the refreshed credential. If the endpoint is confirmed gone, it prunes as before. If the control plane is unreachable, it preserves the session rather than delete on an inconclusive check. This only runs reactively, in response to an actual 404/401 from the runtime proxy; there's no background or proactive token refresh.

Testing

Full test suite passes. Added unit coverage for all three outcomes of prune_or_recover_session, plus CLI-level tests covering both the pruned and preserved paths for exec.

…4/401

The runtime proxy token is short-lived and expires well before an
assignment is torn down, so an expired token produces the exact same
404/401 that a genuinely terminated runtime would. `exec`/`repl`/
`console`/`run` treated that error as proof the session was lost and
deleted the local binding outright -- even though `list_assignments()`
keeps listing the assignment (and keeps minting it a fresh proxy
token) long after the stored token has stopped working. A live,
billing runtime was being destroyed by its own client roughly once per
token lifetime on any long-running session.

`State.prune_or_recover_session()` asks the server before deleting:
if the endpoint is still listed, adopt the fresh token/url and keep
the session (this call still reports failure, but the next one
succeeds); if the endpoint is confirmed gone, prune as before; if the
control plane can't be reached at all, keep the session rather than
delete on an inconclusive check.

Fixes googlecolab#106.
@seantangth

Copy link
Copy Markdown

Independent validation of this fix (3-hour A/B run)

I backported the src/ changes of this PR onto the 0.6.0 PyPI release (the hunks apply with line offsets only; the added/removed lines are identical) and ran it side by side with stock 0.6.0.

Setup: macOS, Python 3.14, google-colab-cli 0.6.0, jupyter-kernel-client 0.14.0, --auth=adc. Two CPU sessions created 15 seconds apart, with identical dependency sets. On each VM a background process appends one line to /content/hb.log every 60 s. Every 5 minutes each arm runs colab download on that file and colab exec -f a script that reads it (a fresh connection each time), and I check list_assignments(). Only the exec path was exercised (not run / repl / console).

stock 0.6.0 0.6.0 + this PR
First token expiry +60.6 min: appears to be lost (404/401). Cleaning up. The local session is deleted and the keep-alive daemon is killed +60.6 min: the "preserved the local session" message; a retry 5 s later succeeded
After that The assignment was still listed by list_assignments() at +65, +70 and +75 min (I then unassigned it manually) Ran to +190.6 min and hit the expiry 3 times (+60.6 / +125.6 / +190.6); each time the first retry succeeded
Keep-alive daemon Killed by the prune Alive throughout, 0 keep_alive_error events
Job on the VM Orphaned The heartbeat file grew by exactly 5 lines per 5-minute poll and had 191 lines at the end (as expected), so the job was never interrupted

The refreshed token's lifetime restarts at issuance (it expired again about 3600 s after each refresh), which matches tokenExpiresInSeconds: 3600.

One gap this PR doesn't cover: file operations. After the token expires, colab download (ContentsClient) also gets a 404 and fails with Download failed: File or directory not found: /content/hb.log, even though the file exists. This failure showed up at every expiry in both arms. File operations don't go through prune_or_recover_session, so in the patched arm download only started working again after an exec had refreshed the token (3/3). upload / ls / rm use the same client, so they are presumably affected too, but I didn't test them. #109 seems to handle that path, so it could be a follow-up here.

Thanks for the fix. It would be great to see it merged and released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Live sessions are pruned locally after 60 minutes — proxy token expiry is misclassified as "session lost"

2 participants