Repository navigation
fix(integration): detect GitHub App rotation, and revoke what it orphaned - #4
Open
pranav8764 wants to merge 1 commit into
Open
pranav8764 wants to merge 1 commit into
pranav8764 wants to merge 1 commit into
Conversation
A repository added through archiatros-sre never reached the frontend. The
installation, 142620052, was created under a previous GitHub App; .env now
carries GITHUB_APP_ID=4464433. A GitHub App can only mint installation tokens
for its own installations, so POST /app/installations/{id}/access_tokens 404s.
Nothing in the stack noticed. The row stayed status='active' forever and the
only symptom was an opaque 500 with "failed to authorize with github".
Three things made that failure invisible, and all three are fixed here.
github_installations never recorded which App an installation belongs to,
because githubInstallationMeta dropped GitHub's app_id at decode. It is
captured and persisted now (migration 008 adds the column), so a rotation is
detectable locally, before a round trip. NULL stays meaningful: rows written
before the migration never observed a binding, and treating unknown as wrong
would revoke every legacy installation on first read. Those fall through to the
token mint, which is authoritative, and a reconnect backfills the binding.
getInstallationToken collapsed every non-201 into one bare fmt.Errorf, so a 404
was indistinguishable from a 500. It now returns ErrInstallationNotFound on 404
only, and callers branch on it with errors.Is. A 404 can never succeed on
retry, so ListRepositories and AttachRepository mark the installation and its
repository_connections 'revoked' and return FailedPrecondition with "reconnect
required" rather than Internal. Same treatment when the stored app_id is
non-NULL and differs from the configured one, except that case never reaches
GitHub at all. Revocation is status only -- the 532 indexed chunks and the
connection history survive, and reattaching clears revoked_at.
Reconcile-on-read is what enforces "remove a repo from the App and it
disappears from the frontend". The App ships a disabled example.com webhook, so
webhooks do not fire in local development and a read is the only reliable
moment to notice. ListRepositories diffs the live list against stored
connections and revokes the absent ones. It fails open: an errored or timed-out
GitHub call is not an empty list, so on failure it logs, serves the stored rows
and revokes nothing. An outage must never be read as "the user removed
everything". The call is bounded by a 10s context so a slow GitHub degrades to
stored data instead of holding the caller for the full client timeout.
ListRepositoryConnections now filters revoked rows, which is what actually
removes them from the dashboard.
HandleGitHubWebhook verified the HMAC, wrote one audit row and returned without
ever reading req.EventType. It now handles installation_repositories/removed
and installation/deleted|suspend. The payload stays untrusted data, not
instructions: it is used solely to select rows that already exist, by numeric
id, every workspace scope is read from the stored installation rather than the
body, and no webhook may restore access -- only a reconnect through the
authenticated install flow can. Audit reasons are fixed server-side constants
so payload prose cannot reach audit_logs.
Expired unconsumed github_install_sessions were never swept; only the DDL, one
INSERT and the claiming UPDATE referenced the table, so abandoned installs
accumulated forever behind migration 005's partial index. CreateGitHubInstallSession
now deletes them opportunistically, after authorization, and tolerates failure.
IntegrationServer.db is a narrow dbConn interface rather than *pgxpool.Pool.
The revocation and reconciliation branches are pure database effects and were
otherwise reachable only against a live cluster; a scripted fake makes them
testable. Every new statement was also prepared and executed against the
running database inside a rolled-back transaction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
A repository added through the
archiatros-sreGitHub App never appeared in the frontend.Installation
142620052was created under a previous GitHub App;.envnow carriesGITHUB_APP_ID=4464433. A GitHub App can only mint installation tokens for its own installations, so the token endpoint 404s. Nothing in the stack detected this, so the row stayedstatus='active'forever and the only symptom was an opaque HTTP 500.What changed
App binding is now recorded and enforced.
githubInstallationMetadropped GitHub'sapp_idat decode, sogithub_installationsnever knew which App owned a row. It is captured and persisted now. A NULLapp_idis not a mismatch — rows predating migration 008 never observed a binding, and treating unknown as wrong would revoke every legacy installation on first read. Those fall through to the token mint, which is authoritative, and a reconnect backfills the binding.404 is a typed, terminal error.
getInstallationTokencollapsed every non-201 into one barefmt.Errorf. It now returnsErrInstallationNotFoundon 404 only; callers branch witherrors.Is. A 404 can never succeed on retry, soListRepositories/AttachRepositoryrevoke the installation and itsrepository_connectionsand returncodes.FailedPreconditionwith "installation belongs to a different GitHub App; reconnect required". The stored-app_idmismatch path does the same without ever calling GitHub.Revocation is reversible. Status only — the 532 indexed chunks, ingestion history and connection metadata all survive, and reattaching clears
revoked_at.Reconcile-on-read enforces "remove a repo from the App and it disappears from the frontend". The App ships a disabled
example.comwebhook, so webhooks do not fire locally and a read is the only reliable moment to notice.ListRepositoriesdiffs the live list against stored connections and revokes the absent ones. It fails open: an errored or timed-out GitHub call is not an empty list, so on failure it logs, serves the stored rows, and revokes nothing. Bounded by a 10s context.ListRepositoryConnectionsnow filters revoked rows, which is what actually removes them from the dashboard.Webhook actions.
HandleGitHubWebhookverified the HMAC, wrote one audit row, and returned without ever readingreq.EventType. It now handlesinstallation_repositories/removedandinstallation/deleted|suspend. Perdocs/ARCHITECTURE.md, payload prose stays untrusted: it selects existing rows by numeric id only, every workspace scope is read from the stored installation rather than the body, and no webhook may restore access. Audit reasons are fixed server-side constants.Session sweep. Expired unconsumed
github_install_sessionswere never swept, so abandoned installs accumulated forever behind migration 005's partial index.CreateGitHubInstallSessionnow deletes them opportunistically, after authorization, tolerating failure. No background goroutine.Testing
IntegrationServer.dbis now a narrowdbConninterface rather than*pgxpool.Pool— the revocation and reconciliation branches are pure database effects and were otherwise reachable only against a live cluster. A scripted fake makes them testable. 24 new tests cover the 404→revoked path, the app-ID mismatch path, NULLapp_idnot being a mismatch, reconcile-on-read (including the empty-list case), fail-open on outage, every webhook branch, and the invariants that no destructive write occurs and no payload prose reaches a query argument.cd services/integration-service && go test ./...passes; the whole Go suite (gateway, auth, workspace, project, integration) passes. Every new statement was additionally prepared and executed against the running database inside a rolled-back transaction; the live baseline is unchanged (installationactive, both connectionsconnected, 532 chunks).Notes for the coordinator
008_installation_app_binding.sql(U1). Already applied to the live database.TestCredentialEncryptionRoundTripAndTamperDetectionincredentials_test.gois a pre-existing flake, unrelated to this change and outside this unit's file scope. It fails roughly 1 run in 40. It flips the last base64 character of the ciphertext; when that character carries unused low bits, several characters decode to the same byte, so the "tampered" ciphertext is byte-identical and decrypts fine. The fix is to flip a byte of the decoded ciphertext instead.🤖 Generated with Claude Code