Repository navigation
op-stack-go: distrust the compromised eMudhra CAs in Go images - #561
Merged
Merged
Conversation
The digest-pinned alpine base freezes ca-certificates-bundle at 20260909, which still trusts emSign ECC Root CA - G3. Its intermediates were compromised (eMudhra, Oct 2026) and Mozilla removed the root from NSS on 2026-10-20. Go does not check revocation, so op-node, op-batcher and the other alpine-based images keep accepting certificates forged under it until their trust store changes. Upgrade the bundle at build time in a shared target-base stage, so a rebuild picks up root removals without waiting for a new base digest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
There's also ethereum-optimism#23546, which hard-codes a blocklist of the eight compromised intermediate keys in crypto/x509. It probably makes sense to do both to be on the safe side. |
Go 1.26.10 adds a crypto/x509 denylist of the eight compromised eMudhra intermediates (GO-2026-6809), so Verify rejects chains through them even when the system trust store still holds their root. Neither 1.25.14 nor 1.24.13 has it. The golang builder image sets GOTOOLCHAIN=local, so the go.mod toolchain line alone does not change the compiler used for the images; bump the builder images too, and mise so local and CI tests use the same version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alvarof2
changed the title
karlb
reviewed
Oct 21, 2026
apk upgrade does not guarantee a minimum version, and a cached build layer would keep whatever bundle it first installed. A version floor fails the build instead of shipping a bundle that still trusts the compromised emSign root, and changes the layer's cache key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
karlb
reviewed
Oct 23, 2026
Comment on lines
+295
to
+296
| FROM $TARGET_BASE_IMAGE AS target-base | ||
| RUN n=0; until apk add --no-cache --upgrade 'ca-certificates-bundle>=20261020'; do n=$((n+1)); [ "$n" -ge 15 ] && exit 1; echo "apk add retry $n/15 in 20s" >&2; sleep 20; done |
There was a problem hiding this comment.
This runs the arm64 base image's /bin/sh and apk. docker-publish-release.yaml builds linux/amd64,linux/arm64 without QEMU (Buildx lists only amd64/386 platforms in the v2.2.4 release log), so the arm64 leg will likely fail with exec format error. PR CI builds amd64 only, so it won't catch this.
Fix: add docker/setup-qemu-action@v3 before setup-buildx-action, or install the bundle in a --platform=$BUILDPLATFORM stage and COPY only ca-certificates.crt into target-base.
The release workflow cross-builds linux/arm64 without QEMU, so running apk in the target-platform target-base stage would fail with "exec format error" on the arm64 leg. PR CI builds amd64 only and would not catch it. Run the upgrade in a BUILDPLATFORM stage and copy only /etc/ssl/certs/ca-certificates.crt into target-base. The bundle is arch-independent and it is the only regular file the package ships; /etc/ssl/cert.pem is a symlink to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
karlb
approved these changes
Oct 23, 2026
palango
approved these changes
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The op-node, op-batcher, op-proposer and other alpine-based images accept certificates that chain to the eight eMudhra intermediates whose keys were compromised (keys recovered from weak generation; Mozilla removed emSign ECC Root CA - G3 from NSS on 2026-10-20, and Chrome removes it on 2026-10-27). Go does not check revocation. Until this is fixed, a MITM on any outbound TLS connection from these binaries, such as L1 RPC or the beacon API, can present a forged certificate and be accepted.
This PR applies two independent defences:
TARGET_BASE_IMAGEfreezesca-certificates-bundleat20260909-r0, which still contains G3 (the deployedcelo-v2.2.4image has20260413-r0). Thealpine:3.23tag still resolves to that digest today, so moving the pin is not enough. Alpine's20261020bundle (which removes G3, C1 and C3) is installed at build time with a>=20261020version floor. If that version is not available the build fails, so an older bundle is never shipped, and a cached layer cannot keep a stale bundle. For a later root removal, raise the floor. Theapkupgrade runs in a$BUILDPLATFORMstage, and only/etc/ssl/certs/ca-certificates.crtis copied into the sharedtarget-basestage (/etc/ssl/cert.pemis a symlink to it). The release workflow buildslinux/arm64without QEMU, so runningapkin a target-platform stage would fail there withexec format error. PR CI builds amd64 only and would not catch that. The bundle is architecture-independent. The image's apk database still lists the base image'sca-certificates-bundleversion, so image scanners report the old version.crypto/x509denylist of the eight intermediates (GO-2026-6809), which works even when a trust store is stale. Neither 1.25.14 nor 1.24.13 has it. The golang builder image setsGOTOOLCHAIN=local, so thego.modtoolchain line alone does not change the compiler used for the images (upstream security: Update go toolchain directive to v1.26.10 [SECURITY] ethereum-optimism/optimism#23546 changes only that line). The builder images andmise.tomlare bumped as well.There is no other effect on users. The cannon and kona prestate Dockerfiles stay on 1.26.5, because changing their Go version changes the reproducible prestate. op-challenger (Wolfi), cannon and op-deployer (
dhi.io/alpine-base) runtime bases andops/docker/deployment-utilsare not changed. The pinned Wolfi base still shipsca-certificates-bundle-20260413-r0, which contains G3, C1 and C3, so op-challenger is protected only by the Go denylist. The cannon and op-deployer images use the unpinneddhi.io/alpine-base:3.23tag, which currently (sha256:2049edbe…) ships a bundle without G3, C1 or C3.A local
op-node-targetbuild gives a binary compiled withgo1.26.10that contains the denylist hashes, and the image hasca-certificates-bundle-20261020-r0withoutemSign ECC Root CA - G3. None of the repo's review agents match this diff.Related: celo-org/celo-kona#360 (the same fix for op-reth's compiled-in roots).
🤖 Generated with Claude Code