ci(docker): build on self-hosted runner with a warm BuildKit cache - #86
Merged
Conversation
The docker image build ran on ubuntu-latest with no build cache, so every hotblocks/archive image recompiled the whole Rust dependency graph from scratch. - Dockerfile: add BuildKit cargo cache mounts (registry/git + per-arch target/) so only changed crates recompile; binaries are copied out of the target cache mount to /out for the runtime stages. - workflow: run on [self-hosted, dev-server] (safe: workflow_dispatch-only), reuse a persistent named buildx builder (keep-state) so the cache survives across jobs, cap it at 15GB via BuildKit GC, add QEMU for arm64. - default platforms to linux/amd64 (arm64 opt-in per dispatch) to keep the cache to one target dir on the disk-constrained host. Pairs with subsquid/dev-server: the github-runner role must stop the nightly `docker system prune --volumes` from wiping the builder's cache volume. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
dockerworkflow (manual hotblocks/archive image builds) ran onubuntu-latestwith no build cache. Every dispatch recompiled the entire Rust dependency graph from scratch. Three compounding causes:ADD cratesbefore a single monolithiccargo build, no cache mounts → any source change recompiles all deps.ubuntu-latest(ephemeral) +build-push-actionwith nocache-from/cache-to.docker system prune -a -f --volumes(see companion dev-server PR).Change
registry/git(shared) andtarget/(per-archid=cargo-target-$TARGETARCH,sharing=locked). Binaries are copied out of thetarget/cache mount to/outfor the runtime stages. Only changed crates recompile now.[self-hosted, dev-server](safe —workflow_dispatch-only, maintainer-triggered), reuses a persistent named buildx builderdev-server-cache(keep-state,cleanup: false) so the cache survives across jobs, caps it at 15GB via BuildKit GC, and adds QEMU for arm64.platforms→linux/amd64(arm64 stays available per dispatch). Keeps the cache to a single target dir, which matters on this disk-constrained host.Requires
subsquid/datamust have access to thedev-serverrunner group.Sizing (measured on a real Linux release build of this repo)
target/release≈ 1.5GB, cargo registry ≈ 1.9GB, base image/layers ≈ 5GB → ~9GB for one arch. 15GB cap leaves comfortable headroom.🤖 Generated with Claude Code