Repository navigation
[ROCm][CI] Make /ci rocm cache a cache-only CI build that also copies the runtime #66032
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
AndreasKaratzas
merged 6 commits into
vllm-project:main
from
AndreasKaratzas:akaratza_rocm_cache_on_ci
Nov 16, 2026
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
4adf949
[CI][ROCm] Make /ci rocm cache a cache-only CI build that also copies…
AndreasKaratzas dc092b3
Merge branch 'main' into akaratza_rocm_cache_on_ci
vllm-agent 7ad9a8e
Merge branch 'main' into akaratza_rocm_cache_on_ci
AndreasKaratzas e2a98ee
[ROCm][CI] Promote: finish a chain a retry left, leave early without …
AndreasKaratzas 08717c3
Merge branch 'main' into akaratza_rocm_cache_on_ci
vllm-agent 3c1bd6e
Merge branch 'main' into akaratza_rocm_cache_on_ci
AndreasKaratzas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| group: Hardware - AMD ROCm Image Cache | ||
|
|
||
| # /ci rocm cache starts a main build that runs only this step (VLLM_CI_ONLY_STEP_KEYS): build.sh promote | ||
| # copies the ROCm images a PR's CI built into rocm/vllm-ci. In image_build/, which only the main CI | ||
| # pipeline renders (ci-infra's AMD pipeline renders all of hardware_tests/). Optional, so other | ||
| # builds skip it; nightlies run optional steps, and without a request build.sh exits at once. | ||
| steps: | ||
| - label: ":docker: Cache a PR's ROCm images for main" | ||
| key: rocm-base-image-cache | ||
| depends_on: [] | ||
| device: amd_cpu | ||
| no_plugin: true | ||
| optional: true | ||
| commands: | ||
| - bash .buildkite/scripts/rocm/build.sh rocm promote | ||
| timeout_in_minutes: 30 | ||
| retry: | ||
| automatic: | ||
| - exit_status: 75 # Registry error (build.sh) | ||
| limit: 1 | ||
| - exit_status: -1 # Agent was lost | ||
| limit: 1 | ||
| - exit_status: -10 # Agent was lost | ||
| limit: 1 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,8 +13,8 @@ | |
| # ci Build and push this build's per-commit test image. | ||
| # release Release image + wheels from source into the release repo (release pipeline, AWS agents). | ||
| # release-base Only the release stack and runtime (a serialized step, so one run builds a new key). | ||
| # promote Copy the stack images a PR's CI built into the trusted repo (/ci rocm cache), so | ||
| # main reuses them after the merge instead of rebuilding the stack. | ||
| # promote Copy the stack, runtime and test-deps images a PR's CI built into the trusted repo | ||
| # (/ci rocm cache), so main reuses them after the merge instead of rebuilding them. | ||
| # | ||
| # Stack: rocm (TheRock ROCm 10). Force a stack rebuild: bump ARG REFRESH | ||
| # in docker/Dockerfile.rocm_base | ||
|
|
@@ -44,6 +44,15 @@ MODE=$2 | |
| REPO=${ROCM_CI_REPO:?ROCM_CI_REPO must be set (by ci-infra or release-pipeline.yaml)} | ||
| TRUSTED=${ROCM_CI_TRUSTED_REPO:-rocm/vllm-ci} | ||
|
|
||
| # promote (/ci rocm cache) runs in every nightly, as an optional step: without a request, leave | ||
| # before the builder, the build context and the tag fetch. | ||
| if [[ $MODE == promote ]]; then | ||
| [[ -n ${ROCM_PROMOTE_PR:-} ]] || { echo "No /ci rocm cache request: nothing to promote" >&2; exit 0; } | ||
| [[ $ROCM_PROMOTE_PR =~ ^[0-9]+$ && ${ROCM_PROMOTE_SHA:-} =~ ^[0-9a-f]{40}$ ]] \ | ||
| || { echo "ROCM_PROMOTE_PR or ROCM_PROMOTE_SHA is malformed" >&2; exit 2; } | ||
| [[ $REPO == "$TRUSTED" ]] || { echo "promote only pushes to $TRUSTED" >&2; exit 2; } | ||
| fi | ||
|
|
||
| export STACK PUSH=true | ||
| export CACHE=${ROCM_CI_CACHE_REPO:-rocm/vllm-ci-cache} # every CI build reads the trusted cache | ||
| export CACHE_TO=$CACHE: # ref prefix trusted builds write (HCL appends <stack>-<name>) | ||
|
|
@@ -102,6 +111,7 @@ docker buildx version | |
| # COPY on file modes, which drift on shared agents. umask 022 gives every agent git's modes. | ||
| [[ $(git rev-parse HEAD) == $(git rev-parse "${BUILDKITE_COMMIT:-HEAD}^{commit}") ]] \ | ||
| || { echo "HEAD is not BUILDKITE_COMMIT ($BUILDKITE_COMMIT)" >&2; exit 1; } | ||
| if [[ $MODE != promote ]]; then # promote copies images: no context, no version | ||
| umask 022 | ||
| rm -rf .rocm-src && mkdir .rocm-src | ||
| git archive HEAD | tar -x --no-same-permissions -C .rocm-src -f - | ||
|
|
@@ -120,6 +130,7 @@ git describe --tags --match 'v[0-9]*' >/dev/null 2>&1 \ | |
| printf 'node: %s\nnode-date: %s\ndescribe-name: %s\n' "$(git rev-parse HEAD)" \ | ||
| "$(git log -1 --format=%cI)" "$(git describe --tags --long --abbrev=40 --match 'v[0-9]*')" \ | ||
| > .rocm-src/.git_archival.txt | ||
| fi | ||
|
Comment on lines
114
to
+133
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. NIT: (Cosmetic change) missing indentation under the if-else block |
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Helper functions | ||
|
|
@@ -277,10 +288,22 @@ ensure() { | |
| rm -f "$mf"; echo "bake recorded no digest for $ref" >&2; return 1 | ||
| fi | ||
| rm -f "$mf" | ||
| # /ci rocm cache copies only a stack this build pushed, never one it reused: other PRs' CI | ||
| # can push to this PR's tags first. | ||
| [[ $MODE != base || $target != base ]] \ | ||
| || meta set "rocm-$STACK-stack-built" "$ref@$d $ref$sibling@$ds" || return 75 | ||
| # /ci rocm cache copies only images this job pushed, and a runtime or test-deps only if this job | ||
| # also pushed the image it was built on: other PRs' CI can push to this PR's tags first, and | ||
| # metadata outlives a retried job. A new stack voids what an earlier attempt recorded on another. | ||
| local parent=none | ||
| case $MODE/$target in | ||
| base/base) | ||
| { meta set "rocm-$STACK-runtime-built" none && meta set "rocm-$STACK-test-deps-built" none; } \ | ||
| || return 75 | ||
| parent= ;; | ||
| base/runtime) parent=base ;; | ||
| base/test-deps) parent=runtime ;; | ||
| esac | ||
| if [[ -z $parent ]] || grep -qsx "$parent" "$BUILT_HERE"; then | ||
| meta set "rocm-$STACK-${target/#base/stack}-built" "$ref@$d${sibling:+ $ref$sibling@$ds}" || return 75 | ||
| echo "$target" >>"$BUILT_HERE" | ||
| fi | ||
| echo "$ref@$d${sibling:+ $ref$sibling@$ds}" | ||
| } | ||
|
|
||
|
|
@@ -303,7 +326,8 @@ heartbeat() { | |
| # or a planted mark), and a stack built after it may not fit the step's 540-min timeout. | ||
| WAIT_UNTIL=$(( $(date +%s) + ${ROCM_TRUSTED_WAIT_MINUTES:-300} * 60 )) | ||
| NO_MAIN_BUILD=$(mktemp -u) # a file, not a variable: each ensure() runs in its own subshell | ||
| trap 'rm -f "$NO_MAIN_BUILD"' EXIT | ||
| BUILT_HERE=$(mktemp -u) # the targets this job built, likewise | ||
| trap 'rm -f "$NO_MAIN_BUILD" "$BUILT_HERE"' EXIT | ||
| wait_for_trusted() { | ||
| local tag=$1 sibling=${2:-} ref=$TRUSTED:$1 d ds rc beat now start | ||
| local stale=$(( ${ROCM_HEARTBEAT_STALE_MINUTES:-15} * 60 )) | ||
|
|
@@ -399,21 +423,40 @@ case $MODE in | |
|
|
||
| promote) | ||
| # /ci rocm cache: ROCM_PROMOTE_IMAGES are the digests #ROCM_PROMOTE_PR's CI build of | ||
| # ROCM_PROMOTE_SHA pushed. Copy the stack pair here under the key main's keys_at gives that | ||
| # commit, if it was built with main's recipe; main reuses it once it gets that key. | ||
| [[ $REPO == "$TRUSTED" ]] || { echo "promote only pushes to $TRUSTED" >&2; exit 2; } | ||
| # ROCM_PROMOTE_SHA pushed. Copy them here under the keys main's keys_at gives that commit, if | ||
| # it was built with main's recipe; main reuses each once it gets that key. The runtime and | ||
| # test-deps are copied only over a parent of #ROCM_PROMOTE_PR's, never over one main built: an | ||
| # existing tag holding the recorded digest is an earlier attempt's copy, so a retry finishes the | ||
| # chain instead of stopping at it. | ||
| git fetch -q origin "${ROCM_PROMOTE_SHA:?}" | ||
| git diff --quiet HEAD "$ROCM_PROMOTE_SHA" -- .buildkite/scripts/rocm/build.sh docker/docker-bake-rocm.hcl \ | ||
| || { echo "$ROCM_PROMOTE_SHA builds with another build.sh or bake file than main" >&2; exit 1; } | ||
| keys=$(keys_at "$ROCM_PROMOTE_SHA") || { echo "cannot compute the keys of $ROCM_PROMOTE_SHA" >&2; exit 1; } | ||
| read -r pKs _ <<<"$keys" | ||
| for tag in "stack-$STACK-$pKs-wheels" "stack-$STACK-$pKs"; do # the stack tag completes the pair | ||
| if digest "$TRUSTED:$tag" >/dev/null; then echo "$TRUSTED:$tag already exists" >&2; continue; fi | ||
| read -r pKs pKr <<<"$keys" | ||
| copied=0 | ||
| for tag in "stack-$STACK-$pKs-wheels" "stack-$STACK-$pKs" \ | ||
| "runtime-$STACK-$pKr" "test-deps-$STACK-$pKr"; do # the stack tag completes the pair | ||
| stacked=0; [[ $tag == stack-* ]] && stacked=1 | ||
| ref=$(tr ' ' '\n' <<<"${ROCM_PROMOTE_IMAGES:-}" \ | ||
| | grep -xE "rocm/vllm-dev:pr${ROCM_PROMOTE_PR:?}-$tag@sha256:[0-9a-f]{64}") \ | ||
| || { echo "#$ROCM_PROMOTE_PR's CI recorded no $tag" >&2; exit 1; } | ||
| | grep -xE "rocm/vllm-dev:pr$ROCM_PROMOTE_PR-$tag@sha256:[0-9a-f]{64}") || ref="" | ||
| if d=$(digest "$TRUSTED:$tag"); then | ||
| if [[ -n $ref && $d == "${ref#*@}" ]]; then | ||
| echo "$TRUSTED:$tag already holds #$ROCM_PROMOTE_PR's image" >&2; copied=1 | ||
| else | ||
| echo "$TRUSTED:$tag already exists" >&2; copied=0 | ||
| fi | ||
| continue | ||
| else | ||
| rc=$?; ((rc == 1)) || exit $rc # a registry error is not a missing tag | ||
| fi | ||
| ((stacked || copied)) || { echo "Not copying $tag over an image main built" >&2; break; } | ||
| if [[ -z $ref ]]; then | ||
| ((stacked)) || { echo "#$ROCM_PROMOTE_PR's CI recorded no $tag; main builds it" >&2; break; } | ||
| echo "#$ROCM_PROMOTE_PR's CI recorded no $tag" >&2; exit 1 | ||
| fi | ||
| echo "--- :docker: Copying $ref to $TRUSTED:$tag" >&2 | ||
| docker buildx imagetools create -t "$TRUSTED:$tag" "$ref" || exit 75 | ||
| copied=1 | ||
| done | ||
| ;; | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could be a misleady green signal. Let's fix this or drop the 75 retry.
Given the
copiedreset in build.sh, this retry can't actually recover a partial copy - it re-runs into thebreakand goes green.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Taking some of it back... retry does earn its place, so don't simply drop it.
The narrower point still stands though: a failure mid-chain retries into the
breakandgoes green having not finished. That's the misleading-green bit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, kept the retry and fixed the chain instead (e2a98ee, see the thread on build.sh). A retry after a mid-chain registry error now finishes the runtime and test-deps rather than going green.