Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .buildkite/image_build/amd_rocm_cache.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
group: Hardware - AMD ROCm Image Cache

# /ci rocm cache starts a main build that runs only this step (VLLM_CI_ONLY_STEP_KEYS): build.sh promote
# copies the ROCm images a PR's CI built into rocm/vllm-ci. In image_build/, which only the main CI
# pipeline renders (ci-infra's AMD pipeline renders all of hardware_tests/). Optional, so other
# builds skip it; nightlies run optional steps, and without a request build.sh exits at once.
steps:
- label: ":docker: Cache a PR's ROCm images for main"
key: rocm-base-image-cache
depends_on: []
device: amd_cpu
no_plugin: true
optional: true
commands:
- bash .buildkite/scripts/rocm/build.sh rocm promote
timeout_in_minutes: 30
retry:
automatic:
- exit_status: 75 # Registry error (build.sh)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be a misleady green signal. Let's fix this or drop the 75 retry.
Given the copied reset in build.sh, this retry can't actually recover a partial copy - it re-runs into the break and goes green.

@divakar-amd divakar-amd Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taking some of it back... retry does earn its place, so don't simply drop it.
The narrower point still stands though: a failure mid-chain retries into the break and
goes green having not finished. That's the misleading-green bit.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, kept the retry and fixed the chain instead (e2a98ee, see the thread on build.sh). A retry after a mid-chain registry error now finishes the runtime and test-deps rather than going green.

limit: 1
- exit_status: -1 # Agent was lost
limit: 1
- exit_status: -10 # Agent was lost
limit: 1
73 changes: 58 additions & 15 deletions .buildkite/scripts/rocm/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@
# ci Build and push this build's per-commit test image.
# release Release image + wheels from source into the release repo (release pipeline, AWS agents).
# release-base Only the release stack and runtime (a serialized step, so one run builds a new key).
# promote Copy the stack images a PR's CI built into the trusted repo (/ci rocm cache), so
# main reuses them after the merge instead of rebuilding the stack.
# promote Copy the stack, runtime and test-deps images a PR's CI built into the trusted repo
# (/ci rocm cache), so main reuses them after the merge instead of rebuilding them.
#
# Stack: rocm (TheRock ROCm 10). Force a stack rebuild: bump ARG REFRESH
# in docker/Dockerfile.rocm_base
Expand Down Expand Up @@ -44,6 +44,15 @@ MODE=$2
REPO=${ROCM_CI_REPO:?ROCM_CI_REPO must be set (by ci-infra or release-pipeline.yaml)}
TRUSTED=${ROCM_CI_TRUSTED_REPO:-rocm/vllm-ci}

# promote (/ci rocm cache) runs in every nightly, as an optional step: without a request, leave
# before the builder, the build context and the tag fetch.
if [[ $MODE == promote ]]; then
[[ -n ${ROCM_PROMOTE_PR:-} ]] || { echo "No /ci rocm cache request: nothing to promote" >&2; exit 0; }
[[ $ROCM_PROMOTE_PR =~ ^[0-9]+$ && ${ROCM_PROMOTE_SHA:-} =~ ^[0-9a-f]{40}$ ]] \
|| { echo "ROCM_PROMOTE_PR or ROCM_PROMOTE_SHA is malformed" >&2; exit 2; }
[[ $REPO == "$TRUSTED" ]] || { echo "promote only pushes to $TRUSTED" >&2; exit 2; }
fi

export STACK PUSH=true
export CACHE=${ROCM_CI_CACHE_REPO:-rocm/vllm-ci-cache} # every CI build reads the trusted cache
export CACHE_TO=$CACHE: # ref prefix trusted builds write (HCL appends <stack>-<name>)
Expand Down Expand Up @@ -102,6 +111,7 @@ docker buildx version
# COPY on file modes, which drift on shared agents. umask 022 gives every agent git's modes.
[[ $(git rev-parse HEAD) == $(git rev-parse "${BUILDKITE_COMMIT:-HEAD}^{commit}") ]] \
|| { echo "HEAD is not BUILDKITE_COMMIT ($BUILDKITE_COMMIT)" >&2; exit 1; }
if [[ $MODE != promote ]]; then # promote copies images: no context, no version
umask 022
rm -rf .rocm-src && mkdir .rocm-src
git archive HEAD | tar -x --no-same-permissions -C .rocm-src -f -
Expand All @@ -120,6 +130,7 @@ git describe --tags --match 'v[0-9]*' >/dev/null 2>&1 \
printf 'node: %s\nnode-date: %s\ndescribe-name: %s\n' "$(git rev-parse HEAD)" \
"$(git log -1 --format=%cI)" "$(git describe --tags --long --abbrev=40 --match 'v[0-9]*')" \
> .rocm-src/.git_archival.txt
fi
Comment on lines 114 to +133

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: (Cosmetic change) missing indentation under the if-else block


# ---------------------------------------------------------------------------
# Helper functions
Expand Down Expand Up @@ -277,10 +288,22 @@ ensure() {
rm -f "$mf"; echo "bake recorded no digest for $ref" >&2; return 1
fi
rm -f "$mf"
# /ci rocm cache copies only a stack this build pushed, never one it reused: other PRs' CI
# can push to this PR's tags first.
[[ $MODE != base || $target != base ]] \
|| meta set "rocm-$STACK-stack-built" "$ref@$d $ref$sibling@$ds" || return 75
# /ci rocm cache copies only images this job pushed, and a runtime or test-deps only if this job
# also pushed the image it was built on: other PRs' CI can push to this PR's tags first, and
# metadata outlives a retried job. A new stack voids what an earlier attempt recorded on another.
local parent=none
case $MODE/$target in
base/base)
{ meta set "rocm-$STACK-runtime-built" none && meta set "rocm-$STACK-test-deps-built" none; } \
|| return 75
parent= ;;
base/runtime) parent=base ;;
base/test-deps) parent=runtime ;;
esac
if [[ -z $parent ]] || grep -qsx "$parent" "$BUILT_HERE"; then
meta set "rocm-$STACK-${target/#base/stack}-built" "$ref@$d${sibling:+ $ref$sibling@$ds}" || return 75
echo "$target" >>"$BUILT_HERE"
fi
echo "$ref@$d${sibling:+ $ref$sibling@$ds}"
}

Expand All @@ -303,7 +326,8 @@ heartbeat() {
# or a planted mark), and a stack built after it may not fit the step's 540-min timeout.
WAIT_UNTIL=$(( $(date +%s) + ${ROCM_TRUSTED_WAIT_MINUTES:-300} * 60 ))
NO_MAIN_BUILD=$(mktemp -u) # a file, not a variable: each ensure() runs in its own subshell
trap 'rm -f "$NO_MAIN_BUILD"' EXIT
BUILT_HERE=$(mktemp -u) # the targets this job built, likewise
trap 'rm -f "$NO_MAIN_BUILD" "$BUILT_HERE"' EXIT
wait_for_trusted() {
local tag=$1 sibling=${2:-} ref=$TRUSTED:$1 d ds rc beat now start
local stale=$(( ${ROCM_HEARTBEAT_STALE_MINUTES:-15} * 60 ))
Expand Down Expand Up @@ -399,21 +423,40 @@ case $MODE in

promote)
# /ci rocm cache: ROCM_PROMOTE_IMAGES are the digests #ROCM_PROMOTE_PR's CI build of
# ROCM_PROMOTE_SHA pushed. Copy the stack pair here under the key main's keys_at gives that
# commit, if it was built with main's recipe; main reuses it once it gets that key.
[[ $REPO == "$TRUSTED" ]] || { echo "promote only pushes to $TRUSTED" >&2; exit 2; }
# ROCM_PROMOTE_SHA pushed. Copy them here under the keys main's keys_at gives that commit, if
# it was built with main's recipe; main reuses each once it gets that key. The runtime and
# test-deps are copied only over a parent of #ROCM_PROMOTE_PR's, never over one main built: an
# existing tag holding the recorded digest is an earlier attempt's copy, so a retry finishes the
# chain instead of stopping at it.
git fetch -q origin "${ROCM_PROMOTE_SHA:?}"
git diff --quiet HEAD "$ROCM_PROMOTE_SHA" -- .buildkite/scripts/rocm/build.sh docker/docker-bake-rocm.hcl \
|| { echo "$ROCM_PROMOTE_SHA builds with another build.sh or bake file than main" >&2; exit 1; }
keys=$(keys_at "$ROCM_PROMOTE_SHA") || { echo "cannot compute the keys of $ROCM_PROMOTE_SHA" >&2; exit 1; }
read -r pKs _ <<<"$keys"
for tag in "stack-$STACK-$pKs-wheels" "stack-$STACK-$pKs"; do # the stack tag completes the pair
if digest "$TRUSTED:$tag" >/dev/null; then echo "$TRUSTED:$tag already exists" >&2; continue; fi
read -r pKs pKr <<<"$keys"
copied=0
for tag in "stack-$STACK-$pKs-wheels" "stack-$STACK-$pKs" \
"runtime-$STACK-$pKr" "test-deps-$STACK-$pKr"; do # the stack tag completes the pair
stacked=0; [[ $tag == stack-* ]] && stacked=1
ref=$(tr ' ' '\n' <<<"${ROCM_PROMOTE_IMAGES:-}" \
| grep -xE "rocm/vllm-dev:pr${ROCM_PROMOTE_PR:?}-$tag@sha256:[0-9a-f]{64}") \
|| { echo "#$ROCM_PROMOTE_PR's CI recorded no $tag" >&2; exit 1; }
| grep -xE "rocm/vllm-dev:pr$ROCM_PROMOTE_PR-$tag@sha256:[0-9a-f]{64}") || ref=""
if d=$(digest "$TRUSTED:$tag"); then
if [[ -n $ref && $d == "${ref#*@}" ]]; then
echo "$TRUSTED:$tag already holds #$ROCM_PROMOTE_PR's image" >&2; copied=1
else
echo "$TRUSTED:$tag already exists" >&2; copied=0
fi
continue
else
rc=$?; ((rc == 1)) || exit $rc # a registry error is not a missing tag
fi
((stacked || copied)) || { echo "Not copying $tag over an image main built" >&2; break; }
if [[ -z $ref ]]; then
((stacked)) || { echo "#$ROCM_PROMOTE_PR's CI recorded no $tag; main builds it" >&2; break; }
echo "#$ROCM_PROMOTE_PR's CI recorded no $tag" >&2; exit 1
fi
echo "--- :docker: Copying $ref to $TRUSTED:$tag" >&2
docker buildx imagetools create -t "$TRUSTED:$tag" "$ref" || exit 75
copied=1
done
;;

Expand Down
29 changes: 22 additions & 7 deletions .github/workflows/scripts/run_ci_command.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
*(command for command in RUN_CI_COMMAND_ENV if command.startswith("/amd-ci ")),
COMMAND_RETRY_AMD_FAILED,
COMMAND_CANCEL_AMD_CI,
*ROCM_CACHE_COMMANDS, # they start AMD CI builds
*ROCM_CACHE_COMMANDS, # searched in AMD CI too; the cache build runs in CI
}
)
RETRY_COMMANDS = frozenset({COMMAND_RETRY_FAILED, COMMAND_RETRY_AMD_FAILED})
Expand Down Expand Up @@ -92,9 +92,18 @@
)
# build.sh promote copies only images built with main's versions of these.
ROCM_RECIPE = (".buildkite/scripts/rocm/build.sh", "docker/docker-bake-rocm.hcl")
# build.sh base mode records the stack pair under this key only when it pushed
# them itself, not when it reused a tag another PR's CI could have pushed.
# build.sh base mode records the images it pushed itself under these keys, not
# ones it reused from a tag another PR's CI could have pushed, and a runtime or
# test-deps only with the parent it was built on ("none" otherwise). Only a build
# that pushed the stack is a source; build.sh promote copies the rest over it.
ROCM_STACK_BUILT_KEY = "rocm-rocm-stack-built"
ROCM_BUILT_KEYS = (
ROCM_STACK_BUILT_KEY,
"rocm-rocm-runtime-built",
"rocm-rocm-test-deps-built",
)
# The one step a cache build runs (.buildkite/image_build/amd_rocm_cache.yaml).
ROCM_CACHE_STEP_KEY = "rocm-base-image-cache"
BUILDS_PER_PAGE = 100
MAX_BUILD_PAGES = 100

Expand Down Expand Up @@ -1315,21 +1324,27 @@ def handle_rocm_cache(
"its CI built the ROCm base image before main changed them. Use "
f"`{COMMAND_ROCM_NOCACHE}`."
)
build = buildkite.create_build(
meta_data = source["meta_data"]
build = ci_buildkite.create_build(
{
"commit": "HEAD",
"branch": "main",
"message": f"Cache the ROCm base image of PR #{pr['number']} by @{actor}",
"env": {
"ROCM_PROMOTE_PR": str(pr["number"]),
"ROCM_PROMOTE_SHA": sha,
"ROCM_PROMOTE_IMAGES": source["meta_data"][ROCM_STACK_BUILT_KEY],
"ROCM_PROMOTE_IMAGES": " ".join(
meta_data[key]
for key in ROCM_BUILT_KEYS
if meta_data.get(key, "none") != "none"
),
"VLLM_CI_ONLY_STEP_KEYS": json.dumps([ROCM_CACHE_STEP_KEY]),
},
}
)
return (
f"Caching the ROCm base image for main: [AMD CI #{build['number']}]"
f"({build['web_url']}) copies the image [this PR's CI]({source['web_url']}) "
f"Caching the ROCm base image for main: [CI #{build['number']}]"
f"({build['web_url']}) copies the images [this PR's CI]({source['web_url']}) "
f"built at `{sha[:12]}`. {ROCM_CACHE_DONE}"
)

Expand Down
50 changes: 40 additions & 10 deletions .github/workflows/scripts/test_run_ci_command.py
Original file line number Diff line number Diff line change
Expand Up @@ -1697,6 +1697,7 @@ def comment(
return amd

def test_commands_are_rocm_ci_commands_on_amd_ci(self) -> None:
# AMD CI is searched for the source build; the cache build runs in CI.
for command in (COMMAND_ROCM_CACHE, COMMAND_ROCM_NOCACHE):
self.assertEqual(parse_command(command), command)
self.assertEqual(pipeline_for_command(command), "amd-ci")
Expand All @@ -1721,28 +1722,57 @@ def test_cache_copies_the_images_this_prs_ci_pushed(self) -> None:

amd = self.comment(github, ci_builds=builds)

(build,) = amd.created_builds
# One CI build that runs only the cache step, not a build in AMD CI.
self.assertEqual(amd.created_builds, [])
(build,) = self.ci.created_builds
self.assertEqual((build["branch"], build["commit"]), ("main", "HEAD"))
self.assertEqual(
build["env"],
{
"ROCM_PROMOTE_PR": "42",
"ROCM_PROMOTE_SHA": "fedcba9876543210",
"ROCM_PROMOTE_IMAGES": "s@d1 w@d2",
"VLLM_CI_ONLY_STEP_KEYS": '["rocm-base-image-cache"]',
},
)
self.assertIn(ROCM_CACHE_MARKER, github.comments[0])
self.assertIn("ci/builds/2", github.comments[0])
self.assertIn("`fedcba987654`", github.comments[0])
self.assertEqual(self.ci.list_requests[0]["branch"], "contributor:feature")

def test_cache_copies_the_runtime_and_test_deps_the_source_pushed(self) -> None:
meta_data = {
**STACK_BUILT,
"rocm-rocm-runtime-built": "r@d3",
"rocm-rocm-test-deps-built": "t@d4",
}
github = FakeGitHub(files=STACK_PR_FILES)

self.comment(github, ci_builds=[ci_build(meta_data=meta_data)])

(build,) = self.ci.created_builds
self.assertEqual(build["env"]["ROCM_PROMOTE_IMAGES"], "s@d1 w@d2 r@d3 t@d4")

def test_cache_skips_children_the_source_did_not_build_on_its_stack(self) -> None:
meta_data = {
**STACK_BUILT,
"rocm-rocm-runtime-built": "none",
"rocm-rocm-test-deps-built": "none",
}
github = FakeGitHub(files=STACK_PR_FILES)

self.comment(github, ci_builds=[ci_build(meta_data=meta_data)])

(build,) = self.ci.created_builds
self.assertEqual(build["env"]["ROCM_PROMOTE_IMAGES"], "s@d1 w@d2")

def test_cache_finds_the_source_on_a_later_page(self) -> None:
reused = ci_build(meta_data={"rocm-rocm-stack": "s@d3"})
github = FakeGitHub(files=STACK_PR_FILES)

amd = self.comment(github, ci_pages=[[reused] * 100, [ci_build()], []])
self.comment(github, ci_pages=[[reused] * 100, [ci_build()], []])

self.assertEqual(len(amd.created_builds), 1)
self.assertEqual(len(self.ci.created_builds), 1)
self.assertEqual(
[(r["branch"], r["page"]) for r in self.ci.list_requests],
[("contributor:feature", 1), ("contributor:feature", 2), ("feature", 1)],
Expand All @@ -1755,9 +1785,9 @@ def test_cache_lists_builds_since_the_pr_opened_off_the_base_branch(
pr["head"]["ref"] = "main"
github = FakeGitHub(files=STACK_PR_FILES, pr=pr)

amd = self.comment(github, ci_pages=[[ci_build()]])
self.comment(github, ci_pages=[[ci_build()]])

self.assertEqual(len(amd.created_builds), 1)
self.assertEqual(len(self.ci.created_builds), 1)
self.assertEqual(
[(r["branch"], r["created_from"]) for r in self.ci.list_requests],
[("contributor:main", "2026-01-01T00:00:00Z")],
Expand All @@ -1772,17 +1802,17 @@ def test_build_pagination_is_bounded(self) -> None:
def test_cache_finds_a_source_created_under_the_bare_ref(self) -> None:
github = FakeGitHub(files=STACK_PR_FILES)

amd = self.comment(github, ci_pages=[[], [ci_build()]])
self.comment(github, ci_pages=[[], [ci_build()]])

(build,) = amd.created_builds
(build,) = self.ci.created_builds
self.assertEqual(build["env"]["ROCM_PROMOTE_SHA"], "fedcba9876543210")

def test_nocache_records_the_decision_without_a_build(self) -> None:
github = FakeGitHub(files=STACK_PR_FILES)

amd = self.comment(github, COMMAND_ROCM_NOCACHE)

self.assertEqual(amd.created_builds, [])
self.assertEqual(amd.created_builds + self.ci.created_builds, [])
self.assertIn(ROCM_CACHE_MARKER, github.comments[0])
self.assertIn("@reviewer chose not to cache", github.comments[0])

Expand Down Expand Up @@ -1819,7 +1849,7 @@ def test_refusals_leave_the_gate_failing(self) -> None:
with self.subTest(text=text):
amd = self.comment(github, ci_builds=builds, actor=actor)

self.assertEqual(amd.created_builds, [])
self.assertEqual(amd.created_builds + self.ci.created_builds, [])
self.assertIn(text, github.comments[0])
self.assertNotIn(ROCM_CACHE_MARKER, github.comments[0])

Expand All @@ -1828,7 +1858,7 @@ def test_prs_that_keep_the_base_image_have_nothing_to_do(self) -> None:

amd = self.comment(github)

self.assertEqual(amd.created_builds, [])
self.assertEqual(amd.created_builds + self.ci.created_builds, [])
self.assertIn("nothing to do", github.comments[0])
self.assertNotIn(ROCM_CACHE_MARKER, github.comments[0])

Expand Down
7 changes: 4 additions & 3 deletions docs/contributing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -372,9 +372,10 @@ review process:
rerun CI without `--allow-stale` on the latest PR commit.
- A PR targeting `main` that changes `docker/Dockerfile.rocm_base` or
`tools/vllm-rocm/therock_wheels.py` fails `pre-commit` until a maintainer
with write access comments `/ci rocm cache`, which copies the ROCm base image
the PR's CI built into `rocm/vllm-ci` so `main` reuses it after the merge, or
`/ci rocm nocache` to skip that. Then re-run the failed `pre-commit` check.
with write access comments `/ci rocm cache`, which starts a CI build that only
copies the ROCm stack, runtime and test-deps images the PR's CI built into
`rocm/vllm-ci`, so `main` reuses them after the merge, or `/ci rocm nocache`
to skip that. Then re-run the failed `pre-commit` check.
- Retrying or cancelling builds does not check branch freshness.
- Apart from that merge, these commands do not modify your branch or enforce
merge requirements. Direct Buildkite launches are outside this workflow.
Expand Down
Loading
Loading