[scanner] fix: repair Build and Deploy KC workflow#20577
[scanner] fix: repair Build and Deploy KC workflow#20577kubestellar-hive[bot] wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
✅ Deploy Preview for kubestellarconsole canceled.
|
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
👋 Hey @kubestellar-hive[bot] — thanks for opening this PR!
This is an automated message. |
When the kc Helm release is in a failed state with no rollback history (e.g. first-install failure), helm rollback exits non-zero. The prior code hard-exited, permanently blocking all future deploys until manual cleanup. Add a fallback: if rollback fails, uninstall the stuck release so the next Deploy with Helm step can do a fresh install. Fixes #20548 Signed-off-by: kubestellar-hive[bot] <280983584+kubestellar-hive[bot]@users.noreply.github.com>
dcf52b5 to
8f04dc8
Compare
|
Closing as redundant: main already contains this exact fallback (helm uninstall when rollback fails, with a 5m timeout) in the deploy-vllm-d job, and the linked issue #20548 was closed before this PR was opened. Thanks anyway, scanner. 🐝 |
Fixes #20548
Root cause
The
deploy-vllm-djob failed at the "Recover stuck Helm release" step. The Helm releasekcwas in afailedstate (from a prior deploy failure), so the step attemptedhelm rollback. The rollback itself failed — likely because there was no prior revision to roll back to (this happens when the very first install fails, leaving no rollback history).The prior code hard-exited on rollback failure, permanently blocking all future deploys of the
deploy-vllm-djob until manual Helm cleanup was performed on the cluster.Fix
Added a fallback: when
helm rollbackfails for a stuckfailedrelease, attempthelm uninstallto clear the stuck state. The subsequent "Deploy with Helm" step (which useshelm upgrade --install) will then perform a clean fresh install.Changed step in
deploy-vllm-d:helm uninstallinstead of immediately exitingWhat was NOT changed
deploy-pok-prodhas an identical "Recover stuck Helm release" step with the same potential issue, but it succeeded in this run and is left for a follow-up to keep this PR minimal.