Skip to content

[scanner] fix: repair Build and Deploy KC workflow#20577

Closed
kubestellar-hive[bot] wants to merge 1 commit into
mainfrom
scanner/fix-20548
Closed

[scanner] fix: repair Build and Deploy KC workflow#20577
kubestellar-hive[bot] wants to merge 1 commit into
mainfrom
scanner/fix-20548

Conversation

@kubestellar-hive

Copy link
Copy Markdown
Contributor

Fixes #20548

Root cause

The deploy-vllm-d job failed at the "Recover stuck Helm release" step. The Helm release kc was in a failed state (from a prior deploy failure), so the step attempted helm rollback. The rollback itself failed — likely because there was no prior revision to roll back to (this happens when the very first install fails, leaving no rollback history).

The prior code hard-exited on rollback failure, permanently blocking all future deploys of the deploy-vllm-d job until manual Helm cleanup was performed on the cluster.

Fix

Added a fallback: when helm rollback fails for a stuck failed release, attempt helm uninstall to clear the stuck state. The subsequent "Deploy with Helm" step (which uses helm upgrade --install) will then perform a clean fresh install.

Changed step in deploy-vllm-d:

  • If rollback fails → try helm uninstall instead of immediately exiting
  • If uninstall also fails → exit with error (manual intervention still needed, but rollback failure alone no longer blocks all future deploys)

What was NOT changed

  • deploy-pok-prod has an identical "Recover stuck Helm release" step with the same potential issue, but it succeeded in this run and is left for a follow-up to keep this PR minimal.

Copilot AI review requested due to automatic review settings July 8, 2026 09:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot can't review bot-authored pull requests automatically. A user with Copilot access can request a review manually.

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: no Indicates the PR's author has not signed the DCO. label Jul 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-hive kubestellar-hive Bot added the hold Blocked — do not touch label Jul 8, 2026
@netlify

netlify Bot commented Jul 8, 2026

Copy link
Copy Markdown

Deploy Preview for kubestellarconsole canceled.

Name Link
🔨 Latest commit 8f04dc8
🔍 Latest deploy log https://app.netlify.com/projects/kubestellarconsole/deploys/6a54f309543a64000862f11e

@kubestellar-prow kubestellar-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubestellar-prow kubestellar-prow Bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jul 8, 2026
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

👋 Hey @kubestellar-hive[bot] — thanks for opening this PR!

🤖 This project is developed exclusively using AI coding assistants.

Please do not attempt to code anything for this project manually.
All contributions should be authored using an AI coding tool such as:

This ensures consistency in code style, architecture patterns, test coverage,
and commit quality across the entire codebase.


This is an automated message.

When the kc Helm release is in a failed state with no rollback
history (e.g. first-install failure), helm rollback exits non-zero.
The prior code hard-exited, permanently blocking all future deploys
until manual cleanup. Add a fallback: if rollback fails, uninstall
the stuck release so the next Deploy with Helm step can do a fresh
install.

Fixes #20548
Signed-off-by: kubestellar-hive[bot] <280983584+kubestellar-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. and removed dco-signoff: no Indicates the PR's author has not signed the DCO. labels Jul 13, 2026
@clubanderson

Copy link
Copy Markdown
Collaborator

Closing as redundant: main already contains this exact fallback (helm uninstall when rollback fails, with a 5m timeout) in the deploy-vllm-d job, and the linked issue #20548 was closed before this PR was opened. Thanks anyway, scanner. 🐝

@clubanderson clubanderson deleted the scanner/fix-20548 branch July 13, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. hold Blocked — do not touch needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. tier/3-restricted

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workflow failure: Build and Deploy KC

2 participants