fix(concourse): stop replacing spot workers on rebalance recommendations

[Ardiea]

What are the relevant tickets?

N/A

Description (What does it do?)

Concourse spot workers were marking themselves Unhealthy on every EC2 rebalance recommendation, so the ASG replaced them. That was the largest cause of worker churn. Over the six weeks to 2026-10-16, QA OCW and the three Production pools each lost a median of 18–28 workers per weekday this way (peaks of 40–57), 1.6–1.8× their real spot interruption count. Each replacement threw away running builds and the baggageclaim cache.

concourse-worker-spot-watch now logs the first rebalance recommendation and does nothing else. Draining on the 2-minute spot interruption notice is unchanged.

Implementation details

• The ASG activity history says "taken out of service in response to a user health-check". CloudTrail shows every SetInstanceHealth call coming from the worker's own instance role, about itself. All of the instances were spot, across every QA and Production pool.
• In a one-day sample, 11 of 50 calls came 18–183s after launch and were rejected with The age of the instance (N seconds) is less than the configured grace period of 300 seconds. On that path the script fell back to concourse-worker-drain, retiring the Concourse worker process on a fresh instance the ASG still considered healthy. That path is gone too.
• The old handle_rebalance ended in sleep infinity, so after a rebalance recommendation the script stopped polling for the interruption notice. The loop now keeps watching for it.
• concourse-worker-preflight still calls set-instance-health when its network checks fail, so the IAM permission stays.
• AWS describes a recommendation as a signal that the instance "is at elevated risk of interruption" (docs), not as an interruption notice. Builds on a worker that is actually reclaimed now get the 2-minute drain instead of up to 10 minutes, and are retried via enable_rerun_when_worker_disappears.
• ELB health-check failures barely touch the OCW pools (2 in six weeks on QA, 0 on Production). Other pools do have real ELB failures (CI OCW 793, QA infra 120, Production generic 41 over the same period). That's a separate problem this PR does not touch.

How can this be tested?

Only bash -n, shellcheck and prek have been run; this can't be checked before merge. After the Concourse worker AMI is rebuilt and rolled out:

• aws autoscaling describe-scaling-activities --auto-scaling-group-name <worker asg>: "user health-check" terminations should drop to near zero, leaving mostly "maximum instance lifetime" and "EC2 health check indicating it has been terminated" (spot reclaims).
• aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=SetInstanceHealth: calls from concourse-instance-role-worker-* should only appear for preflight failures.

Additional context

Worker-side journald units (spot-watch, preflight, lifecycle-hook) never reach Loki, which is why this was diagnosed from CloudTrail. That's tracked separately.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

The focused change preserves interruption draining, and mocked execution confirmed continued polling after rebalance recommendations.

0 open findings

What changed in this PR

Reduces Concourse spot-worker churn by treating EC2 rebalance recommendations as advisory rather than triggering replacement.

Changes:

• Logs the first rebalance recommendation without replacing or draining the worker.
• Continues polling for interruption notices, preserving the existing drain behavior.
• Documents the operational rationale.

File	Description
src/​bilder/​images/​concourse/​files/​concourse-worker-spot-watch	Removes rebalance-triggered replacement and adds one-time logging while preserving interruption handling.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.