fix(runners): pin Windows WSL to 2.4.13 and make WSL provisioning non-hanging - #1124
Merged
shashank-boyapally merged 4 commits intoSep 10, 2026
Conversation
…date drift Windows finch e2e fails at 'finch vm init' with "could not attach persistent disk: ... WSL_E_WSL2_NEEDED" because the runners drifted from the pinned WSL 2.4.13 / kernel 5.15.167.4 onto WSL 2.7.13 / kernel 6.18.33.2 - the kernel that regresses the finch disk-attach e2e. WSL ships as a Microsoft Store app, and the Store auto-updated it past the 2.4.13 MSI pin (verified on a live runner: Get-AppxPackage reported 2.7.13.0, wsl --version reported kernel 6.18.33.2). VirtualMachinePlatform was enabled and the hypervisor present, so the platform was fine - only the version pin failed to hold. - Disable Microsoft Store automatic app updates in first-boot (HKLM\...\WindowsStore\AutoDownload = 2) so the WSL pin holds. - In the boot-time WSL install: shut down WSL and remove any non-2.4.13 WSL Appx (and its provisioned package) before installing the pinned MSI, since an older MSI will not install over a newer Store app. - After install, verify 'wsl --version' is 2.4.13 and log loudly on mismatch so a drifted runner is visible in StartupScript.log / CloudWatch instead of silently failing finch e2e. Existing Windows runners must be recycled to re-provision on the pin. Signed-off-by: ShashankReddy Boyapally <shasboy@amazon.com>
… WSL2 distro The boot-time WSL pin hung and left runners unable to run finch e2e: 1. msiexec hang: installing the pinned 2.4.13 MSI right after removing the newer Store WSL Appx blocks on the Windows Installer lock indefinitely (Start-Process -Wait never returns), stranding the instance in Standby before Exit-ASStandby. Now installs with a 180s timeout + kill + retry (up to 4x), mirroring the manual recovery that worked (kill stuck msiexec, reinstall). 2. Missing WSL2 distro: finch vm init attaches its disk via 'wsl --mount --bare --vhd', which fails with WSL_E_WSL2_NEEDED unless a WSL2 distro is registered (the utility VM only exists then). The previous 'wsl --install Ubuntu' ran AFTER the hanging msiexec so it never executed. Now installs Ubuntu (--no-launch, headless) before Exit-ASStandby and verifies via the per-user Lxss registry key. startup.ps1 runs as Administrator - the same account finch e2e runs under - so the distro is registered for the right user. Also sets 'wsl --set-default-version 2' explicitly and logs the resulting 'wsl --version' so drift/failures are visible in StartupScript.log / CloudWatch. Store auto-update remains disabled (first-boot) so the pin holds. Existing drifted runners must be recycled to pick up this user-data. Signed-off-by: ShashankReddy Boyapally <shasboy@amazon.com>
…ve download unbounded Signed-off-by: ShashankReddy Boyapally <shasboy@amazon.com>
…n-wsl-2.4.13 # Conflicts: # scripts/windows-runner-user-data.yaml
smandhada16
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Windows finch e2e (
finch vm init) fails at disk attach with:Two root causes, both traced on live runners:
startup.ps1hang + missing distro. Installing the pinned 2.4.13 MSI right after removing the newer Store Appx blocks on the Windows Installer lock indefinitely (Start-Process -Waitnever returns), stranding the instance in Standby. Because that step hung, the subsequentwsl --install Ubuntunever ran — so no WSL2 distro was ever registered, andwsl --mount --bare --vhd(how finch attaches its disk,pkg/disk/disk_windows.go) fails withWSL_E_WSL2_NEEDEDsince the WSL2 utility VM only exists when a distro is present.VirtualMachinePlatformwas enabled and the hypervisor present — the platform was fine; only the version + distro were wrong.Fix (
scripts/windows-runner-user-data.yaml)WindowsStore\AutoDownload = 2) so the WSL pin holds.startup.ps1:-Wait, so it can never hang and strand the runner. (Download left unbounded — a slow-but-completing download shouldn't be interrupted.)wsl --set-default-version 2.--no-launch, headless) beforeExit-ASStandbyand verify via the per-userLxssregistry key.startup.ps1runs as Administrator — the same account finch e2e runs under — so the distro registers for the right user.wsl --versionandWrite-Erroron any mismatch/failure so drift is visible inStartupScript.log/ CloudWatch instead of silently failing e2e.Rollout
Existing runners are drifted and must be recycled to pick up this user-data; new instances then self-provision (pinned WSL 2.4.13 + Ubuntu distro + working WSL2).
Testing
npm run build+ unit tests pass; YAML validated. On-box behavior (kill+retry recovery, WSL 2.4.13 pin) was reproduced manually on the affected runners. Note: the headlesswsl --install -d Ubuntu --no-launchbeing sufficient forwsl --mountis verified by the added registry check +wsl --versionlogging on first re-provisioned runner.