Add Storage TSG: repair jobs restart and never complete on a failing high-latency disk#327
Open
1008covingtonlane wants to merge 4 commits into
Open
Conversation
New troubleshooting guide for S2D storage repair/regeneration jobs that reset and never complete when a single disk is failing with high latency and IO errors. Covers detection surfaces (PowerShell, event logs, cluster log, Azure portal, FCM/WAC), the free-space and reserve check before retiring, the retire-and-rebuild resolution, pre-retire gotchas, and prevention. Adds the guide to the Storage index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Storage TSG for diagnosing and resolving repair jobs blocked by a failing high-latency disk.
Changes:
- Documents symptoms, diagnostics, capacity checks, and disk replacement.
- Adds the guide to the Storage index.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md |
Adds the troubleshooting procedure. |
TSG/Storage/README.md |
Links the new guide. |
Resolves the four Copilot PR-review findings: (1) the pre-retire one-fault-domain check now also filters OperationalStatus (Abnormal Latency / Lost Communication) and surfaces the node, so a second stale-Healthy failing disk is not missed; (2) the Step 3 free-space query excludes the retiring disk, shows per-node free capacity, and no longer truncates to five rows; (3) the worked example now distinguishes sufficient evacuation headroom from full reserve (not restored until the replacement drive is added); (4) Step 6 uses Remove-PhysicalDisk -PhysicalDisks (Get-PhysicalDisk -UniqueId ...) because -UniqueId is not a valid parameter. Also adds usability sections from a persona review: a TL;DR fastest-path, an Impact and ownership summary, a Glossary, a Scope note distinguishing RDMA/network issues, an explicit workload-impact note, a When to escalate section, and OEM/hardware framing for the physical swap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd
Collaborator
Author
|
Thanks for the review. Addressed all four findings in 9d63f9c:
|
Replace the "TL;DR" callout with an "## Overview" section that leads with plain-language what, why, and impact and keeps the fast at-a-glance path. Correct the resiliency description: name the Azure Local default (three-way mirror on three or more nodes, two-way mirror on two-node) and cover the other Storage Spaces Direct schemes (parity, dual parity, nested resiliency, and mirror-accelerated parity), plus Simple with no redundancy. Explain that a slow, high-latency but SMART-healthy disk is not auto-failed under any scheme, and that retire then rebuild restores redundancy on any resilient volume while a Simple volume must be restored from backup. Confirm ResiliencySettingName before acting. Update the Glossary and a Symptoms bullet to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd
Reviewer consistency nit: Step 6 named the sibling by description while Related Issues uses exact paths. Point Step 6 at TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md the same way. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new Storage troubleshooting guide,
TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md, and links it from the Storage index.Why
There is no existing TSG for the common Storage Spaces Direct failure mode where
*-Repair/*-Regenerationstorage jobs reset toward 0% and never complete, with a volume stuck atOperationalStatus = No Redundancy. The usual cause is a single physical disk failing with very high latency and a high IO error rate while still reportingHealthStatus = Healthy(OperationalStatus = "OK, Abnormal Latency"). The same disk can also trip aDPC_WATCHDOG_VIOLATION(bugcheck 0x133) on its host node under write load, so a node crash and the stuck repair often share one cause. The condition is frequently misread as a pool-capacity problem.What the guide covers
PhysicalDisk.HighLatency.Outlier.AverageIO,PhysicalDisk.HighErrorCount.Outlier.AverageIO,VirtualDisks.NoRedundancy,VirtualDisks.LastCopy).disk153; Storage Spaces 203/205/207/209/302/304/305/312), cluster log, Azure portal, Failover Cluster Manager and Windows Admin Center.Set-PhysicalDisk -Usage Retired, monitor the rebuild jobs, thenRemove-PhysicalDisk, with risk labels and a last-copy caveat.All commands, fault strings, and event IDs are from an observed case and are scrubbed of environment-specific identifiers.