Path B: route whole-file deletes away from the offline window, correct the reclaim rationale

[Alex Burns (AlBurns-MSFT)]

Path B currently sends every thin-provisioned reclaim down one heavyweight route: merge checkpoints, take all VMs on the volume offline, run slab consolidation, wait, bring VMs back. CSS raised this from a live case where the customer had deleted a number of VMs and simply wanted the space back, and asked whether all of that is really needed, and whether the real requirement is just about active file handles on the CSV.

The answer turned out to be split. The heavyweight route is right for the case it was written for, but it was being applied to a case that does not need it at all.

The main change: not every reclaim needs consolidation

Two different mechanisms return capacity to the pool, and only one needs an offline window.

When a whole file is deleted, the slabs it occupied become entirely free and ReFS returns them to the pool on its own, with no Optimize-Volume at all. The Azure Local thin provisioning FAQ documents this: "Will space be given back to the pool immediately after files are deleted? No. This is a gradual process that can take 15 minutes or so after the files are deleted. If there are many workloads running on the system, it may take longer for all of the space to be returned to the pool." That last clause is the direct answer to the CSS question: running workloads slow reclamation down, they do not block it.

Slab consolidation exists for the other case, interior fragmentation, where data was deleted from inside a VHDX and live data still occupies part of each slab, so no whole slab ever frees.

The customer in the CSS case had deleted whole VMs, which is the first case. They did not need the offline window, and Path B gave them no way to discover that. The guide already said as much in the pool-fill escalation ladder ("Audit and prune... the reclaimed space returns to the pool gradually"), but that note is a long way from Path B and cross-links into it. This PR promotes it into an explicit no-downtime pre-branch at the top of Path B: confirm the deletes, wait, re-measure, and only fall through to consolidation if the pool is still above threshold and real interior free space remains.

The offline window itself is correct, and stays

CSS's other point was that the VM shutdown may be unnecessary because consolidation is a low-priority background job. Half of that holds up and half does not.

The priority part is right, and the guide now says so: Optimize-Volume documents -NormalPriority as "By default, the priority is low", matching defrag /h. But that governs scheduling priority, not total cost. Pool physical disks are shared by every volume in the pool, so hours of relocation I/O on a multi-terabyte volume can still be felt by workloads on other volumes. The guide now makes that distinction explicitly.

The shutdown requirement, however, is real. I initially rewrote the rationale, on the grounds that no public documentation states a handle prerequisite and that FSCTL_FILE_LEVEL_TRIM reclaims ranges inside open virtual-disk files. That was wrong, and the second commit here reverts it. Tracking down the Microsoft file-system engineering guidance this section was originally based on confirms the mechanism as the guide had it: pinned-unmovable slabs are mostly caused by applications holding files pinned, slab consolidation cannot process files that are actively in use, and the recommended sequence is to stop workloads on the volume, consolidate, then let the ReFS unmap work items reclaim. The same guidance notes that lab tests tend to succeed precisely because workloads were stopped, while production workloads keep slabs pinned and reduce how much a pass recovers. It independently confirms the existing advice not to use -ReTrim on thin-provisioned ReFS.

So the step is unchanged and the rationale is now stated in terms of files in use being unmovable, rather than the vaguer "handles are released". The Suspend-VM note is also adjusted: pausing leaves the virtual disk files open and the guest resident, so it is not a reliable substitute for a shutdown.

Other changes

• Do not move VM disks between volumes to relieve pressure. This was the second thing CSS asked for. Scoped three ways relative to their wording: it applies to Arc-managed VMs, because Move-VMStorage plus updating the cluster resource is still supported for traditional clustered Hyper-V VMs and a blanket prohibition would be wrong for that audience; it is not scoped to "beforehand", since storage live migration is unsupported at any time; and it names the supported alternative, choosing a storage path on the target volume at creation, because there is no supported in-place move and the operator otherwise has nowhere to go. Learn lists storage live migration among operations that "can lead to Azure Local VMs becoming unmanageable from the Azure portal", and the failure mode is that the storage path resource keeps pointing at the old volume.
• A stretched-cluster caveat was added and then removed. Learn's thin provisioning article says TRIM is disabled for stretched clusters, but stretched clusters are not a supported Azure Local 23H2+ configuration and that concept page still sits on the legacy azure-stack/hci path, so the sentence is stale text carried onto a renamed page. Repeating it would have implied a deployment shape the reader cannot have and sent them to check a condition that cannot be true. Nothing about stretched clusters ships in this change.
• DisableDeleteNotify precondition check, a documented reason a reclaim appears to do nothing.
• The optional checkpoint merge moved out of the numbered sequence, so it stops reading as a required first step. It is preparation, not part of the window.
• Updated the At a glance downtime row, the quick triage branch, and the options table, which all described Path B as unconditionally requiring a window.
• Noted in the closing troubleshooting note that some slabs report pinned unmovable even on a quiesced volume, so a partial reclaim is not by itself a failure.

Scope and validation

The offline window for the consolidation path is not loosened. Whether consolidation can run safely with live VMs is a separate question, and worth being honest that the physical validation behind this guide ran on a scratch volume with no VMs on it, so it never tested that. A controlled run that separates an open file from an actively writing one, and that measures the effect on other volumes in the pool rather than only the one being consolidated, has not been done. The experiment design and its required control arms are written up in the internal validation spec so it can be picked up, but to be clear about status, it is not scheduled and no cluster is reserved for it. It is also lower priority than it first appeared: the owning team's guidance already answers whether the window is needed, so what remains is quantifying how much yield is lost under live workload, not deciding the step.

The no-downtime pre-branch is sourced from Microsoft documentation rather than a new lab run, so the metadata block is unchanged and still reflects the last hardware validation.

tsg-forge --lint grade A: fences balanced, all relative links and in-page anchors resolve, no bare drive-root deletes, prose style clean.

[Alex Burns (AlBurns-MSFT)]

Reviewed this against the base rather than just reading the diff, checked every external citation against its source, and traced the renumbering and the routing end to end. Four things needed fixing and are now pushed.

1. Deleting a VM does not delete its disks. This was the important one. The pre-branch routed on "you deleted whole VMs or files" and asked only that the deletions be confirmed complete. Remove-VM deletes the virtual machine's configuration file but does not delete any virtual hard drives, so a customer who removed VMs can still be holding every .vhdx on the volume. That customer would take the no-downtime branch, wait fifteen minutes, reclaim nothing, and reasonably conclude the guide is wrong, when the actual fix is to find and remove the orphaned disks. The branch now asks for confirmation that the disk files themselves are gone and gives a command to enumerate them.

2. The DisableDeleteNotify check described a single value. The command returns one line per file system, and a line can read is not currently set, which is neither 0 nor 1. Azure Local CSVs are ReFS, so the ReFS line is the one that governs, and the check did not say so. Someone reading the NTFS line had confirmed nothing about the CSV. It now shows the real output shape, names the file system to read, covers all three states, and says to run it on the CSV owner node. The 0 and 1 mapping itself was correct and is unchanged. The precondition is also re-scoped to gate both paths rather than only the automatic one, since consolidation depends on the same ReFS unmap to return the emptied slabs.

3. An irreversible delete read as a safe action. The options table described the no-downtime branch as "delete, then wait" while the ownership gate listed that branch as always safe to run. Read together those made deleting VMs look like it sat under the safe-to-run exemption. The table now says confirm, wait, re-measure, and the gate is explicit that the branch asks for an irreversible delete and is not covered by the exemption.

4. A quote that read as a typo. The -NormalPriority sentence on Learn is missing a verb. Quoting it verbatim was faithful but looked like an error here, so it is trimmed to the clause that carries the meaning.

What held up: every Learn citation is verbatim and on the page it is attributed to, including the fifteen minute reclamation quote, the workloads clause, the stretched cluster TRIM sentence, and the storage live migration entry in the unsupported operations list. Scoping the disk-move caution to Arc VMs is correct, Move-VMStorage with the cluster resource updated remains supported for traditional clustered Hyper-V VMs. The renumbering is clean, every step reference resolves under the new one through five numbering with no surviving step six, all in-page anchors resolve, and both new reference-style definitions are defined once with every use resolving. The health fault strings match the emitted values with no suffix variants. Doc lint returns the same finding set as the base.

Still unverified: nobody has run these commands on a 23H2 node in this pass, so the exact fsutil output on a deployed cluster and the Get-Volume -FilePath versus -Path behavior are documented rather than captured. One read-only pass on hardware would settle both.

[Alex Burns (AlBurns-MSFT)]

John Neemes (@1008covingtonlane) could you give this a review please?

[John Neemes (1008covingtonlane)]

Thanks for the rework. The split between whole-file deletes and interior fragmentation is a real improvement, and reverting the handle-based rationale back to "files in use can't be relocated" is the right call. I read the whole file at head and checked every cmdlet, parameter, and quoted doc against the source. The cmdlets and quotes all hold up: Optimize-Volume -SlabConsolidate and -NormalPriority, the "By default, the priority is low" line, Remove-VM "does not delete any virtual hard drives", the storage-live-migration "unmanageable from the Azure portal" wording, the storagecontainers resource type, and the thin-provisioning FAQ. The step renumbering after moving the checkpoint merge out is consistent, and the reference links and anchors all resolve. Two suggestions and one bookkeeping note.

1. The orphaned-disk delete branch could use a risk label and a stronger ownership check. [HIGH RISK]
The no-downtime pre-branch has the operator delete .vhdx and .avhdx files, and the ownership gate correctly calls that irreversible, but the step itself carries no risk label. Reversible consolidation is tagged [MEDIUM RISK] and a recoverable forced turn-off is [HIGH RISK], so as it stands the most destructive action in Path B is the only mutating step with no label. The larger gap is that Get-ChildItem ... -Include *.vhdx,*.avhdx,*.vhds only lists files; it does not establish "no owning VM". On a cluster-shared CSV that is a cluster-wide question, and for Arc VMs the attachment lives in Azure on the storage path resource, invisible to a host-side listing. Deleting an .avhdx directly also breaks the checkpoint chain. Suggestion: label the branch [HIGH RISK], build the in-use disk set across every node plus Arc and delete only what is absent from it, exclude templates and backup targets, and merge checkpoints through Hyper-V rather than deleting .avhdx files directly.

2. The DisableDeleteNotify precondition sits after the branch it is meant to gate. [MEDIUM RISK]
The "Preconditions for reclamation" block says it gates both paths, but as placed it comes after the no-downtime branch and that branch's "you are done" exit, so an operator who takes the whole-file-delete path never reaches it. If it really does gate both paths it probably wants to move above that branch.
Separately, and this is a question rather than a finding: the block reads ReFS DisableDeleteNotify = 1 as "explicitly disabled ... until that is reverted", while the example output just below it shows the ReFS line as "is not currently set". Those are different states, no explicit override versus an explicit one, and the fsutil docs say trim is disabled by default on ReFS v2, so the effective behavior and the reported string do not map one to one. I did not run a controlled delete, wait, and re-measure, so I am not claiming reclaim does or does not complete with it set. Your internal counterpart handles this distinction well, treating "is not currently set" as inconclusive rather than a fault. Worth mirroring that same wording here so an operator who sees a value other than the example does not get routed into changing a platform default.

3. The stretched-cluster bullet in the PR description no longer matches the change. [LOW RISK]
There is no mention of stretched clusters anywhere in the file, and having read your internal counterpart I can see that is deliberate rather than a dropped commit: the Learn TRIM sentence is stale text on a 23H2-named page, and stretched clusters are not a supported 23H2+ configuration. That reasoning is sound, and "verifying a quote exists is not the same as verifying the configuration exists" is a good way to put it. The only thing left is bookkeeping, since the PR description still lists the caveat under its notes as something added. Worth updating the description so it matches what shipped.

Happy to pair on any of these. Nothing else stood out.

[Alex Burns (AlBurns-MSFT)]

All three taken, and testing the second one turned up something worse than you flagged. Pushed.

1. Orphaned-disk delete branch. You are right on both counts and the second is the serious one. The step told the operator to delete files while giving them nothing to identify them with, since Get-ChildItem lists files and says nothing about ownership. It now builds the in-use set across every node from Get-ClusterNode rather than the one you happen to be signed in to, includes checkpoint disks via Get-VMSnapshot | Get-VMHardDiskDrive, and presents the result as candidates rather than as garbage. Added the three exclusions a listing cannot see: Arc-managed disks and images, which are Azure resources and can exist unattached and invisible to the host while still being live data, so the step now points at az stack-hci-vm disk list, image list and storagepath list; templates, golden images and backup targets; and checkpoint disks, with an explicit instruction to merge through Hyper-V because deleting an .avhdx breaks the differencing chain. Labeled HIGH RISK, with a warning that a file you cannot positively account for should be left alone and escalated rather than deleted to reclaim space.

2. The precondition. You were right to treat this as a question, and the answer is that the check was wrong, not just misplaced. I moved it above the branch as you suggested, then took your point about the reported string not mapping to the effective behavior and went and ran it on a lab cluster. On a healthy single-node 12.2610 system, pool Healthy and OK at 4.2% used with a thin volume at 191 GB footprint against 8690 GB of size, fsutil returns:

NTFS DisableDeleteNotify = 1  (Do not allow TRIM operations to be sent to storage devices)
ReFS DisableDeleteNotify = 1  (Do not allow TRIM operations to be sent to storage devices)

So a healthy cluster reports exactly the value my text called a fault. As written it would have failed on every healthy Azure Local system and sent operators to change a platform default to make the reading match my example, which is the harm you were pointing at. Learn documents 1 as the ReFS v2 default, and describes the setting as notifying the underlying storage device, which is device-level TRIM and not the Storage Spaces slab return this article depends on. Wrong layer, so I removed the check from the guide rather than rewording it, and recorded why in the internal counterpart so it does not get re-added from the same source.

The same lab pass re-confirmed the guidance that stayed, on a current build rather than only on 2607: Get-Volume -FilePath resolves the CSV to a CSVFS volume object, and Optimize-Volume -Path against the mount fails with exactly the documented "No MSFT_Volume objects found" error.

3. PR description. Updated. It now records that the stretched-cluster caveat was added and then removed, and why, rather than still listing it as shipped.

One honest limit on the testing: the cluster I had was single-node, so the cross-node aggregation in the new in-use query is syntax-validated and runs clean, but the multi-node behavior itself is not proven. Worth a second look from anyone with a multi-node cluster handy. Thanks for the review, the disk-ownership point in particular was the right thing to catch.

[John Neemes (1008covingtonlane)]

Both fixes hold up, and the second is a better outcome than what I asked for. I could not prove the TRIM reading myself, which is why I raised it as a question rather than a finding, so running it on a live cluster and getting ReFS DisableDeleteNotify = 1 on a healthy pool is what settles it. Removing the check rather than rewording it is the right call, since it reads at the device layer and not at the slab return layer this article depends on.

Two things the rework opened up.

1. The removal left the precondition behind in three places. The fsutil block was the only thing in the guide that told an operator how to evaluate delete notification, and it is gone, but three spots still ask them to:

• the note on why a consolidation pass can return nothing ("ReFS delete notification has been explicitly disabled")
• the support case criteria just below it ("ReFS delete notification is not explicitly disabled, all workloads were offline")
• the escalation bullet ("Path B completed with every precondition met (... ReFS delete notification not explicitly disabled ...)")

The escalation one is the one that bites. The operator is told to confirm that precondition before opening a case, has no method left in the guide, finds the obvious command themselves, reads 1 on a healthy cluster, concludes it is disabled, and does not open the case. That is the same harm you just removed, arriving through the escalation path instead of the remediation path. By your own reasoning it is the wrong layer, so I would drop the clause from all three rather than reinstate a way to check it. [LOW RISK]

2. The cross node in use query under reports when a node does not answer, and it fails in the direction that deletes data. Get-VM -ComputerName inside the ForEach-Object has no error handling, so a node that is up but not answering (WinRM, RPC, draining) emits a non terminating error and the loop continues with a short $inUse. I ran the exact shape from the guide with one of two nodes failing: the pipeline completes, and that node's live disks come out the far side as delete candidates. The comment above the second block reads "Files on the volume that nothing on the cluster references", which is then untrue, and nothing in the output marks it as partial.

The error does print, but it prints in the middle of a loop over every node, and the candidate list that follows carries no sign it is incomplete. On a step labeled HIGH RISK that ends in deleting files, I would make it fail closed: collect under -ErrorAction Stop in a try/catch, or assert that the number of nodes that answered matches Get-ClusterNode before trusting the list, and tell the operator to stop if it does not. [MEDIUM RISK]

That is also the concrete form of the multi node gap you flagged. The rest of the pipeline is right: Get-VMSnapshot | Get-VMHardDiskDrive is a documented pattern (Example 4 on the Get-VMHardDiskDrive reference page), and -notin is case insensitive, so path casing between the VM config and the file listing will not produce a false candidate. The unanswered node is the only gap.

The ownership rework itself is a clear improvement, particularly calling out unattached Arc disks as invisible to a host side listing, and routing .avhdx through checkpoint merge instead of deletion.