Fix flaky FileSystemWatcher_Directory_Delete_MultipleFilters due to Windows NTFS pending-delete race by Copilot · Pull Request #125915 · dotnet/runtime

Copilot · 2026-03-22T04:41:14Z

Two race conditions in the FSW test retry infrastructure caused intermittent DirectoryNotFoundException on Windows x86/x64 CI.

Root causes:

NTFS pending-delete race: RemoveDirectory marks a directory for deletion but doesn't remove it until all handles close. The cleanup lambda called directoryOne.Create() while the path was still in pending-delete state — Windows reported it as existing, so CreateDirectory silently no-oped. Once the pending delete completed, the directory was gone, and the next retry's Delete() threw DirectoryNotFoundException.
RecreateWatcher handle leak: Each retry created a new FileSystemWatcher without disposing the previous one. Accumulated watchers holding handles extended the pending-delete window.

Fixes:

ExpectEvent (test utility): Wrap the retry loop in try/finally to ensure any FileSystemWatcher created by RecreateWatcher is always disposed — even if ExecuteAndVerifyEvents or the cleanup() lambda throws. Intermediate watchers are disposed between retries; the final recreated watcher is disposed in the finally block.
FileSystemWatcher_Directory_Delete_MultipleFilters and FileSystemWatcher_File_Delete_MultipleFilters: Add Assert.True(SpinWait.SpinUntil(() => !Directory/File.Exists(...), 2000), "Timed out waiting for '...' to be deleted.") before Create() in cleanup lambdas, ensuring the path is fully gone before recreating it and surfacing a clear diagnostic if the 2s timeout is exceeded:

// Before
cleanup: () => directoryOne.Create()

// After
cleanup: () =>
{
    Assert.True(SpinWait.SpinUntil(() => !Directory.Exists(directoryOne.FullName), 2000),
        $"Timed out waiting for '{directoryOne.FullName}' to be deleted.");
    directoryOne.Create();
}

Original prompt

This section details on the original issue you should resolve

<issue_title>Could not find a part of the path in System.IO.Tests.FileSystemWatcherTests.FileSystemWatcher_Directory_Delete_MultipleFilters</issue_title>
<issue_description>### Error Blob

{
  "ErrorMessage": "at System.IO.Tests.FileSystemWatcherTests.FileSystemWatcher_Directory_Delete_MultipleFilters",
  "BuildRetry": true,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Found in unrelated 7.0 PR: [release/7.0-staging] Update SqlClient and ship Microsoft.Windows.Compatibility #97883
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=551129
Queue: Libraries Test Run checked coreclr windows x86 Debug
Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=551129&view=logs&j=215a953c-1e4b-5cc0-95e2-5484aabbe659&t=312e0115-80c6-5d8d-a731-029b3098bcf6
Log file: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-97883-merge-de8975dcd8f044af9b/System.IO.FileSystem.Watcher.Tests/1/console.87568cb0.log?helixlogtype=result
Output:

===========================================================================================================

C:\h\w\C0550A4F\w\A85A0959\e>"C:\h\w\C0550A4F\p\dotnet.exe" exec --runtimeconfig System.IO.FileSystem.Watcher.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Watcher.Tests.deps.json xunit.console.dll System.IO.FileSystem.Watcher.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.IO.FileSystem.Watcher.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.FileSystem.Watcher.Tests (found 174 of 209 test cases)
  Starting:    System.IO.FileSystem.Watcher.Tests (parallel test collections = on, max threads = 4)
    System.IO.Tests.FileSystemWatcherTests.FileSystemWatcher_Directory_Delete_MultipleFilters [FAIL]
      System.IO.DirectoryNotFoundException : Could not find a part of the path 'C:\h\w\C0550A4F\t\#FileSystemWatcherTests_eq2ujqlt.lfo\FileSystemWatcher_Directory_Delete_MultipleFilters_1009_7eam6lig\FileSystemWatcher_Directory_Delete_MultipleFilters_1010_cn2nu7w1'.
      Stack Trace:
        /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs(436,0): at System.IO.FileSystem.RemoveDirectoryInternal(String fullPath, Boolean topLevel, Boolean allowDirectoryNotEmpty)
        /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs(261,0): at System.IO.FileSystem.RemoveDirectory(String fullPath, Boolean recursive)
        /_/src/libraries/System.Private.CoreLib/src/System/IO/DirectoryInfo.cs(212,0): at System.IO.DirectoryInfo.Delete(Boolean recursive)
        /_/src/libraries/System.Private.CoreLib/src/System/IO/DirectoryInfo.cs(208,0): at System.IO.DirectoryInfo.Delete()
        /_/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.unit.cs(1019,0): at System.IO.Tests.FileSystemWatcherTests.<>c__DisplayClass61_0.<FileSystemWatcher_Directory_Delete_MultipleFilters>b__0()
        /_/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs(309,0): at System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Boolean assertExpected, String[] expectedPaths, Int32 timeout)
        /_/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs(198,0): at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String[] expectedPaths, Int32 attempts, Int32 timeout)
        /_/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs(168,0): at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String expectedPath, Int32 attempts, Int32 timeout)
        /_/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.unit.cs(1019,0): at System.IO.Tests.FileSystemWatcherTests.FileSystemWatcher_Directory_Delete_MultipleFilters()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs(81,0): at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
  Finished:    System.IO.FileSystem.Watcher.Tests
=== TEST EXECUTION SUMMARY ===
   System.IO.FileSystem.Watcher.Tests  Total: 327, Errors: 0, Failed: 1, Skipped: 0, Time: 10.115s
----- end Fri 02/02/2024 18:18:47.75 ----- exit code 1 ----------------------------------------------------------

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/publ...

Fixes Could not find a part of the path in System.IO.Tests.FileSystemWatcherTests.FileSystemWatcher_Directory_Delete_MultipleFilters #98193

📍 Connect Copilot coding agent with Jira, Azure Boards or Linear to delegate work to Copilot in one click without leaving your project management tool.

Two fixes for the Windows pending-delete race condition: 1. In ExpectEvent, dispose intermediate FileSystemWatcher objects created by RecreateWatcher to release handles promptly, reducing the window for the NTFS pending-delete race. 2. In FileSystemWatcher_Directory_Delete_MultipleFilters, add SpinWait.SpinUntil in cleanup lambdas to wait for directories to be fully deleted before recreating them. On Windows NTFS, RemoveDirectory marks a directory for pending deletion but doesn't remove it until all handles are closed. The previous cleanup's Create() would silently do nothing (because the OS reported the path as still existing), causing the next retry's Delete() to throw DirectoryNotFoundException. Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/a235899f-fd87-4a52-9252-d3b8635d85d1

Apply the same SpinWait.SpinUntil fix to FileSystemWatcher_File_Delete_MultipleFilters to prevent the same pending-delete race on Windows where File.Create() silently does nothing when the file is still in pending-delete state after deletion. Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/a235899f-fd87-4a52-9252-d3b8635d85d1

Copilot

Pull request overview

Improves reliability of System.IO.FileSystem.Watcher tests on Windows by addressing retry/cleanup races that can leave paths in NTFS “pending delete” state and by reducing FileSystemWatcher handle retention across retries.

Changes:

Dispose FileSystemWatcher instances created during ExpectEvent retries to avoid accumulating OS handles.
Add cleanup waits (SpinWait.SpinUntil) before recreating deleted files/directories in multi-filter delete tests.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
`src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs`	Updates `ExpectEvent` retry logic to recreate and dispose watchers between attempts.
`src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.unit.cs`	Adds delete-completion waits in cleanup lambdas for file/directory multi-filter delete tests.

Copilot · 2026-03-22T05:41:50Z

src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs

                    cleanup();
            }
+
+            if (newWatcher != watcher)
+                newWatcher.Dispose();


The recreated watcher is disposed only after the retry loop completes. If ExecuteAndVerifyEvents throws on the final attempt (or if cleanup() throws), this code will exit early and the recreated FileSystemWatcher will not be disposed, reintroducing handle leaks. Consider wrapping the retry loop in a try/finally (or a using for the recreated watcher) so any watcher created by RecreateWatcher is always disposed on exit.

Copilot · 2026-03-22T05:41:50Z

src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.unit.cs

+                    cleanup: () => { SpinWait.SpinUntil(() => !File.Exists(fileOne.FullName), 2000); fileOne.Create().Dispose(); },
+                    expectedPath: fileOne.FullName);
+                ExpectEvent(watcher, WatcherChangeTypes.Deleted, fileTwo.Delete,
+                    cleanup: () => { SpinWait.SpinUntil(() => !File.Exists(fileTwo.FullName), 2000); fileTwo.Create().Dispose(); },
+                    expectedPath: fileTwo.FullName);
+                ExpectNoEvent(watcher, WatcherChangeTypes.Deleted, fileThree.Delete,
+                    cleanup: () => { SpinWait.SpinUntil(() => !File.Exists(fileThree.FullName), 2000); fileThree.Create().Dispose(); },


SpinWait.SpinUntil returns false on timeout, but the result is ignored here. If the delete hasn’t fully completed within 2000ms, the subsequent Create() can still no-op and the next retry may hit the same pending-delete race. Please check the return value (e.g., assert it succeeded with a useful message, or otherwise handle the timeout) so cleanup reliably restores the file state before the next attempt.

Suggested change

cleanup: () => { SpinWait.SpinUntil(() => !File.Exists(fileOne.FullName), 2000); fileOne.Create().Dispose(); },

expectedPath: fileOne.FullName);

ExpectEvent(watcher, WatcherChangeTypes.Deleted, fileTwo.Delete,

cleanup: () => { SpinWait.SpinUntil(() => !File.Exists(fileTwo.FullName), 2000); fileTwo.Create().Dispose(); },

expectedPath: fileTwo.FullName);

ExpectNoEvent(watcher, WatcherChangeTypes.Deleted, fileThree.Delete,

cleanup: () => { SpinWait.SpinUntil(() => !File.Exists(fileThree.FullName), 2000); fileThree.Create().Dispose(); },

cleanup: () =>

{

Assert.True(

SpinWait.SpinUntil(() => !File.Exists(fileOne.FullName), 2000),

$"Timed out waiting for {fileOne.FullName} to be deleted.");

fileOne.Create().Dispose();

},

expectedPath: fileOne.FullName);

ExpectEvent(watcher, WatcherChangeTypes.Deleted, fileTwo.Delete,

cleanup: () =>

{

Assert.True(

SpinWait.SpinUntil(() => !File.Exists(fileTwo.FullName), 2000),

$"Timed out waiting for {fileTwo.FullName} to be deleted.");

fileTwo.Create().Dispose();

},

expectedPath: fileTwo.FullName);

ExpectNoEvent(watcher, WatcherChangeTypes.Deleted, fileThree.Delete,

cleanup: () =>

{

Assert.True(

SpinWait.SpinUntil(() => !File.Exists(fileThree.FullName), 2000),

$"Timed out waiting for {fileThree.FullName} to be deleted.");

fileThree.Create().Dispose();

},

Copilot · 2026-03-22T05:41:50Z

src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.unit.cs

+                    cleanup: () => { SpinWait.SpinUntil(() => !Directory.Exists(directoryOne.FullName), 2000); directoryOne.Create(); },
+                    expectedPath: directoryOne.FullName);
+                ExpectEvent(watcher, WatcherChangeTypes.Deleted, action: () => directoryTwo.Delete(),
+                    cleanup: () => { SpinWait.SpinUntil(() => !Directory.Exists(directoryTwo.FullName), 2000); directoryTwo.Create(); },
+                    expectedPath: directoryTwo.FullName);
+                ExpectNoEvent(watcher, WatcherChangeTypes.Deleted, action: () => directoryThree.Delete(),
+                    cleanup: () => { SpinWait.SpinUntil(() => !Directory.Exists(directoryThree.FullName), 2000); directoryThree.Create(); },


SpinWait.SpinUntil can time out and return false, but the return value is ignored. If the directory is still in a pending-delete state after 2000ms, Create() may still no-op and leave the next retry operating on a path that can disappear asynchronously. Please check the return value (e.g., assert success with a clear message, or handle the timeout) so cleanup deterministically restores the directory before the next attempt.

Suggested change

cleanup: () => { SpinWait.SpinUntil(() => !Directory.Exists(directoryOne.FullName), 2000); directoryOne.Create(); },

expectedPath: directoryOne.FullName);

ExpectEvent(watcher, WatcherChangeTypes.Deleted, action: () => directoryTwo.Delete(),

cleanup: () => { SpinWait.SpinUntil(() => !Directory.Exists(directoryTwo.FullName), 2000); directoryTwo.Create(); },

expectedPath: directoryTwo.FullName);

ExpectNoEvent(watcher, WatcherChangeTypes.Deleted, action: () => directoryThree.Delete(),

cleanup: () => { SpinWait.SpinUntil(() => !Directory.Exists(directoryThree.FullName), 2000); directoryThree.Create(); },

cleanup: () =>

{

bool deleted = SpinWait.SpinUntil(() => !Directory.Exists(directoryOne.FullName), 2000);

Assert.True(deleted, $"Timed out waiting for directory '{directoryOne.FullName}' to be deleted.");

directoryOne.Create();

},

expectedPath: directoryOne.FullName);

ExpectEvent(watcher, WatcherChangeTypes.Deleted, action: () => directoryTwo.Delete(),

cleanup: () =>

{

bool deleted = SpinWait.SpinUntil(() => !Directory.Exists(directoryTwo.FullName), 2000);

Assert.True(deleted, $"Timed out waiting for directory '{directoryTwo.FullName}' to be deleted.");

directoryTwo.Create();

},

expectedPath: directoryTwo.FullName);

ExpectNoEvent(watcher, WatcherChangeTypes.Deleted, action: () => directoryThree.Delete(),

cleanup: () =>

{

bool deleted = SpinWait.SpinUntil(() => !Directory.Exists(directoryThree.FullName), 2000);

Assert.True(deleted, $"Timed out waiting for directory '{directoryThree.FullName}' to be deleted.");

directoryThree.Create();

},

danmoseley · 2026-03-22T07:21:36Z

@copilot address feedback

…Wait return value - Wrap ExpectEvent retry loop in try/finally so recreated FileSystemWatcher instances are always disposed even if ExecuteAndVerifyEvents or cleanup() throws - Assert the return value of SpinWait.SpinUntil in cleanup lambdas in both FileSystemWatcher_File_Delete_MultipleFilters and FileSystemWatcher_Directory_Delete_MultipleFilters to surface a clear failure message if the delete hasn't completed within the 2s timeout Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/87e18d9f-0248-4e37-85ee-7daba9ccd599

Copilot · 2026-03-22T07:44:31Z

@copilot address feedback

Done in d28498f. Three changes:

Wrapped the ExpectEvent retry loop in try/finally so the recreated FileSystemWatcher is always disposed even if ExecuteAndVerifyEvents or cleanup() throws.
Changed SpinWait.SpinUntil calls in both FileSystemWatcher_File_Delete_MultipleFilters and FileSystemWatcher_Directory_Delete_MultipleFilters cleanup lambdas to assert the return value, surfacing a clear timeout message if the delete hasn't completed within 2s.

Initial plan

b481a48

Copilot AI assigned Copilot and danmoseley Mar 22, 2026

Copilot started work on behalf of danmoseley March 22, 2026 04:41 View session

github-actions bot added the area-System.IO label Mar 22, 2026

Copilot AI and others added 2 commits March 22, 2026 05:06

Copilot AI changed the title ~~[WIP] Fix file system watcher path issue in directory delete tests~~ Fix flaky FileSystemWatcher_Directory_Delete_MultipleFilters due to Windows NTFS pending-delete race Mar 22, 2026

Copilot AI requested a review from danmoseley March 22, 2026 05:10

Copilot finished work on behalf of danmoseley March 22, 2026 05:10

danmoseley approved these changes Mar 22, 2026

View reviewed changes

danmoseley marked this pull request as ready for review March 22, 2026 05:37

Copilot AI review requested due to automatic review settings March 22, 2026 05:37

Copilot started reviewing on behalf of danmoseley March 22, 2026 05:39 View session

Copilot AI reviewed Mar 22, 2026

View reviewed changes

danmoseley requested a review from adamsitnik March 22, 2026 05:44

Copilot started work on behalf of danmoseley March 22, 2026 07:21 View session

Copilot AI requested a review from danmoseley March 22, 2026 07:45

Copilot finished work on behalf of danmoseley March 22, 2026 07:45

build-analysis bot mentioned this pull request Mar 22, 2026

Cannot find 'arm64-v8a' device dotnet/dnceng#2284

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix flaky FileSystemWatcher_Directory_Delete_MultipleFilters due to Windows NTFS pending-delete race#125915

Fix flaky FileSystemWatcher_Directory_Delete_MultipleFilters due to Windows NTFS pending-delete race#125915
Copilot wants to merge 4 commits intomainfrom
copilot/fix-file-system-watcher-path-issue

Copilot AI commented Mar 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 22, 2026

Uh oh!

Copilot AI Mar 22, 2026

Uh oh!

Copilot AI Mar 22, 2026

Uh oh!

danmoseley commented Mar 22, 2026

Uh oh!

Copilot AI commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reproduction Steps

Known issue validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

danmoseley commented Mar 22, 2026

Uh oh!

Copilot AI commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Mar 22, 2026 •

edited

Loading