[dotnet-watch tests] Cap output-wait timeout below the MTP hang-dump#55406
Draft
Evangelink wants to merge 1 commit into
Draft
[dotnet-watch tests] Cap output-wait timeout below the MTP hang-dump#55406Evangelink wants to merge 1 commit into
Evangelink wants to merge 1 commit into
Conversation
AwaitableProcess derived its per-wait output timeout directly from HELIX_WORK_ITEM_TIMEOUT (work-item timeout minus 10s = 59:50 for the 60-min work items in test/UnitTests.proj). Microsoft.Testing.Platform's hang-dump fires at ~80% of the work-item timeout (48 min), so it always triggered before the graceful per-wait Assert.Fail. A single wedged output wait (e.g. the intermittent WebSocket hot-reload connection hang) therefore hung the entire work item until the 48-min hang-dump, killing the run and losing all results instead of failing that one test. Cap the per-wait timeout at 10 minutes (still honoring a shorter work-item timeout when configured) so a wedged wait fails fast with a useful 'Output not found' message, well below the hang-dump. No single output wait legitimately needs more than a few minutes. Follow-up to the CI analysis on #55297. Related: #55258, #55044.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Member
This is a bug (or lack of feature) in the test runner. The test outputs should be available no matter what. |
Member
Author
|
@tmat I am taking that on me, I'll see how to fix/update MSTest or MTP depending on where the issue is. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to the CI analysis on #55297. Fixes the test-infra half of the intermittent
dotnet-watch.Testshangs that kill an entire Helix work item (exit code 7) instead of failing a single test.Problem
AwaitableProcess(the dotnet-watch test harness) derived its per-output-wait timeout directly from the Helix work-item timeout:With
TestWorkItemTimeout = 60min(test/UnitTests.proj), that is 59:50. But Microsoft.Testing.Platform''s hang-dump fires at ~80% of the work-item timeout (48:00). So the hang-dump always wins the race against the graceful per-waitAssert.Fail("Output not found within …").Result: when a single test wedges on an output wait (e.g. the intermittent WebSocket hot-reload connection hang in
HotReload_WithWebSocketCapability— see #55258, #55044, #53289), the whole work item hangs ~48 min until the hang-dump, is killed with exit code 7, and loses all results — instead of that one test failing fast with a useful message.Fix
Cap the per-wait timeout at 10 minutes (still honoring a shorter work-item timeout when configured), well below the 48-min hang-dump. A single output wait never legitimately needs more than a few minutes, so a genuinely wedged wait now fails that test fast with
Output not found within 00:10:00, and the rest of the work item''s results are preserved.This does not fix the underlying WebSocket connection flakiness (that''s a product-side change in the hot-reload agent — raise/retry the 5s connect timeout and surface the failure; deferred pending owner input). It only ensures the flake fails gracefully instead of nuking the work item.
Validation
Microsoft.DotNet.HotReload.Test.Utilitiesbuilds clean (0 warnings / 0 errors).cc @tmat