Skip to content

[dotnet-watch tests] Cap output-wait timeout below the MTP hang-dump#55406

Draft
Evangelink wants to merge 1 commit into
mainfrom
dev/amauryleve/watch-test-hangdump-timeout-cap
Draft

[dotnet-watch tests] Cap output-wait timeout below the MTP hang-dump#55406
Evangelink wants to merge 1 commit into
mainfrom
dev/amauryleve/watch-test-hangdump-timeout-cap

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Summary

Follow-up to the CI analysis on #55297. Fixes the test-infra half of the intermittent dotnet-watch.Tests hangs that kill an entire Helix work item (exit code 7) instead of failing a single test.

Problem

AwaitableProcess (the dotnet-watch test harness) derived its per-output-wait timeout directly from the Helix work-item timeout:

s_timeout = HELIX_WORK_ITEM_TIMEOUT is set ? (workItemTimeout - 10s) : 10 min;

With TestWorkItemTimeout = 60min (test/UnitTests.proj), that is 59:50. But Microsoft.Testing.Platform''s hang-dump fires at ~80% of the work-item timeout (48:00). So the hang-dump always wins the race against the graceful per-wait Assert.Fail("Output not found within …").

Result: when a single test wedges on an output wait (e.g. the intermittent WebSocket hot-reload connection hang in HotReload_WithWebSocketCapability — see #55258, #55044, #53289), the whole work item hangs ~48 min until the hang-dump, is killed with exit code 7, and loses all results — instead of that one test failing fast with a useful message.

Fix

Cap the per-wait timeout at 10 minutes (still honoring a shorter work-item timeout when configured), well below the 48-min hang-dump. A single output wait never legitimately needs more than a few minutes, so a genuinely wedged wait now fails that test fast with Output not found within 00:10:00, and the rest of the work item''s results are preserved.

This does not fix the underlying WebSocket connection flakiness (that''s a product-side change in the hot-reload agent — raise/retry the 5s connect timeout and surface the failure; deferred pending owner input). It only ensures the flake fails gracefully instead of nuking the work item.

Validation

  • Microsoft.DotNet.HotReload.Test.Utilities builds clean (0 warnings / 0 errors).
  • Change is confined to test infrastructure; no product code affected.

cc @tmat

AwaitableProcess derived its per-wait output timeout directly from HELIX_WORK_ITEM_TIMEOUT (work-item timeout minus 10s = 59:50 for the 60-min work items in test/UnitTests.proj). Microsoft.Testing.Platform's hang-dump fires at ~80% of the work-item timeout (48 min), so it always triggered before the graceful per-wait Assert.Fail. A single wedged output wait (e.g. the intermittent WebSocket hot-reload connection hang) therefore hung the entire work item until the 48-min hang-dump, killing the run and losing all results instead of failing that one test.

Cap the per-wait timeout at 10 minutes (still honoring a shorter work-item timeout when configured) so a wedged wait fails fast with a useful 'Output not found' message, well below the hang-dump. No single output wait legitimately needs more than a few minutes.

Follow-up to the CI analysis on #55297. Related: #55258, #55044.
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@tmat

tmat commented Jul 22, 2026

Copy link
Copy Markdown
Member

hang that kills the whole run and loses all results

This is a bug (or lack of feature) in the test runner. The test outputs should be available no matter what.

@Evangelink

Copy link
Copy Markdown
Member Author

@tmat I am taking that on me, I'll see how to fix/update MSTest or MTP depending on where the issue is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants