[8.0] [mono] Test process gets killed

[carlossanlop]

Error Blob

{
  "ErrorMessage": "Tests' END OF WORK ITEM LOG: Command timed out, and was killed]",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

This failure happens often in System.Runtime.Tests specifically.

• PR: [release/8.0] HTTP & SSL Stress: Run nightlies against release/8.0 #90809
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=378797&view=logs&j=1f44e919-17d8-5098-631c-31ef11f9e57b&t=d617d884-da52-5b82-bd49-f7613b408af4&l=87
• Queue: Libraries Test Run release mono linux x64 Debug
• Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-90809-merge-956e379e01714658b2/System.Runtime.Tests/1/console.73e1394a.log?helixlogtype=result
• Output:

Console log: 'System.Runtime.Tests' from job 956e379e-0171-4658-b2cd-b234086ec9af workitem 50980a74-2595-4c8f-b4a1-bbcc0991bb7b (ubuntu.1804.amd64.open.svc) executed on machine a005SZ1 running Linux-5.4.0-1109-azure-x86_64-with-Ubuntu-18.04-bionic
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/9F780922/p
----- start Fri Aug 18 17:28:49 UTC 2023 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/9F780922/p/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/9F780922/w/A2490973/e /datadisks/disk1/work/9F780922/w/A2490973/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 9160 of 9278 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
    System.Tests.ArrayEnumeratorTests_int.ICollection_Generic_Remove_ReferenceRemovedFromCollection [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Known issue validation

Build: 🔎
Result validation: ⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.
Validation performed at: 8/29/2023 9:18:10 PM UTC

Report

Build	Definition	Test	Pull Request
390358	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91314
390315	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91239
390327	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
390312	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91240
390304	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91311
390232	dotnet/runtime	System.Globalization.Tests.WorkItemExecution	#90970
390131	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91303
389716	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91293
389636	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91290
389641	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91291
389595	dotnet/runtime	System.Net.NetworkInformation.Functional.Tests.WorkItemExecution
389529	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#91283
389491	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
389478	dotnet/runtime	Microsoft.Extensions.Configuration.Functional.Tests.WorkItemExecution	#91239
389287	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91267
389320	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#91216
389280	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91264
389142	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#91262
388978	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
388942	dotnet/runtime	normal-System.Threading.Tasks.Tests.WorkItemExecution
388902	dotnet/runtime	Microsoft.Extensions.HostFactoryResolver.Tests.WorkItemExecution	#91176
388878	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91240
388706	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91245
388797	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91237
388672	dotnet/runtime	System.Net.Sockets.Tests.WorkItemExecution	#91247
388667	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91246
388556	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89952
388379	dotnet/runtime	System.IO.Pipes.Tests.WorkItemExecution	#91231
388449	dotnet/runtime	System.Net.Http.Unit.Tests.WorkItemExecution
388406	dotnet/runtime	System.Security.Cryptography.OpenSsl.Tests.WorkItemExecution	#91190
388166	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91218
388145	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91154
387880	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91172
387528	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91189
387355	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89952
387323	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90982
387294	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91171
387117	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91160
387116	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
387106	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
386969	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91154
386929	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90982
386516	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
386357	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
386352	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91133
386233	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91074
386241	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91107
386139	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91123
386115	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91122
386015	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91116
385733	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90518
385671	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91102
385446	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89405
385484	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
385364	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
385005	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#89102
384990	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91077
384688	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91057
384597	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91050
384450	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91033
384447	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#91032
384038	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90790
383772	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#88034
382759	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90954
378797	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#90809

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
18	64	65

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Error Blob

{
  "ErrorMessage": "System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": true
}

Reproduction Steps

This failure happens often in System.Runtime.Tests specifically.

• PR: [release/8.0] HTTP & SSL Stress: Run nightlies against release/8.0 #90809
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=378797&view=logs&j=1f44e919-17d8-5098-631c-31ef11f9e57b&t=d617d884-da52-5b82-bd49-f7613b408af4&l=87
• Queue: Libraries Test Run release mono linux x64 Debug
• Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-90809-merge-956e379e01714658b2/System.Runtime.Tests/1/console.73e1394a.log?helixlogtype=result
• Output:

Console log: 'System.Runtime.Tests' from job 956e379e-0171-4658-b2cd-b234086ec9af workitem 50980a74-2595-4c8f-b4a1-bbcc0991bb7b (ubuntu.1804.amd64.open.svc) executed on machine a005SZ1 running Linux-5.4.0-1109-azure-x86_64-with-Ubuntu-18.04-bionic
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/9F780922/p
----- start Fri Aug 18 17:28:49 UTC 2023 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/9F780922/p/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/9F780922/w/A2490973/e /datadisks/disk1/work/9F780922/w/A2490973/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 9160 of 9278 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
    System.Tests.ArrayEnumeratorTests_int.ICollection_Generic_Remove_ReferenceRemovedFromCollection [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Author:	carlossanlop
Assignees:	-
Labels:	area-System.Runtime, os-linux, arch-x64, blocking-clean-ci, untriaged, runtime-mono, Known Build Error
Milestone:	-

[carlossanlop]

Here's another 8.0 repro. This time in osx, but also mono:

• Affected PR: [release/8.0] Update dependencies from dotnet/roslyn-analyzers #91057
• Queue: Libraries Test Run release mono osx x64 Debug
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=384688&view=logs&j=ab5d32de-6d19-5068-e03b-b69cf4db6cd5&t=eb2a8345-733e-5823-7aa4-11fb75e4a1ff&l=55
• Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-91057-merge-18f6ab2e6a76443fb6/System.Runtime.Tests/1/console.dbd0bad1.log?helixlogtype=result
• Output:

...
    System.Tests.WeakReferenceTests.NonGeneric [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
    System.Tests.ArrayEnumeratorTests_int.ICollection_Generic_Remove_ReferenceRemovedFromCollection [SKIP]
      Condition(s) not met: "IsPreciseGcSupported"
['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]