Add generic ITestHostLauncher extension point + PackagedApp reference extension#9454
Add generic ITestHostLauncher extension point + PackagedApp reference extension#9454Evangelink wants to merge 17 commits into
Conversation
Introduce a public, experimental Microsoft.Testing.Platform extension point that lets an extension control how the out-of-process test host is launched, replacing the platform's default Process.Start. The abstraction is agnostic of the launch mechanism (process, packaged/MSIX deploy+activate, container, remote): the launcher returns an ITestHostHandle exposing only lifecycle (WaitForExitAsync, ExitCode, HasExited, Exited, Terminate) with an optional ProcessId for diagnostics. - New public types: ITestHostLauncher, ITestHostHandle, TestHostLaunchContext (all [Experimental(TPEXP)], no init accessors). - ITestHostControllersManager.AddTestHostLauncher overloads + manager wiring; registering a launcher forces the controller host (RequireProcessRestart) and at most one launcher is allowed (localized OnlyOneTestHostLauncherSupported). - TestHostControllersTestHost delegates the launch to the registered launcher and adapts the returned handle to the internal IProcess monitoring contract, tolerating a null PID for container/remote launches. - Unit tests for restart-forcing, singleton, and duplicate-id validation. - Acceptance test with a real consuming launcher proving end-to-end delegation. - Rework RFC 017 to the generic ITestHostLauncher/ITestHostHandle shape and a package/deploy framing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Prove the ITestHostLauncher hook works for a launch that is not a plain Process.Start: add a real shipping extension Microsoft.Testing.Extensions.AppDeployment that deploys (stages) the test host into an isolated directory, launches the deployed copy, and returns an ITestHostHandle that exposes no local process id. - Platform fix: the test host controller gated premature-exit on "testHostProcessId is null", which would reject a launcher that returns no PID (AUMID/container/remote). Gate on HasExited only; the real test host PID still arrives via the IPC handshake. No behavior change for the default Process.Start path (a null PID there always coincides with HasExited). - New extension: AddAppDeployment, AppDeploymentLauncher, DeployedTestHostHandle (ProcessId => null), TestingPlatformBuilderHook, build props, PublicAPI, PACKAGE.md; added to TestFx.slnx; targets the SupportedNetFrameworks set. - Acceptance test AppDeploymentTests references the packed package and asserts the host was deployed to a separate directory and the run succeeded with a PID-less handle. - RFC updated to describe the HasExited-only gating. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The "AppDeployment" name was too generic: it over-promised a broad deployment feature while the implementation only demonstrated the ITestHostLauncher hook. Rename the consuming extension to Microsoft.Testing.Extensions.WinUI, anchoring it to the concrete motivating scenario (#2784) instead. - Microsoft.Testing.Extensions.WinUI: AddWinUIDeployment, WinUITestHostLauncher, WinUITestHostHandle (ProcessId => null), TestingPlatformBuilderHook (new GUID), build props, PublicAPI, PACKAGE.md; renamed in TestFx.slnx. - The launcher is framed for WinUI: it implements the unpackaged deploy-and-launch path (stage the loose layout, launch the deployed app) and documents the packaged AUMID-activation branch as clearly-marked follow-up. - Acceptance test renamed to WinUIDeploymentTests and gated to Windows; still proves end-to-end deploy + PID-less launch against the packed package. - RFC non-goal updated: a reference WinUI consumer now exists (unpackaged path); packaged AUMID activation remains a separate follow-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UWP and packaged WinUI are not distinct scenarios for launching the test host: both produce MSIX packages and share the same deploy + AUMID-activate mechanism (which is why VSTest exposes a single UwpTestHostRuntimeProvider for both). Naming the extension after either app model is too narrow; name it after the shared packaging format instead. - Microsoft.Testing.Extensions.WinUI -> Microsoft.Testing.Extensions.Msix (AddMsixDeployment, MsixTestHostLauncher, MsixTestHostHandle, MsixExtensions). - Casing is PascalCase "Msix" (not "MSIX"), matching .NET guidelines and the repo convention for 3+ letter acronyms (HtmlReport, TrxReport, CtrfReport). - Docs/launcher text now describe UWP and packaged WinUI as the same MSIX mechanism; packaged AUMID activation remains a clearly-marked follow-up. - Acceptance test renamed to MsixDeploymentTests (Windows-gated); still proves end-to-end deploy + PID-less launch against the packed package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
"Msix" reads like a tool that creates MSIX packages; this extension is about running tests *inside* a packaged Windows app. Name it after the scenario instead of the package format. - Microsoft.Testing.Extensions.Msix -> Microsoft.Testing.Extensions.PackagedApp (AddPackagedAppDeployment, PackagedAppTestHostLauncher, PackagedAppTestHostHandle, PackagedAppExtensions). - Still covers both UWP and packaged WinUI (both ship as MSIX and share the same deploy + AUMID-activate mechanism); docs keep "MSIX" as the format acronym in prose while the package/API name describes the scenario. - Acceptance test renamed to PackagedAppDeploymentTests (Windows-gated); still proves end-to-end deploy + PID-less launch against the packed package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR implements RFC 017 by adding an experimental, public Microsoft.Testing.Platform (MTP) extension point (ITestHostLauncher + ITestHostHandle + TestHostLaunchContext) to allow extensions to control how the out-of-process test host is launched, and introduces a reference consumer extension (Microsoft.Testing.Extensions.PackagedApp) that exercises PID-less host monitoring.
Changes:
- Added the experimental
ITestHostLauncherlaunch hook (and related handle/context types) plus controller-manager registration and enforcement (single launcher, forces out-of-process controller host). - Updated
TestHostControllersTestHostto delegate the launch step to a registered launcher and to support PID-less launchers by relying on lifecycle + IPC PID handshake. - Added acceptance/unit tests and a new packable extension (
Microsoft.Testing.Extensions.PackagedApp) demonstrating deploy-to-isolated-directory + PID-less handle.
Show a summary per file
| File | Description |
|---|---|
| TestFx.slnx | Adds the new PackagedApp extension project to the main solution. |
| test/UnitTests/Microsoft.Testing.Platform.UnitTests/TestApplicationBuilderTests.cs | Adds unit tests validating launcher registration, singleton enforcement, and process-restart forcing. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/TestHostLauncherTests.cs | Adds acceptance coverage for end-to-end custom launcher usage. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/PackagedAppDeploymentTests.cs | Adds Windows-gated acceptance coverage for PID-less launcher behavior via PackagedApp extension. |
| src/Platform/Microsoft.Testing.Platform/TestHostControllers/TestHostLaunchContext.cs | Introduces the launch context passed to a custom launcher. |
| src/Platform/Microsoft.Testing.Platform/TestHostControllers/TestHostControllersManager.cs | Adds registration/build pipeline for ITestHostLauncher and enforces single launcher. |
| src/Platform/Microsoft.Testing.Platform/TestHostControllers/TestHostControllerConfiguration.cs | Carries the optional launcher through controller configuration. |
| src/Platform/Microsoft.Testing.Platform/TestHostControllers/ITestHostLauncher.cs | Adds the experimental launcher interface contract. |
| src/Platform/Microsoft.Testing.Platform/TestHostControllers/ITestHostHandle.cs | Adds the experimental handle contract used for monitoring launched hosts. |
| src/Platform/Microsoft.Testing.Platform/TestHostControllers/ITestHostControllerManager.cs | Adds public controller-manager APIs to register a launcher (factory + composite). |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.zh-Hant.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.zh-Hans.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.tr.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.ru.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.pt-BR.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.pl.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.ko.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.ja.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.it.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.fr.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.es.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.de.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.cs.xlf | Adds localized entry for single-launcher enforcement message. |
| src/Platform/Microsoft.Testing.Platform/Resources/PlatformResources.resx | Adds the OnlyOneTestHostLauncherSupported resource string. |
| src/Platform/Microsoft.Testing.Platform/PublicAPI/PublicAPI.Unshipped.txt | Tracks the new experimental public APIs. |
| src/Platform/Microsoft.Testing.Platform/Hosts/TestHostControllersTestHost.cs | Delegates host launch via ITestHostLauncher and supports PID-less launchers. |
| src/Platform/Microsoft.Testing.Platform/Helpers/System/TestHostHandleToProcessAdapter.cs | Adapts public ITestHostHandle to internal IProcess. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/TestingPlatformBuilderHook.cs | Builder hook for MSBuild-driven extension registration. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/PublicAPI/PublicAPI.Unshipped.txt | Tracks new public APIs for the PackagedApp extension. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/PublicAPI/PublicAPI.Shipped.txt | Initializes shipped API tracking file for the new package. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/PackagedAppTestHostLauncher.cs | Implements deploy + launch using the new launcher extension point. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/PackagedAppTestHostHandle.cs | Implements a PID-less test host handle over a process. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/PackagedAppExtensions.cs | Adds AddPackagedAppDeployment() builder extension to register the launcher. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/PACKAGE.md | Documents the new experimental PackagedApp extension package. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/Microsoft.Testing.Extensions.PackagedApp.csproj | Adds the new PackagedApp extension project and packaging layout. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/buildTransitive/Microsoft.Testing.Extensions.PackagedApp.props | Wires transitive MSBuild import to the multi-targeting props. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/buildMultiTargeting/Microsoft.Testing.Extensions.PackagedApp.props | Declares the TestingPlatformBuilderHook MSBuild item for extension discovery. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/build/Microsoft.Testing.Extensions.PackagedApp.props | Wires non-transitive build import to the multi-targeting props. |
| src/Platform/Microsoft.Testing.Extensions.PackagedApp/BannedSymbols.txt | Adds banned-symbols rules for the new PackagedApp extension project. |
| docs/RFCs/017-TestHost-Launcher.md | Adds/updates the RFC describing the launcher design and scenarios. |
Review details
- Files reviewed: 40/40 changed files
- Comments generated: 2
- Review effort level: Low
- TestHostHandleToProcessAdapter: give the "no process id" InvalidOperationException a descriptive message instead of an empty one, so the PID-less path is diagnosable. - PackagedApp launcher: clean up the staged deployment directory once the host has exited (the handle now owns the directory and best-effort deletes it on Dispose), preventing temp-dir accumulation in CI. Update the acceptance test to no longer require the deployment directory to remain on disk after the run. - Mirror the RFC review fixes (env-var wording, packaged-app example) in the doc shipped with the implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…entifier Per design review on the RFC: - Remove the redundant Exited event from ITestHostHandle; consumers use WaitForExitAsync. The internal IProcess adapter synthesizes its informational Exited event from the exit task instead. - Replace int? ProcessId with an optional free-form string Identifier (diagnostics only: PID, container id, remote host:pid, ...). The controller host logs it where the handle is visible; the adapter no longer pretends to expose a numeric PID. - Update the PackagedApp handle (Identifier => null) and the acceptance asset handle (Identifier => process id string) accordingly, plus PublicAPI and the RFC doc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- PackagedAppTestHostHandle.Dispose: the two best-effort cleanup catches (IOException, UnauthorizedAccessException) now log via Debug.WriteLine instead of being empty, keeping cleanup non-fatal while satisfying the empty-catch rule. - TestHostHandleToProcessAdapter.RaiseExitedWhenDoneAsync: replace the bare generic catch with catch (Exception ex) and a Debug.WriteLine, addressing the generic catch-clause finding while preserving the swallow-and-continue behavior for the informational Exited event. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…sthost-launcher-rfc # Conflicts: # src/Platform/Microsoft.Testing.Platform/PublicAPI/PublicAPI.Unshipped.txt
ITestHostHandle has no Exited event (consumers use WaitForExitAsync); drop the stale "Exited" from the lifecycle-contract comment so it matches the interface and the adapter comment. Also spell "queryable". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
…isposable Per RFC design review: - ITestHostHandle now extends IDisposable so the platform deterministically releases handle-held OS resources (it already wraps the handle in `using` via the adapter). - WaitForExitAsync takes a CancellationToken (the runtime API and the repo's netstandard2.0 polyfill both support one). Threaded through the internal IProcess contract, SystemProcess, and the handle adapter. - TestHostControllersTestHost passes its cancellation token when waiting for the host to exit; on cancellation it terminates the host and waits (uncancelable) for full exit so the existing exit-code reconciliation still observes a real OS exit code. The normal (non-canceled) path is unchanged. - Pre-existing internal callers (Retry, HangDump) pass CancellationToken.None to keep their exact prior behavior; the adapter synthesizes its informational Exited event with a dispose-linked token. - Document ExitCode-before-HasExited as undefined, clarify Quote/PasteArguments are placeholders, and fix the docker example so Terminate() tears down the container rather than only killing the local client. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ITestHostHandle now extends IDisposable and WaitForExitAsync takes a CancellationToken (both supported by the runtime and the netstandard2.0 polyfill). The platform disposes the handle after exit and passes its cancellation token while waiting, reconciling the real exit code afterwards. - Document ExitCode-before-HasExited as undefined; note Quote/PasteArguments are placeholders for proper argument quoting; fix the container example so Terminate() tears down the container (docker stop) rather than only killing the local client. - Record the design evolution in Alternatives. Implemented in PR #9454. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
… WorkingDirectory - Remove the PackagedApp launcher's write of a deployment-marker file into the original output directory (a shipping-side effect that could fail on read-only dirs and leave files behind). The acceptance test now proves deployment by having the deployed test host self-report its AppContext.BaseDirectory via a platform-forwarded env var, keeping the affordance in the test asset. - Honor cancellation: ThrowIfCancellationRequested before the deploy and during the recursive copy. - Honor TestHostLaunchContext.WorkingDirectory when set, defaulting to the deployment directory only when it is null. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…sthost-launcher-rfc
This comment has been minimized.
This comment has been minimized.
…egistration acceptance test
- Mark PackagedAppExtensions / AddPackagedAppDeployment [Experimental("TPEXP")] and
prefix the PublicAPI entries with [TPEXP] (the MSBuild codegen hook stays non-experimental).
- Give the package an experimental (alpha) version via
MicrosoftTestingExtensionsPackagedApp{VersionPrefix,PreReleaseVersionLabel} +
SuppressFinalPackageVersion, matching the OpenTelemetry/CtrfReport convention.
- Update the buildTransitive props to the build/<tfm> forwarding pattern (#9431).
- Plumb MicrosoftTestingExtensionsPackagedAppVersion through AcceptanceTestBase and the
PackagedApp deployment asset (NoWarn TPEXP/NETSDK1201).
- Add PackagedApp.MSBuildRegistration acceptance test asserting the build/buildTransitive
props auto-register the TestingPlatformBuilderHook via the MSBuild self-registration codegen.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nistic marker path; drop redundant IDisposable - TestHostHandleToProcessAdapter: return from the catch in RaiseExitedWhenDoneAsync so the informational Exited event only fires on a genuine host exit, not when the wait faults or is cancelled (e.g. the adapter is disposed before the host exits). - TestHostLauncherTests asset: write the LaunchTestHostAsync marker next to context.FileName so its location is deterministic regardless of the controller process working directory. - PackagedAppTestHostHandle: ITestHostHandle already extends IDisposable, so drop the redundant IDisposable on the class declaration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
…ackage packs on Linux/macOS
The Linux/macOS CI legs build and pack via NonWindowsTests.slnf (NonWindowsBuild=true);
Microsoft.Testing.Platform.slnf is the matching dev filter. Because the PackagedApp project was
not a member of either filter, its NuGet package was never produced on those legs, so
artifacts/packages/Debug/Shipping had no Microsoft.Testing.Extensions.PackagedApp.*.nupkg.
AcceptanceTestBase's static constructor calls ExtractVersionFromPackage("Microsoft.Testing.Extensions.PackagedApp.")
which throws when the package is absent, failing the type initializer and therefore EVERY acceptance
test in the assembly (523 failures on Linux Debug). Windows legs build the full TestFx.slnx, so the
package was already present there.
Add the project alongside OpenTelemetry (its experimental sibling) in both filters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nt asset - TestHostControllersTestHost: when the wait is cancelled, the host may exit between cancellation and the Kill() call (Kill throws InvalidOperationException when there is no process left to terminate). Make termination best-effort by swallowing that exception; the subsequent WaitForExitAsync still reconciles the real exit code. - PackagedAppDeploymentTests asset: collapse the two consecutive <NoWarn> entries into a single <NoWarn>$(NoWarn);TPEXP;NETSDK1201</NoWarn> to avoid any ambiguity about the effective value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| try | ||
| { | ||
| testHostProcess.Kill(); | ||
| } | ||
| catch (InvalidOperationException) | ||
| { | ||
| // The host may have exited between the cancellation and this Kill call, in which | ||
| // case Kill throws because there is no longer a process to terminate. That is the | ||
| // outcome we wanted, so treat termination as best-effort and ignore it. | ||
| } |
| /// <summary> | ||
| /// This class is used by Microsoft.Testing.Platform.MSBuild to hook into the Testing Platform Builder to add packaged Windows (UWP/WinUI) test host deployment support. | ||
| /// </summary> | ||
| public static class TestingPlatformBuilderHook | ||
| { |
🧪 Test quality grade — PR #9454
This advisory comment was generated automatically. Grades are heuristic
|
Summary
Implements RFC 017 (reworked to be generic — see #9349). Adds a public, experimental MTP extension point —
ITestHostLauncher— that lets an extension control how the out-of-process test host is launched, instead of the platform always callingProcess.Start. The platform keeps owning everything around the launch (argument/environment preparation, the controller↔host IPC pipe, PID tracking,ITestHostProcessLifetimeHandlercallbacks, and exit-code reconciliation) and delegates only the single "create and start the test host" step.The hook is deliberately agnostic of the launch mechanism: the launcher does not have to start a local OS process. It can deploy and activate a packaged app, launch a container, or start the host on a remote machine. The returned
ITestHostHandleexposes only lifecycle (WaitForExitAsync(CancellationToken),ExitCode,HasExited,Terminate, andIDisposable); there is noExitedevent —WaitForExitAsyncis the lifecycle mechanism. An opaque diagnostics-onlystring? Identifieris optional (it is intentionally not a numeric process id, so it can carry whatever a non-process launcher has — a container id, an AUMID-activated token, etc., ornull).Motivating scenario: testing packaged Windows apps (UWP and packaged WinUI), which must be deployed and AUMID-activated rather than
Process.Start-ed — the blocker behind #2784.What's included
Platform extension point (
Microsoft.Testing.Platform)[Experimental("TPEXP")], noinit):ITestHostLauncher—Task<ITestHostHandle> LaunchTestHostAsync(TestHostLaunchContext, CancellationToken)ITestHostHandle— generic, mechanism-agnostic (IDisposable); lifecycle viaWaitForExitAsync(CancellationToken)/ExitCode/HasExited/Terminate; optional opaquestring? Identifierfor diagnostics (noExitedevent, no numeric process id)TestHostLaunchContext—FileName,Arguments,EnvironmentVariables,WorkingDirectoryITestHostControllersManager.AddTestHostLauncher(...)(factory + composite overloads).TestHostControllersTestHostdelegates the launch at theprocess.Startsite and adapts the handle to the internalIProcessmonitoring contract.HasExitedonly (not on the availability of any id), so a launcher that returns no identifier (container/remote/AUMID) is monitored purely through the handle lifecycle + the IPC PID handshake. No behavior change for the defaultProcess.Startpath.Reference consumer (
Microsoft.Testing.Extensions.PackagedApp)UwpTestHostRuntimeProviderfor both). All public API is[Experimental("TPEXP")]and the package ships an experimental (alpha) version.builder.AddPackagedAppDeployment()registers a launcher that deploys (stages the loose layout into an isolated directory) and launches the deployed copy, returning a handle whoseIdentifierisnull— exercising the mechanism-agnostic path end-to-end. Packaged AUMID activation is scaffolded as clearly-marked follow-up.Tests
TestApplicationBuilderTests): launcher forces process restart, singleton enforcement, duplicate-id validation. ✅TestHostLauncherTests): a custom launcher drives the host end-to-end (with an identifier). ✅PackagedAppDeploymentTests, Windows-gated): deploy-to-separate-directory + run succeeds with an identifier-less handle. ✅PackagedApp.MSBuildRegistration): the package'sbuild/buildTransitiveprops auto-register itsTestingPlatformBuilderHook(build-time, asserted from the binlog; OS-agnostic). ✅Notes
docs/RFCs/017-TestHost-Launcher.md(generic rework of the RFC in Add RFC 017: Custom test host launcher #9349).PublicAPI.Unshipped.txtwith the[TPEXP]prefix.Related: #2784, #9349