Migrate to custom OTel Exporter in dotnetup to prevent perf delay and data loss

[nagilson]

Resolves #55353

Validation

I was able to add end-to-end tests that pushed to the real telemetry service, and confirmed that for both product and user error, whether in local dev scenarios with a new detached uploader which runs after dotnetup, or in CI, telemetry now got sent 100% of the time in these tests. I used kusto queries to validate the data with special ids to ensure the data was mine, let me know if you want those queries as evidence.

#55438 <-- To trigger a failure in CI, I created this failing yml scenario.

Changes

I adjusted the blob push to no longer happen async at launch due to the fact that most dotnetup executions returned too quickly for it to ever get pushed, and it seemed to hamper the performance. I liked this detached process approach more as it creates no exit delay and does not persist too long. The .lock approach on the exporter should protect against any problems of concurrent dotnetup proccesses.

I also moved the storage directory to be shared with the SDK so that when the SDK flushes, dotnetup's telemetry is also flushed, considering that the SDK will likely be used a lot more often than dotnetup.

Changes to Exporter Lib

Finally, I added a retry/back-off as polish on top of the exporter library.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR updates dotnetup telemetry to use the SDK’s persistent-storage exporter model, aiming to improve reliability for short-lived executions (avoid exit-delay, reduce dropped telemetry) by persisting synchronously and draining out-of-band via a detached “drainer” process, while also adding retry/backoff behavior in the shared exporter library.

Changes:

• Add a persistent-storage telemetry drainer (with directory locking + backoff) and wire dotnetup to spawn it detached on local runs, while keeping inline export for CI/one-and-done environments.
• Extend the persistent exporter pipeline to propagate Retry-After and expose “should back off” results from storage drain passes.
• Add unit + E2E test coverage for retry/backoff, drain locking, and dotnetup detached drainer behavior (including a local mock ingestion server).

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/Microsoft.DotNet.Cli.Telemetry.Tests/PersistentStorageTelemetryUploaderTests.cs	Updates uploader tests for new drain result/backoff semantics.
test/Microsoft.DotNet.Cli.Telemetry.Tests/PersistentStorageTelemetryDrainerTests.cs	Adds unit tests for drainer retry delay escalation, lifetime, and directory lock exclusivity.
test/Microsoft.DotNet.Cli.Telemetry.Tests/HttpTelemetryUploadTransportTests.cs	Adds coverage for Retry-After propagation on retriable responses.
test/dotnetup.Tests/Utilities/TelemetryTestEnvironment.cs	Adds a helper to run dotnetup under a telemetry E2E test environment and wait for blob deletion.
test/dotnetup.Tests/Utilities/DotnetupTestUtilities.cs	Adds helper to require a NativeAOT-published dotnetup executable path for E2E tests.
test/dotnetup.Tests/TelemetryTests.cs	Adds unit tests for new shutdown budget and connection string resolution behavior.
test/dotnetup.Tests/TelemetryDrainE2ETests.cs	Adds E2E tests validating drain mode and detached drainer upload behavior.
test/dotnetup.Tests/Mocks/MockTelemetryIngestionServer.cs	Adds a local TCP-based mock ingestion endpoint to validate uploads without external dependencies.
test/dotnetup.Tests/DotnetupTelemetryDrainProcessTests.cs	Adds unit tests for drain-mode fast path and storage directory resolution.
src/Installer/dotnetup.Library/Telemetry/TelemetryTestHooks.cs	Adds test hook to record shutdown budget without impacting production behavior.
src/Installer/dotnetup.Library/Telemetry/DotnetupTelemetryDrainProcess.cs	Adds drain-mode process entry + detached drainer spawn logic for dotnetup.
src/Installer/dotnetup.Library/Telemetry/DotnetupTelemetry.cs	Switches between CI inline export and local persist+detached-drain; adds new shutdown budget logic and connection string override.
src/Installer/dotnetup.Library/Program.cs	Adds early drain-mode fast path and refactors console-output setup.
src/Installer/dotnetup.Library/DotnetupPaths.cs	Adds logic to resolve a local telemetry storage directory shared with the SDK.
src/Installer/dotnetup.Library/dotnetup.Library.csproj	Links shared CLI folder path calculator and references the shared telemetry project.
src/Installer/dotnetup.Library/docs/dotnetup-telemetry.md	Documents related environment variables for the updated telemetry model.
src/Installer/dotnetup.Library/Constants.cs	Adds env vars for drain mode, CI shutdown timeout override, and local delivery forcing.
src/Common/CliFolderPathCalculatorCore.cs	Adjusts OS detection helper used by profile-path resolution.
src/Cli/Microsoft.DotNet.Cli.Telemetry/PersistentStorageTracerProviderBuilderExtensions.cs	Threads new StartBackgroundDrain option into trace exporter creation.
src/Cli/Microsoft.DotNet.Cli.Telemetry/PersistentStorageTraceExporter.cs	Adds StartBackgroundDrain gating for in-process background drain.
src/Cli/Microsoft.DotNet.Cli.Telemetry/PersistentStorageTelemetryOptions.cs	Adds StartBackgroundDrain option to support out-of-band draining.
src/Cli/Microsoft.DotNet.Cli.Telemetry/PersistentStorageTelemetryDrainer.cs	Adds a standalone drainer with directory locking and retry/backoff.
src/Cli/Microsoft.DotNet.Cli.Telemetry/PersistentStorageLoggerOptionsExtensions.cs	Threads new StartBackgroundDrain option into log exporter creation.
src/Cli/Microsoft.DotNet.Cli.Telemetry/PersistentStorageLogExporter.cs	Adds StartBackgroundDrain gating for in-process background drain.
src/Cli/Microsoft.DotNet.Cli.Telemetry/Implementation/TelemetryUploadResult.cs	Adds RetryAfter to upload results and introduces TelemetryDrainResult.
src/Cli/Microsoft.DotNet.Cli.Telemetry/Implementation/PersistentStorageTelemetryUploader.cs	Changes drain to return result/backoff info and stop-pass-on-backoff behavior.
src/Cli/Microsoft.DotNet.Cli.Telemetry/Implementation/HttpTelemetryUploadTransport.cs	Parses/propagates Retry-After header to enable backoff behavior.

Comments suppressed due to low confidence (2)

src/Cli/Microsoft.DotNet.Cli.Telemetry/Implementation/PersistentStorageTelemetryUploader.cs:146

• This unconditional early break on shouldBackOff means the uploader stops the drain pass after the first retryable/failed upload. With the current logic, any rejection (even without Retry-After) prevents attempting additional blobs that might succeed, which can stall draining when one blob is persistently rejected.

            {
                // A retryable response normally indicates service throttling or a transient
                // failure. Stop this pass rather than submitting every remaining blob to a
                // service that has already asked us to retry.
                break;

src/Installer/dotnetup.Library/Telemetry/DotnetupTelemetryDrainProcess.cs:51

• Empty catch blocks make it hard to tell whether swallowing exceptions is intentional. Adding a short comment here would clarify that drain failures are deliberately ignored and that the drainer returns success regardless.

        catch
        {

        }

[Copilot]

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 2 comments.