[release/10.0] Helix reporter: add portable JSON output (#16774)

[wfurt]

Backport of #16774 to release/10.0.

Description

The current format uses Python's Pickle that is not portable. The reporter also depends on helix scripts that may or may not be present anymore. This change preserves the existing Pickle serialization when possible but makes it optional. It adds portable JSON serialization that can be consumed by anyone in any language.

[Copilot]

Pull request overview

This backport adds helix-scripts-independent result handling and portable JSON output while retaining legacy pickle reporting when available.

Changes:

• Added compatibility types and JSON serialization.
• Added fallback imports for environments without helix-scripts.
• Documented the v2 JSON schema and file location.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/test_results_reader/init.py	Updated as part of this pull request.
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.py	Updated as part of this pull request.
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/RESULTS_FORMAT.md	Updated as part of this pull request.
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/formats/xunit.py	Updated as part of this pull request.
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/formats/trx.py	Updated as part of this pull request.
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/formats/result_format.py	Updated as part of this pull request.
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/formats/junit.py	Updated as part of this pull request.
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/_helix_compat.py	Updated as part of this pull request.

Suppressed comments (5)

src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/_helix_compat.py:169

• For a successful work item with no detectable XML results, __no_results_result() supplies the literal string "None" as failure_message (via u'{}'.format(None)). This emits a non-null failure message for a Pass, contrary to the JSON schema's null-for-passes contract; fix that input or normalize absent failure messages before serializing.

            "duration_seconds": self._duration_seconds,
            "result": self._result,
            "exception_type": self._exception_type,
            "failure_message": self._failure_message,
            "stack_trace": self._stack_trace,
            "skip_reason": self._skip_reason,

src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/_helix_compat.py:248

• The parsers deliberately stream XML results and clear each element, but this method materializes every result and then builds a second full list of dictionaries before writing. Large suites with console/stack-trace attachments can therefore retain the entire result object graph plus a second copy and hit the work item's memory limit. Write the JSON array incrementally (or otherwise avoid retaining both representations) while preserving the dual-reporter behavior.

    def report_results(self, results):
        results = [r for r in (results or []) if r is not None]
        path = json_results_path()
        payload = {
            "schema_version": SCHEMA_VERSION,
            "azdo": _azdo_to_dict(self._azdo),
            "results": [_result_to_dict(r) for r in results],

src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/_helix_compat.py:258

• The payload contains azdo.access_token, but this creates the file with the default open mode (0666 before umask, commonly 0644). On a shared Helix agent, another user who can read the work-item directory can recover the bearer token from this new plaintext file. Create it with restrictive permissions (including the existing-file case), or omit the token and require consumers to obtain it out-of-band.

        with open(path, "w", encoding="utf-8") as f:
            json.dump(payload, f, ensure_ascii=False)

src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.py:103

• This unconditional write replaces a producer-written __test_report_v2.json. When no XML exists, read_results yields the synthetic no-results result (or an empty sequence), so the direct JSON producer path documented in RESULTS_FORMAT.md cannot work if the reporter also runs. Check for an existing v2 file before writing it, or add JSON-input handling.

        _helix_compat.JsonReporter(azdo_parameters, log=log).report_results(all_results)

src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.py:91

• This catches every exception from the legacy reporter, including Azure DevOps upload failures, and lets the work item exit successfully. With helix-scripts present, existing consumers rely on that legacy report, so a failed upload can leave tests unreported without failing the pipeline. Keep legacy reporting failures observable/fatal while making only the missing-package case optional.

        except Exception:
            log.exception("Legacy pickle reporter failed; continuing to write JSON results")

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.