[release/8.0] Helix reporter: add portable JSON output (#16774) - #17383
Open
wfurt wants to merge 1 commit into
Open
[release/8.0] Helix reporter: add portable JSON output (#16774)#17383wfurt wants to merge 1 commit into
wfurt wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds portable JSON test-result output while retaining legacy pickle reporting when helix-scripts is available.
Changes:
- Adds compatibility types and a versioned JSON reporter.
- Makes Helix imports optional across parsers and runner.
- Documents the JSON schema and file location.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/test_results_reader/init.py | Updated as part of this pull request. |
| src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.py | Updated as part of this pull request. |
| src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/RESULTS_FORMAT.md | Updated as part of this pull request. |
| src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/formats/xunit.py | Updated as part of this pull request. |
| src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/formats/trx.py | Updated as part of this pull request. |
| src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/formats/result_format.py | Updated as part of this pull request. |
| src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/formats/junit.py | Updated as part of this pull request. |
| src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/_helix_compat.py | Updated as part of this pull request. |
Suppressed comments (4)
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/RESULTS_FORMAT.md:17
- Although
_results_dir()falls back to the current directory,run.pystill callsget_env("HELIX_WORKITEM_UPLOAD_ROOT")before reachingJsonReporter. Thus the documented ad-hoc invocation withHELIX_WORKITEM_ROOTunset and no upload-root variable exits without writing the promised cwd file. Make that environment variable optional for this path or remove the ad-hoc fallback claim.
If `HELIX_WORKITEM_ROOT` is unset (e.g. ad-hoc local invocation), the
file is written to the current working directory.
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/_helix_compat.py:167
- When no XML files are found and
_commandExitCodeis0,read_resultsemits the synthetic pass whosefailure_messageis built asu'{}'.format(None), i.e. the string"None". This serializer therefore emits a non-null failure message for a passing result, contradicting the schema and potentially confusing consumers; fix the synthetic result or normalize this value before serialization.
"failure_message": self._failure_message,
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.py:103
read_resultsis a generator, and the legacy reporter consumesall_resultsat line 89. Consequently, on machines withhelix-scriptsinstalled this call receives an exhausted iterator and writesresults: [], so the new JSON output loses every parsed test. Materialize the results once before passing them to both reporters, while preserving the existingNonefiltering.
_helix_compat.JsonReporter(azdo_parameters, log=log).report_results(all_results)
src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.py:103
- The new documentation recommends emitting
__test_report_v2.jsondirectly and says the reporter is a no-op when that file already exists, butread_resultsonly discovers the XML formats and this call unconditionally rewrites the JSON file. A direct producer's results are therefore discarded (or replaced by the synthetic no-results result); either implement loading/preserving the existing JSON or remove this recommendation.
_helix_compat.JsonReporter(azdo_parameters, log=log).report_results(all_results)
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+34
to
+37
| Recommended use: write `__test_report_v2.json` to `$HELIX_WORKITEM_ROOT` | ||
| from your test runner's own reporting hook, then either omit | ||
| `EnableAzurePipelinesReporter` or let it run — the reporter is a no-op | ||
| on a missing/empty results XML when the JSON file already exists. |
Comment on lines
+257
to
+258
| with open(path, "w", encoding="utf-8") as f: | ||
| json.dump(payload, f, ensure_ascii=False) |
Comment on lines
+90
to
+91
| except Exception: | ||
| log.exception("Legacy pickle reporter failed; continuing to write JSON results") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #16774 to release/8.0.
Description
The current format uses Python's Pickle that is not portable. The reporter also depends on helix scripts that may or may not be present anymore. This change preserves the existing Pickle serialization when possible but makes it optional. It adds portable JSON serialization that can be consumed by anyone in any language.