Use complog for source indexer - #17317
Conversation
There was a problem hiding this comment.
Pull request overview
Updates the Arcade source indexing Stage 1 pipeline templates to generate a .complog from the build binlog (via complog) and upload Stage 1 artifacts, replacing the prior binlog-to-sln processing approach.
Changes:
- Replace
BinLogToSlnusage withcomplog createto producebuild.complogfor Stage 1 output. - Add
runAsPublicplumbing and a dedicated public package source for fetching thecomplogtool. - Make the “Build Repository” step conditional when
sourceIndexBuildCommandis empty.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| eng/common/core-templates/steps/source-index-stage1-publish.yml | Switch Stage 1 processing to complog and adjust tool acquisition/feeds. |
| eng/common/core-templates/job/source-index-stage1.yml | Adjust job behavior (conditional build step, pass-through parameters, job-level settings). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - job: SourceIndexStage1 | ||
| dependsOn: ${{ parameters.dependsOn }} | ||
| condition: ${{ parameters.condition }} | ||
| continueOnError: true |
There was a problem hiding this comment.
I think source index stage should not block official builds by default. But happy to change this if others disagree.
| sourceIndexProcessBinlogPackageVersion: 1.0.1-20260521.2 | ||
| runAsPublic: false | ||
| sourceIndexUploadPackageVersion: '2.0.0-20260521.2' | ||
| sourceIndexComplogPackageVersion: '0.9.50' |
There was a problem hiding this comment.
Why did you pick a fixed package version here vs. letting it float to latest?
There was a problem hiding this comment.
Just continuing what it was doing previously for the BinlogToSln tool. I think the reason was that pinned versions are more deterministic and avoid sudden breaks across all pipelines.
However, with my other change to make this stage not fail the whole build, I guess it would be more okay to use a floating version here.
What kind of version would you suggest? Just * or 0.9.*...? (I'm not sure what's complog's versioning scheme)
| - script: | | ||
| mkdir ".source-index/stage1output" | ||
| git rev-parse HEAD > .source-index/stage1output/hash | ||
| $(Agent.TempDirectory)/.source-index/tools/complog create ${{parameters.BinlogPath}} -o .source-index/stage1output/build.complog |
There was a problem hiding this comment.
How does this work when a repository produces multiple binlogs? That is what say VMR would do.
There was a problem hiding this comment.
This infra doesn't currently support that (and never did). VMR doesn't use this, it has its own logic, specifically it has a loop:
|
@tannergooding can you please take a look? thanks |
tannergooding
left a comment
tannergooding
left a comment
There was a problem hiding this comment.
LGTM, I think allowing the version to float makes sense especially with failure to source index no longer blocking
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (2)
eng/common/core-templates/steps/source-index-stage1-publish.yml:19
- Because this template accepts a custom
poolandscriptruns under Bash on Linux agents, the unquoted*rendered from the default version is expanded against$(Agent.TempDirectory)(which contains the newly installeddotnetdirectory) beforedotnetsees it. The install then receives--version dotnetinstead of the intended floating version; quote the value or use a pinned/omitted version so Linux-based consumers can install the tool.
$(Agent.TempDirectory)/dotnet/dotnet tool install complog --version ${{parameters.sourceIndexComplogPackageVersion}} --source ${{parameters.sourceIndexPublicPackageSource}} --tool-path $(Agent.TempDirectory)/.source-index/tools
eng/common/core-templates/steps/source-index-stage1-publish.yml:4
- Using
*makes the default restore a floatingcomplogversion, so a newly published release can change or break every source-index run without a template change. This violates Arcade's servicing requirement that each build tool have a specific version (Documentation/Servicing.md:10); pin the exact version validated for this template and update it explicitly.
sourceIndexComplogPackageVersion: '*'
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
eng/common/core-templates/steps/source-index-stage1-publish.yml:4
- Using
*makes the complog tool version non-reproducible and violates Arcade's servicing requirement that every build-support tool use a specific version (Documentation/Servicing.md:10). A future complog release can change indexing behavior or break this pipeline without a repository change; pin this to an approved exact version and roll it forward deliberately.
sourceIndexComplogPackageVersion: '*'
To double check:
Validated on dotnet/extensions - the currently published stage1 artifacts of dotnet/extensions are complog-based and published on https://source.dot.net. Official build run with this change is https://dev.azure.com/dnceng/internal/_build/results?buildId=3055692&view=results