[DRAFT] Windows HW baked install.wim pipeline (Packer, Azure build host) - #830
Draft
markcor wants to merge 149 commits into
Draft
[DRAFT] Windows HW baked install.wim pipeline (Packer, Azure build host)#830markcor wants to merge 149 commits into
markcor wants to merge 149 commits into
Conversation
…d host) Add provisioners/windows/nuc-wim/ — a scalable, config-driven pipeline that bakes a golden NUC install.wim from a BYO base WIM (nested Hyper-V + Packer), and treat pools.yml versions as a minimum in the MDC1 deploy bootstrap so a pre-baked WIM skips reinstall. - config/<image>.yaml (+ nuc-wim-defaults.yaml, "default" resolution) — one file per WIM; scalable across OS version and specialized worker pool. - bin/NucWim/New-NucWim.ps1 — orchestrator (prep -> build -> publish), per-image namespaced outputs (captured/<image>/<image>-<buildid>.wim). - nuc-wim.pkr.hcl + scripts/ — prepare-base-vhdx, register-base-vm, bake-bootstrap (openvox/puppet/git from ronin assets), sysprep-generalize, capture-wim, download/upload-wim (Entra auth), publish-wim. - New-NucWimBuildVm.ps1 + scripts/bootstrap-build-host.ps1 — provision the Azure nested-virt build host (managed identity -> Storage Blob Data Contributor, Hyper-V + Packer + ADK + azcopy). - MDC1Windows/bootstrap.ps1 Get-PreRequ: detect installed versions, treat pools.yml as a minimum (skip reinstall when the baked WIM already satisfies it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y the GHA) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bake) The pipeline bakes install.wim for any Windows hardware pool, not just NUC, so drop the nuc- naming from files/identifiers and use win-hw-wim / WinHwWim: - provisioners/windows/nuc-wim/ -> provisioners/windows/win-hw-wim/ - bin/NucWim/New-NucWim.ps1 -> bin/WinHwWim/New-WinHwWim.ps1 - New-NucWimBuildVm.ps1 -> New-WinHwWimBuildVm.ps1 - config/nuc-wim-defaults.yaml -> config/win-hw-wim-defaults.yaml - nuc-wim.pkr.hcl -> win-hw-wim.pkr.hcl - ci/kickoff-nuc-wim-build.ps1 -> ci/kickoff-win-hw-wim-build.ps1 Deployed Azure resource names (nucwimfxci, rg/vn/sn-central-us-nuc-wim) are kept as-is (can't rename in place). Factual NUC-hardware references (NUC13 drivers, canary NUC, deploy target) are intentionally left. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tion name
- config: base.index -> base.edition (edition NAME); prepare-base-vhdx resolves
it to the image index via Get-WindowsImage (fails with the available list if the
name isn't found).
- config: add drivers.{inject,cab_url}, default inject=false. When true,
prepare-base-vhdx downloads the cab, expands it, and DISM /Add-Driver /Recurse
into the applied image before capture (drivers land in the golden WIM).
- New-WinHwWim.ps1 passes -Edition and, when enabled, -InjectDrivers/-DriverCabUrl.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…297) Pin base.ronin.hash so the bake checks out a known ronin commit instead of moving branch HEAD; bump when the bake role branch advances. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tId convention Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Throwaway variant of win11-24h2-hw for validating the pipeline/bake without touching the production output: own capture namespace (captured/win11-24h2-hw-test/), test worker_pool_id, and tracks the bake-role branch HEAD (hash: "") for iteration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ver bool - bootstrap.ps1 Get-PreRequ: the new post-install version asserts now fall back to the bin-dir existence check, so a registry DisplayName detection miss can't fail a deploy where the agent/git actually installed (agent + git). - New-WinHwWim.ps1: parse drivers.inject robustly (quoted 'false' was truthy under [bool]); now only true/1/yes enables injection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ot retry - New-WinHwWim.ps1: fail fast if base.wim / base.edition / ronin.bake_role missing, or drivers.inject set without cab_url. Empty edition previously fell through to index 1 silently (wrong edition). - New-WinHwWimBuildVm.ps1: after the Hyper-V reboot, wait + retry the phase-2 run-command (guest agent may not be ready immediately). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Rewrite STORAGE-DESIGN.md to the Entra-only posture (open network, keys disabled, RBAC-gated; managed identity for the build VM; PR #313), replacing the stale Tier-1/deny-by-default/service-endpoint description. - download-wim.ps1 / upload-wim.ps1: drop 'firewall must allow your IP' comments; state the Entra identity + Blob Data role requirement instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Align assignments/comments in win-hw-wim.pkr.hcl + example.pkrvars.hcl so the repo's pre-commit packer_fmt hook passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… MI fallback) azcopy has its own credential store and does NOT inherit 'az login', so 'azcopy --auth-mode login' would fail headless on the build VM despite the MI grant. Fixes: - download/upload-wim.ps1: default AZCOPY_AUTO_LOGIN_TYPE=AZCLI so azcopy reuses the az CLI identity (managed identity on the VM, SP, or user). - New-WinHwWim.ps1: if no SP creds and nothing is logged in, 'az login --identity' so the headless build (scheduled task) authenticates as the VM's managed identity. This is what lets the prebake read base/ and write captured/ on nucwimfxci. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Support the workflow spinning the build VM up/down per run: - ci/win-hw-wim-vm.ps1 (-Action create|destroy): create a nested-virt VM (no public IP/NSG; driven via run-command), grant its managed identity Storage Blob Data Contributor, bootstrap Hyper-V+tooling; destroy removes the VM + OS/data disks + NIC + the role assignment (idempotent/best-effort for if: always() teardown). - scripts/run-build-task.ps1: on-VM wrapper that runs New-WinHwWim.ps1, tees to build.log, and writes build.done=<exitcode> when finished. - kickoff-win-hw-wim-build.ps1: rewritten to start the detached build task then POLL build.done to completion (the build exceeds the ~90-min run-command limit), stream the log tail, and exit with the build result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… role grant) The ephemeral VM now attaches the Terraform-managed UAMI (id-central-us-wim-builder, pre-granted Blob Data Contributor) instead of a system-assigned MI + per-run role assignment. This means the workflow SP needs no role-assignment rights at runtime. Teardown no longer removes a role assignment (UAMI is persistent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A function named 'Az' shadowed the 'az' CLI (PowerShell is case-insensitive), so every call recursed until 'call depth overflow' — the Create VM step failed on it. Resolve the real az executable once ($azExe = Get-Command az -CommandType Application) and invoke that. Also fix destroy's best-effort az calls to use $azExe (not the throwing wrapper) and pass empty pip/nsg as "" for pwsh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Create VM failed with 'Unable to find type [System.Web.Security.Membership]' on the ubuntu runner. Replace GeneratePassword with a Guid-based portable password (meets Azure complexity) in win-hw-wim-vm.ps1, New-WinHwWimBuildVm.ps1, and New-WinHwWim.ps1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
az vm create used the long run-scoped VM name as the Windows computer name -> InvalidParameter (>15 chars). Derive a <=15-char alphanumeric computer name from the VM name. (Empty pip/nsg + UAMI attach confirmed working in the failed run.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of the hung build: az vm run-command --parameters didn't populate bootstrap-build-host.ps1's Mandatory -Phase, so it errored and installed nothing (no Hyper-V, no choco/git/packer) — yet run-command returns exit 0, so Create VM falsely "succeeded" and the kickoff then polled a VM where nothing ran. - bootstrap-build-host.ps1: drop the param; read $Phase (set by a prepended --scripts line); hard-fail on Hyper-V/choco/package errors; verify tools; refresh PATH; print BOOTSTRAP_PHASE_OK sentinel. - win-hw-wim-vm.ps1: pass phase via `$Phase='...'` script line and ASSERT the sentinel (run-command hides inner failures). - kickoff: refresh PATH before git, verify checkout, print KICKOFF_OK + fail fast. - run-build-task.ps1: refresh PATH so packer/az/azcopy resolve under the task. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eate cleanup) Add a name-prefix sweep to the destroy action so resources az creates before a failed 'vm create' (e.g. <VmName>VMNic) are removed even when the VM never existed and 'az vm show' finds nothing. Idempotent/best-effort. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The bootstrap phase produced an empty message (Create failed 'did not succeed:')
because $boot used backslashes ('..\provisioners\...') which don't resolve on the
Linux GH runner, so 'az --scripts @$boot' couldn't read the file. Use a
forward-slash path, Test-Path it, and inline ($Phase line + file body) as one
--scripts value instead of the fragile literal+@file mix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase failures showed 'did not succeed:' with an empty message because a thrown error lands in stderr (value[1].message) while we only queried value[0] (stdout). Query join of all value[].message so the real error and the sentinel are both seen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…5 + D96ads_v5 Root cause of the persistent Hyper-V (create) failure: new Gen2 VMs default to Trusted Launch, which is INCOMPATIBLE with nested virtualization, so the Hyper-V role couldn't function. Fixes (researched): - az vm create --security-type Standard (disable Trusted Launch). - Image -> Windows Server 2025 gen2 PLAIN (2025-datacenter-g2); azure-edition pushes Trusted Launch and is deprecated. - SKU -> Standard_D96ads_v5 (AMD EPYC, 96 vCPU/384 GiB, nested-virt capable, larger for a faster bake; confirmed unrestricted in centralus). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
az vm create failed QuotaExceeded: standardDADSv5Family limit is 64 cores in centralus; D96ads_v5 needs 96. Use D64ads_v5 (64 vCPU/256 GiB), the largest that fits. Security-type Standard + WS2025 gen2 + AMD nested-virt were all accepted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The bootstrap failed to PARSE (so no phase ran) on line 42: "as $DataDriveLetter:"
— PS reads the ':' as a scope/drive qualifier. Delimit with ${DataDriveLetter}:.
Scanned all pipeline scripts; no other $var: hazards. The fail-loud + stderr
capture correctly surfaced the exact file:line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ISM is native) Phase 1 (Hyper-V) + packer/azcopy/git/azure-cli all installed successfully; the only failure was choco 'windows-adk-deploymenttools' (package not found). The pipeline uses the built-in DISM cmdlets (Expand-WindowsImage/New-WindowsImage), so no ADK is needed. Verify step still checks native dism. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Install-Module hangs forever in a non-interactive az vm run-command session, prompting to bootstrap the NuGet provider and trust the PSGallery repo. Phase 2 reached this line for the first time (previous runs died earlier at the ADK choco step) and hung ~34min. Pre-install the NuGet provider and mark PSGallery trusted so nothing prompts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The here-string emitted `"`$task`" which produces plain double-quotes around
$task in the generated script, terminating the outer $arg string early and
leaving $task as a bare token (ParserError on the VM). Emit doubled quotes
(""), which PowerShell reads as a literal quote inside the double-quoted
string while $task still expands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…es leaked into $Image) The scheduled-task -Argument is parsed by powershell.exe -File via CommandLineToArgvW, which strips double quotes but keeps single quotes literal. -Image 'win11-24h2-hw-test' therefore set $Image = "'win11-24h2-hw-test'", so New-WinHwWim looked for config\'win11-24h2-hw-test'.yaml and threw 'Config not found'. Inputs are already validated to [A-Za-z0-9._/-]+ (no spaces), so pass them unquoted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d VM The build VM is attached only a user-assigned managed identity (no system-assigned), so bare 'az login --identity' fails with 'Please run az login'. Thread the UAMI client id through: kickoff resolves it (runner is az-authed) -> run-build-task -> New-WinHwWim runs 'az login --identity --username <clientId>' and verifies an active account. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…un 31428853582 The win11-24h2-hw bake (run 31428853582) reached a clean puppet apply (rc=2) and then died in the post-bake windows-restart provisioner: "A system shutdown is in progress.(1115)" -> "Timeout waiting for machine to restart" at the 30m cap, after 1h41m of work. win11-24h2-hw is the only config with windows_update: true, so it is the first image whose reboots run the WU "Working on updates" apply pass through shutdown AND boot -- on a nested-virt guest whose reboots also land as a full power-off that the host watchdog has to restart. 30m was not enough for that; both windows-restart provisioners now allow 60m, and the kickoff poll cap goes 180 -> 240 min to keep headroom above the larger restart budget. The retry then wasted the run and hid the cause: the clone-flake classifier matched a bare 'StepCloneVM' anywhere in the log, but Packer prints `aborted: skipping cleanup of step "StepCloneVM"` on EVERY step failure. So the restart timeout was retried as a "clone flake", and attempt 2 died in 3 minutes on the genuine Enable-VMIntegrationService host flake -- which is what the run reported. Match `Step "<name>" failed` instead, for StepCloneVM and StepEnableIntegrationService (same retryable host-flake family). Also give the boot watchdog a log (boot-watchdog.log: state transitions + every Start-VM it issues), tailed to the console when the build fails. It runs as a background job, so this is the only way to tell a "machine never came back" timeout (watchdog restarted it, WinRM stayed dead) from one where the VM was Running the whole time and the watchdog was never involved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Bake-time patching fights the bake role. win116424h2hwbake includes roles_profiles::profiles::disable_services -> win_disable_services::disable_windows_update, which disables wuauserv, so the windows_update: true order was: WU installs updates -> reboot -> puppet disables the update stack -> reboot and that last reboot never came back: "A system shutdown is in progress.(1115)" -> "Timeout waiting for machine to restart", losing the whole 1h41m of run 31428853582 after a clean puppet apply. The (1115) says something had already started a shutdown when Packer issued its restart, which fits pending servicing work completing after puppet crippled the service that manages it. win11-24h2-hw-test bakes the SAME role from the SAME branch onto the SAME base WIM with windows_update: false and has gone green repeatedly, so WU was the only difference between the green bakes and this failure. Accepted consequence: the WIM ships at base_wim's patch level, and the deployed workers never patch themselves anyway (that same puppet class keeps WU disabled in production), so refreshing patches means newer base media. Also saves ~40 min per bake and makes image content deterministic rather than build-date dependent. If bake-time patching is wanted back, do it OFFLINE - a pinned SSU+LCU .msu via DISM /Add-Package in prepare-base-vhdx.ps1, which already does offline driver injection - not via the online WU provisioner. The 60m windows-restart timeouts from 1876078 stay as margin: reboots on this nested-virt guest are slow regardless. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d SolarWinds tail Debugging run 31428853582 meant waiting two hours to see a 200-line tail, and the lines that mattered (the WU pass, the first restart) were above the cutoff. On a hang it is worse than that: the job times out, the VM is destroyed, and the full build.log dies with it. run-build-task.ps1 now pushes the WHOLE build.log to _status/<image>.live.log every 60s from a background job, plus a final complete push just before the .done marker so the closing delta includes the failure. It reads the log with FileShare::ReadWrite because Tee-Object holds it open, and every error is caught and retried - streaming must never affect the build. kickoff-win-hw-wim-build.ps1 downloads that blob each poll and prints only the new lines, so the GH log follows the bake in real time. Same freshness gate as the .done marker, so a previous run's log can't replay. The 200-line tail is kept as a fallback for when live streaming produced nothing. It also tails SolarWinds. The bake guest ships as nuc-bake once puppet's logging profile starts nxlog (win_nxlog::service ensure => running), which I verified against the 20260811-164648 bake. Two caveats worth knowing: it only covers the final boot + sysprep window (nxlog isn't up before then), and the failed run shipped NOTHING at all - so this is a heartbeat, not a substitute for the live log above. Per request it carries only our own script output: programs are allowlisted to nxlog/BootStrap/puppet/ronin/maintainsystem and the OS event-log noise is dropped, as is nxlog's chatter about the generic-worker logs that never exist on a bake VM (that pair takes a green bake from 59 lines to 6 useful ones). Both filters are env-overridable and the dropped count is always reported, so nothing vanishes silently. With no SOLARWINDS_API_TOKEN the tail prints one notice and skips - it is a diagnostic, never a build gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…es do The baked WIMs shipped with no inventory while every Azure gallery image gets one. This gives the WIM bake the same three steps azure.pkr.hcl uses - drop the BootStrap module into the guest, call Set-ReleaseNotes, download the markdown - and routes the result to the same place: .github/workflows/upload-release-notes.yml commits it to sboms/ on main. Naming matches the gallery convention, <config>-<version>.md, which here resolves to <image>-<buildid>.md - the same stem as the WIM itself. Published twice: beside the WIM in captured/WIMs/<image>/ as provenance, and to _status/sbom/ where the runner can find it by prefix (it never knows the build id, which New-WinHwWim generates). DeploymentId carries the ronin hash, our equivalent of the gallery images' deploymentId - it pins the puppet content the image was baked from. Four shared BootStrap functions needed a fix first, or this would have taken the bake down with it. Get-GenericWorkerVersion, Get-LiveLogVersion, Get-WorkerRunnerVersion and Get-ProxyVersion all Start-Process their binary with no existence check, and Start-Process on a missing path throws a TERMINATING error that Set-ReleaseNotes' trap rethrows. The win-hw-wim bake role excludes windows_worker_runner on purpose - generic-worker and worker-runner arrive at DEPLOY time - so all four would have thrown, failing the build immediately before Sysprep and losing a ~2h bake. They now return nothing when the binary is absent, and Show-TaskclusterBinaries drops the resulting blank rows. No behaviour change for the gallery images, where the binaries are present. Verified the missing-binary path: 2-of-4 present yields a clean 2-row table, 0-of-4 yields an empty section rather than an error. Known coverage gap, inherent rather than introduced: the notes inventory what the WIM ships, so the Taskcluster binaries installed at deploy time are absent by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pture SolarWinds only ever covered a bake's final boot + sysprep window (~6 useful lines), and the run we actually needed it for shipped nothing at all, because nxlog isn't up before then. Its one real advantage over the live build log was being a PUSH channel that survives a dead guest network - and that advantage never materialised. Since the destination doesn't matter as long as the logs reach the GH job, this trades it for something that covers the case SolarWinds was supposed to. PowerShell Direct talks over the Hyper-V VMBus, so it needs neither network nor WinRM: it works precisely when Packer has gone blind. The boot watchdog now notices when Packer's log stops growing for 10 minutes while the VM is Running, and captures the guest's uptime, pending-reboot/servicing state, and last 20 minutes of System+Application events. That is exactly the question run 31428853582 left unanswered for 30 minutes: applying updates, or wedged? Capped at 6 captures so a dead build can't fill the log, and the cap is logged rather than silently applied. Stalls while the VM is Off are skipped - the watchdog restarts it instead, and PS Direct can't reach a powered-off guest. The capture streams live: the watchdog log now goes to run-build-task's dir, which uploads it as a second blob, and the kickoff tails it with a [guest] prefix. So it shows up WHILE a hang is happening, not after Packer gives up. Both blob tails now share one Show-LogDelta, since the SolarWinds-specific polling is gone. Verified the stall logic against a simulated clock: growth resets the timer, quiet under the threshold does nothing, 11 min quiet while Running captures, quiet while Off does not, resumed progress resets, and the cap holds at 6. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first live-streamed run (31531731106) put 12,884 lines in the job log, and 11,137 of them were az success payloads - the actual build output was 1,329. --only-show-errors suppresses warnings, not the result object, so every poll of every stream dumped ~40 lines of blob properties. Add -o none to the blob download/upload calls that don't already set an output format. Worst offender was the per-poll download in Show-LogDelta: two streams times ~80 polls. The pre-existing .done and tail downloads had the same flaw but ran once each, so it never showed until streaming made it per-minute. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moves the pool off the last -hw-test WIM (win11-24h2-hw-test-20260805-193514) and onto win11-24h2-hw-20260811-202701, built by run 31531731106 from the production win11-24h2-hw config - the first bake with Windows Update off and the first to publish release notes (sboms/win11-24h2-hw-20260811-202701.md). Only the image changes. relops1213 keeps the -hw-test WIM; its worker client is still under-scoped, so it can't validate anything either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5230f32 predates the image the pool now deploys. 48a8b9d4 is what win11-24h2-hw-20260811-202701 was baked from (win11-24h2-hw.yaml's pin, and the DeploymentId recorded in its release notes), so deploy-time puppet starts from the same content the WIM was built with. Aligning them is a convenience for this canary, not a rule: the WIM is a base layer and the worker takes incremental ronin changes on top, so this hash is expected to move ahead of the bake hash over time. A difference here is normal and not a defect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The branch's node lists were ~85 commits stale, and the branch's OS-deploy.ps1 refreshes pools.yml from the branch - so a canary deploy would have acted on membership that predates the PSU-swap work done on main. Node blocks are copied from main verbatim; each pool's settings and comments are left as they were. perf-debug therefore keeps the canary fields (dev, image, src_Branch, hash) over main's current nodes: nuc13-158, 074, 115 rather than the stale 024, 059, 119. relops1213 is left entirely alone as requested, which leaves nuc13-159 and nuc13-160 listed BOTH there and in main's win11-64-24h2-hw. Node -> pool lookup is ambiguous for those two until relops1213 is reconciled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…erlap All seven pools now carry main's node distribution. This removes the last duplicate: nuc13-159 and nuc13-160 were listed both here and in win11-64-24h2-hw, and relops1213 won outright - it sits third in the file and listed them first, while win11-64-24h2-hw has them at index 92 of 94 and never gets that far, because every later pool breaks on its first node once a match is recorded. Either node would have re-imaged onto the stale -hw-test WIM instead of the production image main assigns it. Its deployment details are left as they are; those nodes have dead PSUs and aren't being re-imaged for weeks, so image/hash/dev here are moot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
image win11-24H2-NUC-01-16-2025 -> win11-24h2-hw-20260811-202701, matching perf-debug. main's ref-alpha already carries the dev: nuc-wim-pipeline trigger, so the deploy reaches this branch's OS-deploy and its Apply-Image path. Only the image changes; src_Branch/hash are left as they are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
src_Branch RELOPS-2195-thermal -> wim-bake-role, hash e47164b -> 48a8b9d4, so ref-alpha runs the same ronin content as perf-debug while testing the same WIM. This keeps the benchmarking: wim-bake-role is 43 commits AHEAD of RELOPS-2402-fleetbench and already carries both fleetbench commits, cherry-picked (06badfa5 detector, 48a8b9d4 gw_exe_check skip), along with the win_fleetbench module. Nothing benchmark-related is lost by moving off the thermal branch. It also picks up the canary's deploy-side fixes that the thermal branch lacks - notably maintainsystem-hw's drift check reading pools.yml from the dev branch rather than main, which is what stops a hash mismatch turning into a Set-PXE re-image loop. Branch stays wim-bake-role; the hash moves when ronin does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The baked AppXSvc disable defeated the ref/ref-alpha codec exemption (bug 2013985): the extensions stayed provisioned but never registered for the per-task user, so mochitest-media-mda-gpu failed with ffvpx software fallback instead of the WMF hardware decoders. b64eb0b5 skips disable_appxsvc during a bake and skips the 600s Wait-AppxIdle when the image is already debloated, so the WIM ships AppXSvc Manual at no deploy-time cost. Bake config only -- pools.yml is untouched, so no deployed node changes until its hash pin is moved separately.
New golden WIM from bake run 32408074628, baked from ronin wim-bake-role b64eb0b5 (release notes DeploymentId confirms it). That commit stops the bake disabling AppXSvc, so this image ships the service Manual and the per-task user can register the HEVC/AV1/VP9/WebMedia extensions -- which is what mochitest-media-mda-gpu needs to stop falling back to ffvpx software decode. Both pools' ronin hash moved to b64eb0b5 as well, deliberately. ref-alpha is exempt from disable_appxsvc by pool name so it would be fine either way, but perf-debug is NOT exempt: on the old pin its deploy-time run would disable AppXSvc using the pre-fix uninstall script, which with the service enabled in the image can block up to 600s in Wait-AppxIdle. b64eb0b5 carries the pre-check that skips that wait on an already-debloated image. Does NOT include the chipset/platform driver pack -- the 10 error-28 devices (DPTF, SMBus, Serial-IO, SST audio, GNA, Wi-Fi, Bluetooth) are still open.
7dac7f0a supersedes b64eb0b5. Stopping the bake from disabling AppXSvc fixed the codecs but changed sysprep: the WIM got a full generalize, so the first-boot specialize pass regenerated a random WIN-xxxxxxxx into ActiveComputerName and maintainsystem-hw PXE-looped on the unrecognised name. The bake never needed changing -- the media extensions are provisioned at bake time (DISM-level, no AppXSvc), and only per-user registration at first task_* logon needs the service. 7dac7f0a bakes the disable again and re-enables AppXSvc at deploy time for ref/ref-alpha instead. pools.yml is deliberately NOT touched: the deployed nodes still carry GITHASH b64eb0b5, and moving the pool hash before they are re-imaged would make the drift check mismatch and PXE them all over again.
….cmd The offline rename in WinPE is necessary but not sufficient. The first-boot specialize pass runs after it and regenerates a random WIN-xxxxxxxx into ActiveComputerName -- the unattend's <ComputerName> does not take effect on this image, which is why the offline write exists in the first place. Observed on nuc13-158: ComputerName=NUC13-158 but ActiveComputerName=WIN-D81J5HC82S0, with Tcpip Hostname/NV Hostname still correct. That mismatch is not cosmetic. maintainsystem-hw looked the node up under the WIN- name, missed, and Set-PXE'd -- re-imaging into a fresh random name, forever. generic-worker's workerId reads the same value. SetupComplete.cmd is the first hook that runs after specialize/oobeSystem and before any logon, so it is the earliest point the name can be made authoritative. It is deliberately not a Rename-Computer: that cmdlet compares against the persistent name, which is already correct, so it refuses with 'the new name is the same as the current name'. Writing ActiveComputerName directly is the only thing that works -- verified on all five canary nodes. This makes the deploy robust to specialize running, rather than depending on it not running, which is the assumption that broke when the bake changed.
Replaces the fixed 'Have a nice day! :)' at both hand-off points with a random pick from Get-DeploySendoff. Cosmetic only; nothing parses this line.
New golden WIM from bake 32430019978 (ronin wim-bake-role 7dac7f0a): the bake disables AppXSvc again and the ref pools register the codec extensions at deploy time via win_disable_services::enable_appxsvc. image and hash move TOGETHER - maintainsystem-hw's drift check compares the pool hash against the node's GITHASH, so bumping one without the other re-images the fleet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Picks up the fix for the Get-AppxSnapshot return-value corruption that made win_uninstall_appx_packages.ps1 exit 1 on every node deployed from the win11-24h2-hw-20260820-235936 WIM: puppet 6 -> bootstrap Set-PXE -> re-image loop on all five canary nodes. Image is unchanged - the script is pulled from the ronin branch at deploy time, so no re-bake is needed. The pool hash now leads the hash the WIM was baked from (7dac7f0a), which is the expected base-layer-plus-increment arrangement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fix) Picks up the second drift checker being repointed at this branch's pools.yml. On f1eeb1fd the canary nodes were rebooting roughly hourly: self_redeploy_check compared their GITHASH against main's stale hash and restarted them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…phics Software) Adds an `extras.files` config list: URLs copied VERBATIM into the image at C:\bake\extras\ during prepare-base-vhdx. Not drivers - never expanded, never DISM-injected, never executed here. The bake's puppet apply runs them (ronin win_intel_graphics_software globs C:\bake\extras\gfx_win_*.exe). This is the missing half of the Intel Graphics Software fix. Production MDT nodes have IntelGraphicsSoftwareService; baked nodes do not, because it ships inside the AppUp.IntelArcSoftware MSIX, which under DCH arrives via Windows Update as a driver companion app - and WU is disabled by design, so the MU-catalog driver cab (INF only) can never supply it. Intel's full installer carries it at Resources/Extras/IntelGraphicsSoftware_26.18.2353.2_Release.exe. Why staging offline is required rather than letting the guest fetch it: the payload lives in hardwareimaging, which is Entra-only - an anonymous GET returns 409. Only the build host holds a managed identity; neither the packer guest nor a deployed NUC has one. So anything the bake must run in-guest has to be placed into the image from here. The download dispatch (azcopy AAD for *.blob.core.windows.net, Invoke-WebRequest otherwise) is factored out of the driver-injection block into Get-BakeAsset and shared, rather than duplicated. Behaviour of the driver path is unchanged. sysprep-generalize.ps1 already does `Remove-Item C:\bake -Recurse`, so extras are build-only and add nothing to the golden WIM. Configured for win11-24h2-hw only; defaults carry `extras.files: []` so every other image is unaffected. Both scripts AST_OK, all three configs parse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rebase of wim-bake-role rewrote aaf6203f away and both perf-debug and ref-alpha still pinned it. Get-Ronin does `git clone --single-branch --branch <branch>` then `git checkout <hash>`, and a single-branch clone carries no unreachable objects, so any re-image would have failed at checkout. ref-alpha also gains the `dev: nuc-wim-pipeline` trigger perf-debug already had and main carries on both. A no-op in practice - the recursion guard means the feature branch's own copy of `dev` is never read once the dev re-run is underway - but the two canary pools should be configured identically. Only perf-debug and ref-alpha are touched. ref, win11-64-24h2-hw and relops1213 keep their existing pins, which are deliberate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two independent reasons the old 7dac7f0a pin had to move:
a) wim-bake-role was rebased onto master on 2026-08-26 and 7dac7f0a no longer exists on
the branch. Get-Ronin does `git clone --single-branch --branch <branch>` then
`git checkout <hash>`, and a single-branch clone carries no unreachable objects, so
the bake would have failed at checkout.
b) it predates the two things this bake exists to deliver - win_device_guard (VBS +
HVCI, so baked nodes match the MDT image's Credential Guard + HVCI) and
win_intel_graphics_software (run against the installer that extras.files now stages
to C:\bake\extras).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ppable The puppet apply output WAS reaching the bake log, but unreadable: every line arrived space-interleaved, e.g. [ m N o t i c e : / S t a g e [ m a i n ] / W i n _ p a c k a g e s : : S e v e n z i p Windows PowerShell 5.1's Tee-Object has no -Encoding and writes UTF-16LE. The parent's Write-NewLog tails that file by seeking to a byte offset, so every read after the first starts mid-stream where there is no BOM, and its StreamReader fell back to UTF-8. The practical cost: the bake log could not be searched. After run 32996058864 there was no way to answer whether win_intel_graphics_software had provisioned the MSIX, because "intel_graphics_software" does not match "i n t e l _ g r a p h i c s". That is exactly the line we need to read to know whether a per-user-only install is about to be stripped by sysprep /generalize. Child script now writes the log itself with a StreamWriter in UTF-8 (no BOM), AutoFlush on so it still streams live instead of landing in one lump at the end. Write-NewLog decodes UTF-8 explicitly rather than relying on BOM detection it cannot see. Also passes --color=false, dropping the ANSI escapes that were littering the log. $LASTEXITCODE is captured into $rc immediately after the pipeline and written from there, so wrapping puppet in ForEach-Object cannot lose the detailed exit code. Verified by extracting the real here-string from this file, expanding it the way the runtime would, and parsing the result: AST_OK with correct quoting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
New golden WIM from bake 32996058864 (ronin 4c2e223d, DeploymentId confirmed in the SBOM). First WIM carrying win_device_guard (VBS + HVCI) and win_intel_graphics_software, and the first built with extras.files staging gfx_win_101.7088.exe to C:\bake\extras. hash is unchanged at 4c2e223d - the canary nodes already re-imaged onto that ronin and confirmed VBS is working (VirtualizationBasedSecurityStatus=2, HypervisorPresent=True on all five). This bump is the image only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…atch production A baked node has 10 devices in error 28 that the MDT production image binds cleanly; drivers.cabs only ever carried an Intel wired-NIC pack and the graphics cab, so there was zero chipset/System/SoftwareComponent/Bluetooth coverage. Adds two packs exported with Export-WindowsDriver -Online off the MDT reference nodes - nuc13-006 (NUC13 / Raptor Lake-P) and t-nuc12-005 (NUC12 / Alder Lake-P) - which is exactly the driver set production runs. Closes: Serial IO GPIO INTC1055 + I2C 51E8/51E9, SMBus 51A3 + SPI 51A4, Smart Sound audio 51CA/51C8, MEI 51E0, GNA A74F/464F, Wi-Fi 51F1/51F0 and Bluetooth PID_0033. A baked node's only sound device today is the Virtual Audio Cable. Both packs go in because one WIM serves both canary pools (perf-debug = NUC13, ref-alpha = NUC12); DISM stores every version and PnP binds per device. The DCH graphics family (iigd_dch, iigd_ext, hdbusext, cui_dch, igcc_dch, mshdadac) is pruned from both packs: the existing cab already supplies a complete and NEWER matched 32.0.101.7085 set, while the reference nodes carry 7079 / 31.0.101.3729, and mixing DCH graphics component versions gains nothing. Also corrects a comment: INTC1055 is Serial IO GPIO, not DPTF. Intel Dynamic Tuning is absent from the MDT image too, so it is not a baked-vs-prod delta. No pipeline code change needed - prepare-base-vhdx already fetches a .zip from hardwareimaging via azcopy AAD, expands each pack into its own subdir and runs a single recursive DISM /Add-Driver before capture. Ref: RELOPS-2487 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 33094857023 errored after 3m16s at StepStartVM with Error starting vm: Hyper-V\Start-VM : 'packer-nuc' failed to change state. The operation cannot be performed while the object is in its current state. Watchdog log for the same instant: 17:18:45 phase 1: waiting for Packer to start the VM 17:22:19 phase 2: restarting the VM on any guest-initiated power-off 17:22:19 VM state -> Off 17:22:19 VM is Off; issuing Start-VM <-- watchdog 17:22:37 packer: Error starting vm ... <-- Packer's own start, collided Phase 1 breaks on Packer's log line 'Starting the virtual machine', which Packer emits BEFORE its Start-VM completes. Phase 2 then polled immediately, saw the VM still Off, and started it itself; whichever Start-VM landed second failed. Nothing gated on having actually seen the VM run, so this was a coin flip on poll timing - which is why it presented as an intermittent host flake. Phase 2 now only restarts a VM it has positively observed Running. That is precisely what "guest-initiated power-off" means, so the post-specialize power-off this watchdog exists for is still covered (the guest boots and runs specialize for minutes against a 6s poll), while the watchdog becomes structurally incapable of fighting Packer's initial start. The driver-pack work in 3e9142d was unaffected and verified good in that run: all 4 packs downloaded and expanded and DISM /Add-Driver /Recurse reported "The operation completed successfully". Ref: RELOPS-2487 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drivers-baked WIM from run 33098637272 - adds the NUC13 + NUC12 platform/chipset packs (3e9142d), so the 10 error-28 devices the MDT image binds cleanly should now bind here too, including the Intel Smart Sound audio stack a baked node has been missing entirely. image only; hash stays 4c2e223d because ronin has not moved. Both canary pools move together, as always - a node whose GITHASH matches but whose image does not is fine, but splitting image/hash across the two pools is what causes mismatch-driven Set-PXE loops. WIM is staged on the deploy share. Ref: RELOPS-2487 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two canary pools are meant to differ only by hardware (NUC12 vs NUC13), but ref-alpha carried git_version 2.50.1 against perf-debug's 2.50.0. Now identical on every deployment field: image, org, repo, branch, hash, dev, domain suffix, puppet/openvox/git version. Inert in practice - the WIM bakes Git 2.54.0 and the deploy-time install is version-guarded, so neither value ever triggered an install. Aligned so a config diff between the two canary pools means something. secret_date deliberately NOT matched: it is per-pool secret rotation state (ref-alpha 02-24-2026, perf-debug 02-06-2026), not a deployment knob, and forcing them equal risks breaking secret retrieval for the pool. Ref: RELOPS-2487 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A config-driven pipeline that bakes a golden Windows HW
install.wimfrom a BYO base WIM (nested Hyper-V + Packer, run on an Azure VM), plus a deploy-side tweak so pre-baked images skip redundant reinstalls. Scalable to many WIMs across OS version and specialized worker pool.Lives under
provisioners/windows/win-hw-wim/.Contents
config/—win-hw-wim-defaults.yaml(shared defaults;"default"fields resolve here) + per-imagewin11-24h2-hw.yaml. Add a WIM = add a config file.bin/WinHwWim/New-WinHwWim.ps1— orchestrator:-Image <name>→ prep → build → publish, with per-image namespaced outputs (captured/<image>/<image>-<buildid>.wim).win-hw-wim.pkr.hcl+scripts/—prepare-base-vhdx,register-base-vm,bake-bootstrap(ronin bake role; openvox/puppet/git from the ronin assets blob),sysprep-generalize,capture-wim,download-wim/upload-wim(Entra auth),publish-wim.New-WinHwWimBuildVm.ps1+scripts/bootstrap-build-host.ps1— provision the Azure nested-virt build host (system-assigned managed identity → Storage Blob Data Contributor; Hyper-V + Packer + ADK + azcopy).ci/kickoff-win-hw-wim-build.ps1— kick off a build on the VM (used by workflow ci: NUC WIM build workflow (kick off on Azure build host) #829).MDC1Windows/bootstrap.ps1(Get-PreRequ) — treatpools.ymlversions as a minimum; skip reinstall when the baked WIM already satisfies it.Flow
Notes
nucwimfxciis Entra-only (no keys, no anonymous; RBAC-gated). Base WIMwin11-24h2-base-install.wimalready uploaded.nuc-wimlabel (can't rename in place); the pipeline naming is genericwin-hw-wim.🤖 Generated with Claude Code