From 754f59e5522b5975e430168b890ee72b78ecf84b Mon Sep 17 00:00:00 2001 From: Jochen Date: Fri, 29 May 2026 14:30:13 +0200 Subject: [PATCH] Make the Windows build work headlessly; add signing + one-command release build.py assumed a warm, interactive SDK; on a clean/headless host the Windows build failed in several ways. Fix them and add the missing pieces so the whole release is one command: release.ps1 (or `python build.py --package --publish`) builds, bundles, signs, packages, and publishes. build.py: - _sdk_bash: run the Git compile + MinGit steps via bash.exe -lc (synchronous, output streamed) instead of the detached git-bash.exe launcher; export a writable TMP/TEMP inside the build shell (headless shells have none, so the MinGW toolchain fell back to C:\Windows and failed); drop inherited GIT_EDITOR/EDITOR/VISUAL (GIT_EDITOR=true defeated MinGit release.sh). - ensure_sdk_sources: auto fetch + checkout usr/src/git and usr/src/build-extra on first build (no manual `sdk init`). - create the MinGit --output dir before release.sh runs. - prune_programs: also remove .exe variants (git-upload-pack.exe etc. were shipping on Windows). - package: write the .sha256 LF-only so `sha256sum -c` works for consumers. - bundle_less: ship less.exe + msys-pcre2-8-0.dll + terminfo (MinGit omits the pager); git defaults to less, apps pass `git -P` for captured output. - sign(): default WINDOWS_SIGN_SCRIPT to ./sign-windows.ps1; read [windows].sign_thumbprint from config.ini into the signing env. New files: - sign-windows.ps1: sign the dist's .exe/.dll via signtool using the SimplySign cloud cert (selected by thumbprint), SHA-256, RFC3161-timestamped at Certum. - release.ps1: verify the SimplySign cert is mounted, then run build.py (default --package --publish). Docs: - README: one-time host setup + the single-command release; drop the stale "anchorpoint MinGit flavor" prerequisite (it is just a version label). - config.example.ini: document the [windows] signing section. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 36 ++++++++--- build.py | 152 ++++++++++++++++++++++++++++++++++++++------- config.example.ini | 22 +++++-- release.ps1 | 37 +++++++++++ sign-windows.ps1 | 57 +++++++++++++++++ 5 files changed, 265 insertions(+), 39 deletions(-) create mode 100644 release.ps1 create mode 100644 sign-windows.ps1 diff --git a/README.md b/README.md index df4b4b9..8417735 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,9 @@ Each is a self-contained Git tree with the fork's `git-lfs` in ``` build.py # git + fork git-lfs -> dist// -> zip + .sha256 -config.example.ini # copy to config.ini; host-specific SDK / source paths +release.ps1 # Windows one-command release (checks SimplySign, runs build.py) +sign-windows.ps1 # signs the dist's .exe/.dll via signtool (SimplySign cert) +config.example.ini # copy to config.ini; host-specific paths + signing third_party/git-lfs/ # submodule -> the git-lfs fork .github/workflows/ci.yml # compile-checks the fork (no sign/publish) ``` @@ -36,19 +38,32 @@ third_party/git-lfs/ # submodule -> the git-lfs fork Clone with submodules (`git clone --recurse-submodules ...`), then: ```sh -cp config.example.ini config.ini # edit the paths -python build.py --package # --nosign to skip signing +cp config.example.ini config.ini # set the SDK path (Windows) / source path (macOS) +python build.py --package # add --nosign to skip signing python build.py --package --arch x86_64 # macOS: cross-build Intel ``` Output: `dist//` plus `dist/.zip` + `.sha256`. Paths may instead come from `GIT_SDK_PATH` / `GIT_LFS_PATH` / `GIT_SOURCE_PATH` (env wins over -`config.ini`). Signing is env-driven and a no-op when unset -(`MACOS_SIGN_IDENTITY`, `WINDOWS_SIGN_SCRIPT`). +`config.ini`). On Windows the SDK's `git`/`build-extra` sources are +auto-initialized on the first build, the `less` pager is bundled, and signing +defaults to `./sign-windows.ps1`. -**Prerequisites:** Go (per the fork's `go.mod`); on Windows the Git for Windows -SDK with the `anchorpoint` MinGit flavor; on macOS Xcode command-line tools and -a `git/git` source checkout at the target tag. +### First-time host setup + +One-time per machine (not per release): + +- **Windows** — install [Go](https://go.dev/dl/), the Git for Windows SDK + (`git-sdk-64`), the Windows SDK (for `signtool`), and SimplySign Desktop. Copy + `config.example.ini` to `config.ini` and set `[gitsdk].path`; the Git and + build-extra sources are fetched automatically on the first build. +- **macOS** — install Go and Xcode command-line tools, and check out `git/git` + at the target tag; set `[gitsource].path`. + +Signing is config/env-driven and a no-op when unset: macOS uses +`MACOS_SIGN_IDENTITY`; Windows uses `sign-windows.ps1` with the +`[windows].sign_thumbprint` cert (overridable via `WINDOWS_SIGN_SCRIPT` / +`WINDOWS_SIGN_THUMBPRINT`). ## Releasing @@ -58,8 +73,9 @@ machine and publish to a shared tag with `--publish` (needs the `gh` CLI authenticated with write access here): ```sh -# Windows, with the signing token in the environment: -python build.py --package --publish +# Windows — log into SimplySign Desktop first, then one command does it all +# (verifies the cert, builds, signs, packages, and publishes the release): +.\release.ps1 # == python build.py --package --publish # macOS — Apple Silicon, then Intel (notarytool leaves the bytes unchanged): python build.py --package --arch arm64 --publish diff --git a/build.py b/build.py index 7ad039f..54111aa 100644 --- a/build.py +++ b/build.py @@ -15,10 +15,13 @@ git-lfs fork defaults to the bundled third_party/git-lfs submodule. Usage: - python build.py [--package] [--nosign] [--arch arm64|x86_64] + python build.py [--package] [--nosign] [--publish] [--arch arm64|x86_64] + release.ps1 # Windows: verify SimplySign, then build+sign+publish Paths come from config.ini (copy config.example.ini) or env vars -GIT_SDK_PATH / GIT_LFS_PATH / GIT_SOURCE_PATH (env wins). +GIT_SDK_PATH / GIT_LFS_PATH / GIT_SOURCE_PATH (env wins). On Windows the SDK's +git/build-extra sources are auto-initialized on first build, and signing +defaults to ./sign-windows.ps1. """ from __future__ import annotations @@ -102,27 +105,48 @@ def build_gitlfs(gitlfs: str, dest: str, goos: str, goarch: str) -> None: # --------------------------------------------------------------------------- # # Git (Windows: MinGit via Git SDK) # --------------------------------------------------------------------------- # +def ensure_sdk_sources(gitsdk: str) -> None: + """Fetch + checkout the SDK's git and build-extra sources if not yet present. + + git-sdk-64 ships usr/src/git and usr/src/build-extra as repos with their + remote configured but no working tree (`sdk init` normally populates them). + Do it here with plain git (shallow main) so a freshly-unpacked SDK builds + without any manual steps. + """ + for rel in ("usr/src/git", "usr/src/build-extra"): + repo = os.path.join(gitsdk, rel) + if not os.path.exists(os.path.join(repo, ".git")): + raise SystemExit( + f"[git-dist] {rel} is not a git repo under {gitsdk}; " + "reinstall the Git for Windows SDK." + ) + if _capture(["git", "-C", repo, "rev-parse", "--verify", "HEAD"]).strip(): + continue # already checked out + print(f"[git-dist] initializing SDK source: {rel}") + if _run(["git", "-C", repo, "fetch", "--depth=1", "origin", + "+refs/heads/main:refs/remotes/origin/main"]) != 0: + raise SystemExit(f"[git-dist] failed to fetch {rel}") + if _run(["git", "-C", repo, "checkout", "-B", "main", "origin/main"]) != 0: + raise SystemExit(f"[git-dist] failed to checkout {rel}") + + def build_git_windows(gitsdk: str, dest: str) -> None: - src_git = os.path.join(gitsdk, "usr/src/git") - build_extra = os.path.join(gitsdk, "usr/src/build-extra") - if not os.path.exists(src_git) or not os.path.exists(build_extra): - raise SystemExit( - f"[git-dist] Git SDK not initialized. Open {gitsdk}/git-bash.exe and run " - "'sdk cd git && sdk init git' and 'sdk cd build-extra && sdk init build-extra'." - ) + ensure_sdk_sources(gitsdk) print("Building Git (Windows)") os.environ["PATH"] = f"{gitsdk}/usr/bin;{os.environ['PATH']}" - if _run_shell( - f'{gitsdk}/git-bash.exe -c \'cd {gitsdk}/usr/src/git; ' - f'make install CFLAGS="-O3 -DNDEBUG -Wno-error"\'' + if _sdk_bash( + gitsdk, + 'cd /usr/src/git && make install CFLAGS="-O3 -DNDEBUG -Wno-error"', ) != 0: raise SystemExit("[git-dist] git build failed") print("Building MinGit (anchorpoint flavor)") - if _run_shell( - f"{gitsdk}/git-bash.exe -c 'cd {gitsdk}/usr/src/build-extra/mingit; " - f"sh release.sh --output={gitsdk}/usr/src/build-extra/build anchorpoint'" + if _sdk_bash( + gitsdk, + "mkdir -p /usr/src/build-extra/build && " + "cd /usr/src/build-extra/mingit && " + "sh release.sh --output=/usr/src/build-extra/build anchorpoint", ) != 0: raise SystemExit("[git-dist] MinGit build failed") @@ -136,6 +160,30 @@ def build_git_windows(gitsdk: str, dest: str) -> None: zf.extractall(dest) +def bundle_less(gitsdk: str, dest: str) -> None: + """Bundle the `less` pager (MinGit omits it) so git can page output. + + less.exe needs three MSYS runtime DLLs; msys-2.0.dll and msys-ncursesw6.dll + already ship in MinGit, so only msys-pcre2-8-0.dll is added. The terminfo + database is copied too so ncurses can read terminal capabilities. git's + default pager is `less`; Anchorpoint disables paging per call with `git -P` + for programmatic (captured) output so it never blocks on the pager. + """ + usr_bin_src = os.path.join(gitsdk, "usr", "bin") + usr_bin_dst = os.path.join(dest, "usr", "bin") + os.makedirs(usr_bin_dst, exist_ok=True) + for name in ("less.exe", "msys-pcre2-8-0.dll"): + src = os.path.join(usr_bin_src, name) + if not os.path.exists(src): + raise SystemExit(f"[git-dist] cannot bundle pager: missing {src}") + shutil.copy2(src, os.path.join(usr_bin_dst, name)) + terminfo_src = os.path.join(gitsdk, "usr", "share", "terminfo") + terminfo_dst = os.path.join(dest, "usr", "share", "terminfo") + if os.path.isdir(terminfo_src): + shutil.copytree(terminfo_src, terminfo_dst, dirs_exist_ok=True) + print("Bundled less pager + terminfo") + + # --------------------------------------------------------------------------- # # Git (macOS: from source) # --------------------------------------------------------------------------- # @@ -175,9 +223,12 @@ def build_git_macos(git_source: str, dest: str, arch: str) -> None: def prune_programs(dest: str) -> None: for rel in PRUNE_PROGRAMS: for base in (dest, os.path.join(dest, "mingw64")): - p = os.path.join(base, rel) - if os.path.exists(p): - os.remove(p) + # On Windows these are .exe; PRUNE_PROGRAMS lists the bare names so + # the same list works for macOS. Try both so the server-side + # programs are actually removed on Windows too. + for p in (os.path.join(base, rel), os.path.join(base, rel) + ".exe"): + if os.path.exists(p): + os.remove(p) def patch_git_config(dest: str) -> None: @@ -232,11 +283,14 @@ def sign(dest: str) -> None: ]) print("macOS signing complete (notarization happens in CI after packaging)") elif platform.system() == "Windows": - script = os.environ.get("WINDOWS_SIGN_SCRIPT") - if not script or not os.path.exists(script): - print("[git-dist] WARNING: WINDOWS_SIGN_SCRIPT unset — skipping signing") + script = os.environ.get("WINDOWS_SIGN_SCRIPT") or os.path.join(ROOT, "sign-windows.ps1") + if not os.path.exists(script): + print(f"[git-dist] WARNING: sign script not found ({script}) — skipping signing") return - _run_shell(f'powershell -File "{script}" -folderPath "{dest}"') + _run_shell( + f'powershell -NoProfile -ExecutionPolicy Bypass -File "{script}" ' + f'-folderPath "{dest}"' + ) print("Windows signing complete") @@ -256,7 +310,9 @@ def package(dest: str, asset_name: str) -> None: arc = os.path.relpath(abs_path, dest) zf.write(abs_path, arc) digest = _sha256(zip_path) - with open(zip_path + ".sha256", "w") as f: + # newline="\n": keep an LF-only sidecar so `sha256sum -c` works for consumers + # (text mode would write CRLF on Windows and break filename matching). + with open(zip_path + ".sha256", "w", newline="\n") as f: f.write(f"{digest} {asset_name}.zip\n") print(f"sha256 {digest}") @@ -403,6 +459,49 @@ def _run_shell(cmd: str) -> int: return os.system(cmd) +def _sdk_bash(gitsdk: str, script: str) -> int: + """Run a script in the SDK's MSYS2/MINGW64 login shell, synchronously. + + git-bash.exe is a detached launcher (it returns before the command finishes + and sends output to a separate window), so it can't be driven from a build + script. bash.exe -lc runs in-process under MSYSTEM=MINGW64 and returns the + real exit code. MSYS paths (/usr/src/...) resolve under the SDK root. + + The MinGW toolchain needs a writable temp dir. The SDK's /etc/profile keeps + TMP/TEMP from the Windows environment but does not default them, so in a + headless/CI shell (where Windows provides none) they stay empty and the + tools fall back to C:\\Windows and fail. Export them *inside* the script + (i.e. after profile has run) so they point at a writable directory. Child + output is streamed through this process so it lands in our logs. + """ + import subprocess + bash = os.path.join(gitsdk, "usr", "bin", "bash.exe") + tmp = os.path.join(ROOT, "build-temp", "tmp") + os.makedirs(tmp, exist_ok=True) + tmp_msys = tmp.replace("\\", "/") + script = f'export TMP="{tmp_msys}" TEMP="{tmp_msys}" TMPDIR="{tmp_msys}"; {script}' + env = os.environ.copy() + env["MSYSTEM"] = "MINGW64" + env["CHERE_INVOKING"] = "1" + # A build shell is non-interactive; scripts like MinGit's release.sh drive + # the editor themselves (`-c core.editor=echo`) to read paths back. An + # inherited GIT_EDITOR/EDITOR/VISUAL (e.g. GIT_EDITOR=true in CI) overrides + # that and makes them return nothing, so drop them. + for _var in ("GIT_EDITOR", "EDITOR", "VISUAL"): + env.pop(_var, None) + kwargs = {} + if platform.system() == "Windows": + kwargs["creationflags"] = 0x08000000 # CREATE_NO_WINDOW + proc = subprocess.Popen( + [bash, "-lc", script], env=env, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, text=True, bufsize=1, **kwargs, + ) + for line in proc.stdout: + print(line, end="", flush=True) + proc.wait() + return proc.returncode + + # --------------------------------------------------------------------------- # # Entry # --------------------------------------------------------------------------- # @@ -421,6 +520,12 @@ def main() -> None: args = parser.parse_args() config = load_config() + # Windows signing config -> env (an explicit env var still wins). sign() + # reads these; sign_script defaults to ./sign-windows.ps1 when unset. + for opt, var in (("sign_script", "WINDOWS_SIGN_SCRIPT"), + ("sign_thumbprint", "WINDOWS_SIGN_THUMBPRINT")): + if config.has_option("windows", opt): + os.environ.setdefault(var, config["windows"][opt]) gitlfs = resolve_path(config, "gitlfs", "GIT_LFS_PATH", required=True) arch = args.arch or ("arm64" if platform.machine() in ("arm64", "aarch64") else "x86_64") asset = asset_name_for(arch) @@ -430,6 +535,7 @@ def main() -> None: gitsdk = resolve_path(config, "gitsdk", "GIT_SDK_PATH", required=True) build_git_windows(gitsdk, dest) build_gitlfs(gitlfs, dest, goos="windows", goarch="amd64") + bundle_less(gitsdk, dest) elif platform.system() == "Darwin": git_source = resolve_path(config, "gitsource", "GIT_SOURCE_PATH", required=True) build_git_macos(git_source, dest, arch) diff --git a/config.example.ini b/config.example.ini index 06f95fa..c76f44a 100644 --- a/config.example.ini +++ b/config.example.ini @@ -1,13 +1,15 @@ -# Copy to config.ini and adjust the paths for your build host. -# config.ini is gitignored — it holds host-specific absolute paths only. +# Copy to config.ini and adjust for your build host. +# config.ini is gitignored — it holds host-specific values only. # -# Every path may also be supplied via an environment variable (which takes +# Most values may also be supplied via environment variables (which take # precedence over this file), so CI can stay file-free: -# GIT_SDK_PATH, GIT_LFS_PATH, GIT_SOURCE_PATH +# GIT_SDK_PATH, GIT_LFS_PATH, GIT_SOURCE_PATH, +# WINDOWS_SIGN_SCRIPT, WINDOWS_SIGN_THUMBPRINT [gitsdk] -# Windows only. Path to the Git for Windows SDK (git-sdk-64), set up with the -# `anchorpoint` MinGit flavor under usr/src/build-extra/mingit/. See README. +# Windows only. Path to the Git for Windows SDK (git-sdk-64). build.py +# auto-initializes the usr/src/git and usr/src/build-extra sources on first +# build, so a freshly unpacked SDK works with no manual `sdk init`. path = C:\git-sdk-64 [gitlfs] @@ -19,3 +21,11 @@ path = third_party/git-lfs # macOS only. Path to a checked-out git/git source tree at the target tag # (e.g. `git clone https://github.com/git/git && git checkout v2.47.0`). path = /path/to/git/source + +[windows] +# Windows code signing (optional). Env vars WINDOWS_SIGN_SCRIPT / +# WINDOWS_SIGN_THUMBPRINT take precedence over these. +# sign_script - signing script (default: ./sign-windows.ps1 when unset) +# sign_thumbprint - code-signing cert in CurrentUser\My (e.g. SimplySign) +#sign_script = sign-windows.ps1 +#sign_thumbprint = F4D5B4774A99A3424B96BB7A5E3631526CA0644A diff --git a/release.ps1 b/release.ps1 new file mode 100644 index 0000000..5131cb1 --- /dev/null +++ b/release.ps1 @@ -0,0 +1,37 @@ +#Requires -Version 5 +<# +.SYNOPSIS + One-command Windows release for git-dist. +.DESCRIPTION + Verifies the SimplySign code-signing cert is mounted (SimplySign Desktop + logged in) BEFORE the long build, then runs build.py. Any arguments are + passed through to build.py; with no arguments it defaults to + "--package --publish" (build -> bundle less -> sign -> package -> release). + The SimplySign check is skipped when --nosign is passed. +.EXAMPLE + .\release.ps1 # build, sign, and publish the GitHub release +.EXAMPLE + .\release.ps1 --package --nosign # local unsigned build, no publish +#> +[CmdletBinding()] +param([Parameter(ValueFromRemainingArguments = $true)] $BuildArgs) +$ErrorActionPreference = "Stop" +$root = Split-Path -Parent $MyInvocation.MyCommand.Path + +if (-not $BuildArgs -or $BuildArgs.Count -eq 0) { $BuildArgs = @("--package", "--publish") } + +if ($BuildArgs -notcontains "--nosign") { + $thumb = if ($env:WINDOWS_SIGN_THUMBPRINT) { $env:WINDOWS_SIGN_THUMBPRINT } ` + else { "F4D5B4774A99A3424B96BB7A5E3631526CA0644A" } + $cert = Get-ChildItem Cert:\CurrentUser\My | + Where-Object { $_.Thumbprint -eq $thumb -and $_.HasPrivateKey } + if (-not $cert) { + throw ("Code-signing cert $thumb is not available. Log into SimplySign " + + "Desktop so the virtual card is mounted, then re-run (or pass --nosign).") + } + Write-Host "SimplySign ready: $($cert.Subject) (expires $($cert.NotAfter))" +} + +Write-Host "Running: python build.py $($BuildArgs -join ' ')" +& python (Join-Path $root "build.py") @BuildArgs +exit $LASTEXITCODE diff --git a/sign-windows.ps1 b/sign-windows.ps1 new file mode 100644 index 0000000..a40d703 --- /dev/null +++ b/sign-windows.ps1 @@ -0,0 +1,57 @@ +# Sign every .exe/.dll in a folder with the Anchorpoint code-signing cert via +# signtool, using the SimplySign cloud key (Certum). Invoked by build.py's +# sign() as: powershell -File sign-windows.ps1 -folderPath +# +# Prereqs: SimplySign Desktop logged in (virtual card mounted), Windows SDK +# (signtool) installed. The cert is selected by thumbprint; override with +# $env:WINDOWS_SIGN_THUMBPRINT if the cert ever changes. +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] [string] $folderPath, + [string] $Thumbprint = $(if ($env:WINDOWS_SIGN_THUMBPRINT) { $env:WINDOWS_SIGN_THUMBPRINT } else { "F4D5B4774A99A3424B96BB7A5E3631526CA0644A" }), + [string] $TimestampUrl = "http://time.certum.pl" +) +$ErrorActionPreference = "Stop" + +# Locate the newest signtool (x64) from the Windows SDK. +$signtool = Get-ChildItem "C:\Program Files (x86)\Windows Kits\10\bin\*\x64\signtool.exe" -ErrorAction SilentlyContinue | + Sort-Object FullName -Descending | Select-Object -First 1 -ExpandProperty FullName +if (-not $signtool) { throw "signtool.exe not found - install the Windows 10/11 SDK." } + +# Confirm the signing cert is present and usable (SimplySign logged in). +$cert = Get-ChildItem Cert:\CurrentUser\My | Where-Object { $_.Thumbprint -eq $Thumbprint } +if (-not $cert) { throw "Cert $Thumbprint not in CurrentUser\My - is SimplySign Desktop logged in?" } +if (-not $cert.HasPrivateKey) { throw "Cert $Thumbprint has no usable private key - SimplySign card not mounted?" } +Write-Host "signtool : $signtool" +Write-Host "signing : $($cert.Subject)" +Write-Host "expires : $($cert.NotAfter)" + +# Collect signable binaries. +$files = @(Get-ChildItem -Path $folderPath -Recurse -Include *.exe, *.dll -File -ErrorAction SilentlyContinue | + Select-Object -ExpandProperty FullName) +if ($files.Count -eq 0) { throw "No .exe/.dll found under $folderPath" } +Write-Host "files : $($files.Count) to sign + timestamp" + +# Sign in batches (one signtool call per batch keeps cloud round-trips down); +# retry a batch a few times since the RFC3161 timestamp server can be flaky. +$batchSize = 40 +for ($i = 0; $i -lt $files.Count; $i += $batchSize) { + $batch = $files[$i .. ([Math]::Min($i + $batchSize - 1, $files.Count - 1))] + $attempt = 0 + do { + $attempt++ + & $signtool sign /sha1 $Thumbprint /fd sha256 /tr $TimestampUrl /td sha256 $batch + $ok = ($LASTEXITCODE -eq 0) + if (-not $ok) { + if ($attempt -ge 3) { throw "signtool failed (exit $LASTEXITCODE) after $attempt attempts at index $i" } + Write-Warning "signtool exit $LASTEXITCODE - retry $attempt/3 in 5s (timestamp server busy?)" + Start-Sleep -Seconds 5 + } + } while (-not $ok) + Write-Host (" signed {0}/{1}" -f ([Math]::Min($i + $batchSize, $files.Count)), $files.Count) +} + +# Sanity-check one signature. +& $signtool verify /pa /all $files[0] | Out-Null +if ($LASTEXITCODE -ne 0) { throw "verification failed on $($files[0])" } +Write-Host "Done: $($files.Count) files signed + timestamped, verified OK."