Migrate to full-feature FFmpeg build (forge) + ffmpeg-bins2 + manifest.json catalog, preserving legacy installs

[zackees]

Summary

Move static-ffmpeg onto a freshly built, full-feature FFmpeg (all features enabled, including GPL/nonfree/licensed codecs), built reproducibly via zackees/forge, published to a new zackees/ffmpeg-bins2 repo (content stored via Git LFS), and cataloged with a zackees/manifest.json-format manifest so the Python client resolves the correct artifact per platform.

Hard constraint: existing static-ffmpeg installs that download from the legacy zackees/ffmpeg_bins repo must keep working forever, and we must be able to ship future FFmpeg upgrades without breaking already-deployed installs.

---

Goals

1. Build FFmpeg + FFprobe with all features enabled, including options that require GPL and --enable-nonfree (see Licensing risk below).
2. Build the full target matrix (see below), including Linux glibc 2.17 and musl variants.
3. Create zackees/ffmpeg-bins2, store binaries via Git LFS.
4. Publish a manifest.json catalog (per the zackees/manifest.json schema) describing every artifact with sha256, size_bytes, and download urls[].
5. Update the static-ffmpeg client to resolve + verify artifacts through the manifest.
6. Backward compatibility: never mutate ffmpeg_bins or its raw URLs; old installs keep resolving their pinned binaries.
7. Forward compatibility: future FFmpeg upgrades are a manifest/channel-pointer change, not a client release, and don't disturb existing installs.

---

Target build matrix

Key (platform tuple)	os	arch	libc	Notes
win_x64	windows	x86_64	–
win_arm64	windows	arm64	–	forge windows_arm64
darwin_x64	macos	x86_64	–
darwin_arm64	macos	arm64	–
linux_x64_glibc217	linux	x86_64	glibc 2.17	manylinux2014 baseline
linux_arm64_glibc217	linux	arm64	glibc 2.17	manylinux2014 baseline
linux_x64_musl	linux	x86_64	musl	forge linux_x64_musl
linux_arm64_musl	linux	arm64	musl	forge linux_arm64_musl

8 targets. Each artifact packages ffmpeg + ffprobe (+ LICENSE/build-info).

---

Build: zackees/forge

• Author/point to a Conan recipe for FFmpeg configured with the full option set (GPL + nonfree + all external codec libs the recipe supports).
• Drive builds via the reusable workflow zackees/forge/.github/workflows/forge-conan.yml@main, toggling the platform inputs (windows_arm64, linux_x64_musl, linux_arm64_musl, etc.).
• forge uploads a max-compression archive per platform → collect these and republish into ffmpeg-bins2.

⚠️ glibc 2.17 risk: forge builds on native GitHub runners (Ubuntu with glibc 2.35+). A native build will not run on glibc 2.17 systems. The glibc-2.17 targets need a manylinux2014 / CentOS 7 container (or an equivalent sysroot toolchain) so the resulting binaries carry a 2.17 floor. Decide whether this is a forge job variant or a separate container build feeding the same manifest.

---

New repo: zackees/ffmpeg-bins2

• Create the repo (owner note: the goal text says "fmpeg-bins2" — assuming ffmpeg-bins2).
• .gitattributes: * binary binaries tracked by LFS (*.zip/*.tar.zst filter=lfs diff=lfs merge=lfs -text).
• Layout: versioned dirs, e.g. ffmpeg/<version>/<platform-key>.<ext> plus a top-level manifest.json.

⚠️ LFS + download-URL pitfall (learned the hard way in this repo): do not assume github.com/.../raw/... will serve LFS content the way it serves plain blobs. The manifest must carry the canonical download URLs plus sha256, and the client must verify the hash — so integrity never depends on raw-URL smudging behavior.

Transport (decided): serve the LFS files behind a www site fronted by a CDN. The binaries are stored in-repo via LFS, but the manifest's Asset.urls[] point at CDN-fronted https://<www-site>/... URLs (not GitHub raw/LFS-media or Release-asset endpoints). This puts delivery on a CDN with no bandwidth/quota limits (avoiding both GitHub LFS bandwidth caps and Release-asset rate limits), gives stable cacheable URLs, and keeps integrity guaranteed by the manifest sha256. LFS remains the in-repo source of truth; the www/CDN layer is the distribution front-end.

---

Catalog: manifest.json (per zackees/manifest.json)

Produce a Catalog document for tool: "ffmpeg":

• kind: "Catalog", schema_version: 1, tool: "ffmpeg"
• channels: { "latest-stable": "<version>" }
• releases[]: newest-first, each with version, published_at, and platforms[] mapping platform tuples → Asset { filename, sha256, size_bytes, media_type, urls[] }.
• Platform tuple fields used: {os, arch, libc} (libc = glibc/musl; optionally os_version for the 2.17 floor).

Client resolution contract: resolve("ffmpeg", <detected-platform>, "latest-stable") → Asset.

---

Client changes (static_ffmpeg/run.py)

1. Platform detection must gain libc awareness on Linux (distinguish glibc vs musl — e.g. platform.libc_ver() / probe for ld-musl-* / ctypes), producing the platform tuple {os, arch, libc}.
2. Manifest resolver: fetch ffmpeg-bins2/manifest.json, resolve the tuple + latest-stable channel to an Asset, download from Asset.urls, and verify sha256 before extracting.
3. Version pinning: record the resolved version + sha256 in installed.crumb. Existing installs stay on their pinned artifact and don't re-download; only fresh installs (or an explicit upgrade) pull the newest channel version.
4. Fallback: if the manifest fetch/resolve fails, fall back to the current hardcoded legacy behavior so we never regress availability.

---

Backward compatibility & upgrade strategy (must-have)

• Freeze zackees/ffmpeg_bins. Never migrate it to LFS, never move/rename v8.0/*.zip. Old clients hardcode ffmpeg_bins/raw/main/v8.0/<platform>.zip; those exact URLs must keep serving the exact bytes forever. (This is why the recent local LFS migration of ffmpeg_bins was reverted rather than pushed.)
• New installs use ffmpeg-bins2 + manifest.
• Upgrading FFmpeg later = publish a new release into ffmpeg-bins2 and repoint the latest-stable channel in manifest.json. No client release required. Already-installed users keep their pinned version (crumb) and are unaffected; new installs get the new version. This decouples FFmpeg version cadence from static-ffmpeg releases.

---

⚠️ Licensing risk (decision needed, not a blocker)

Enabling --enable-gpl --enable-nonfree produces binaries whose redistribution is legally restricted — nonfree output is, per FFmpeg's own docs, not redistributable. Publishing such binaries in a public repo/release may violate those terms. Options to decide:

• Keep nonfree artifacts in a private distribution channel, or
• Ship a public GPL (no-nonfree) default plus an opt-in nonfree channel, or
• Accept the risk explicitly (owner's call).

Flagging per repo policy; proceeding as the owner directs.

---

Acceptance criteria / task checklist

• FFmpeg Conan recipe with full feature set (GPL + nonfree) builds green in forge for all 8 targets.
• Linux glibc-2.17 binaries verified to run on a glibc 2.17 host; musl binaries verified on Alpine.
• zackees/ffmpeg-bins2 created; binaries stored (LFS in-repo and/or Release assets).
• manifest.json (Catalog) validates against the zackees/manifest.json schema and resolves all 8 tuples.
• static-ffmpeg client resolves via manifest, verifies sha256, and falls back to legacy on failure.
• Client distinguishes glibc vs musl on Linux.
• Regression test: an "old" client config still downloads from ffmpeg_bins/raw/main/v8.0/*.zip successfully.
• Documented upgrade runbook: how to publish a new FFmpeg version by repointing the channel without breaking existing installs.
• Licensing decision recorded.

---

Open questions

1. glibc 2.17: forge container job vs. separate manylinux2014 build?
2. Distribution transport Decided: LFS files served over a www site fronted by a CDN (no bandwidth limits); manifest urls[] point at the CDN. Remaining sub-question: which host/CDN for the www site?
3. Nonfree licensing channel: public, private, or split?
4. FFmpeg version to target for the first ffmpeg-bins2 release?

Filed via Claude Code.

[zackees]

Progress: client foundation + cross-platform encode tests — PR #21

PR #21 lands the automatable, mergeable half of this issue while guaranteeing existing installs are never disturbed.

Done in #21

• Manifest-based resolution (static_ffmpeg/manifest.py): platform-tuple detection {os, arch, libc}, glibc vs musl distinction, zackees/manifest.json Catalog resolution, sha256-verified downloads.
• Backward compatibility: legacy ffmpeg_bins URLs remain the default and the guaranteed fallback; resolution is best-effort and disabled by default (STATIC_FFMPEG_MANIFEST_URL to opt in).
• Client distinguishes glibc vs musl on Linux.
• Real small-encode test across every existing platform workflow (lavfi → H.264/AAC mp4 → probe).
• manifest.example.json documenting the 8-target catalog contract + 8-target resolution test.
• README migration/backward-compat docs.

Remaining (operational / infra — cannot be done from a coding session)

• Build FFmpeg (full features incl. GPL + nonfree) via zackees/forge for all 8 targets, with glibc-2.17 baseline (manylinux2014 container) + musl.
• Create ffmpeg-bins2, store binaries in LFS.
• Serve LFS files behind the CDN-fronted www site (no bandwidth limits).
• Publish the real manifest.json (real versions/sizes/sha256/urls) and flip DEFAULT_MANIFEST_URL on.
• Add Windows-ARM64 and Linux-musl test workflows (need the new binaries first).
• New-artifact zips must extract into the same bin/<platform-key>/ folder layout the client expects.

[zackees]

Update: ffmpeg-bins2 repo created + PR #21 merged

Merged: #21 (client resolver + backward-compat fallback + cross-platform encode tests) — all 20 CI jobs green (5 platforms × 4 Python versions).

Created: https://github.com/zackees/ffmpeg-bins2 — scaffolded with:

• .gitattributes LFS config for binary archives
• manifest.json — placeholder Catalog for all 8 targets (real hashes/urls land with the first build)
• scripts/gen_manifest.py — generates a real Catalog (sha256 + size + CDN urls) from built artifacts (smoke-tested)
• .github/workflows/build-ffmpeg-forge.yml — draft 8-target forge matrix (dispatch-only)
• README documenting layout, targets, CDN distribution, and the backward-compat guarantee

What's left (needs external execution / a decision — cannot be done from a coding session):

1. Author/verify an ffmpeg Conan recipe with the full feature set (GPL + nonfree + external codec libs). The draft workflow currently points at conan-center-index recipes/ffmpeg/all as a starting point but the full-feature option set is unverified.
2. Solve glibc 2.17: forge's native Linux runners ship newer glibc; the 2.17 targets need a manylinux2014 container/sysroot.
3. Run forge to produce all 8 artifacts, drop them under <version>/ in ffmpeg-bins2 (LFS), and run gen_manifest.py.
4. Stand up the CDN-fronted www site and put its URLs in the manifest.
5. Flip static_ffmpeg/manifest.py::DEFAULT_MANIFEST_URL to the published catalog (a small follow-up PR) and add Windows-ARM64 + Linux-musl test workflows.

Old installs remain fully functional throughout — ffmpeg_bins is untouched.

[zackees]

Status: new full-feature FFmpeg live for macOS + glibc-Linux; Windows/musl blocked on forge

✅ Shipped & merged

• static_ffmpeg#21 — manifest-based client (glibc/musl-aware platform tuple, sha256-verified downloads) with guaranteed legacy fallback, plus a real cross-platform encode test.
• static_ffmpeg#22 — flips DEFAULT_MANIFEST_URL to the published catalog. CI (20/20 green) downloaded the new binaries via the manifest and passed the encode test on macOS + Linux — real end-to-end validation.
• zackees/ffmpeg-bins2 created: forge build workflow (recipe-full/ wrapper → GPL+nonfree full-codec ffmpeg), gen_manifest.py, assemble_release.py, docs/ffmpeg-options.md.
• Release v8.1.2 — 4 assets (CDN, HTTP 200), archived in LFS.
• manifest.json — real sha256/size/CDN urls.

Platform matrix (built via forge, ffmpeg 8.1.2, --enable-gpl --enable-nonfree)

target	status
macOS x64 / arm64	✅ shipped
glibc-Linux x64 / arm64	✅ shipped
Windows x64 / arm64	❌ blocked — forge MSVC pkg-config (zackees/forge#10)
musl x64 / arm64	❌ blocked — forge C-only musl toolchain (zackees/forge#10)

Windows/musl clients (and any manifest failure) fall back to the legacy ffmpeg_bins URLs — no regression. ffmpeg_bins remains frozen; old installs keep resolving their pinned binaries.

Feature-set notes

conan-center ffmpeg defaults already yield --enable-gpl --enable-nonfree with x264, x265, fdk-aac, vpx, aom, dav1d, svt-av1, mp3lame, opus, vorbis, openh264, openjpeg, webp. The recipe has no nvenc/qsv/amf/libass (ceiling documented in docs/ffmpeg-options.md). Nonfree ⇒ not for redistribution.

Remaining

• Unblock Windows + musl once ffmpeg build fails on Windows/MSVC (pkg-config .pc not found) and musl (C++ builds against glibc) forge#10 is fixed, then re-run the build, add those 4 assets to the release, regenerate manifest.json, and add Windows-ARM64 + musl encode-test workflows.
• Future ffmpeg upgrades: re-run the forge build → new release → repoint latest-stable in manifest.json (no client release; existing installs keep their pinned version).

[zackees]

Update: Windows deep-dive — 3 forge fixes merged; blocked on a flaky pkg-config env

Pursued the Windows gap hard. Landed three real forge fixes (all merged) that took the MinGW build from instant failure to compiling the entire dependency tree:

• fix(mingw): forward-slash toolchain paths so msys2 bash builds work forge#11 — forward-slash MinGW toolchain paths (fixed ar: command not found)
• fix(mingw): relax GCC 14/15 legacy-C errors for autotools deps forge#12 — -Wno-error for GCC-15 legacy-C errors (fixed libx264)
• fix(mingw): static-link runtime for self-contained executables forge#13 — -static -static-libgcc -static-libstdc++ (self-contained exe, per the no-DLL-deps requirement)

Remaining Windows blocker (both MSVC and MinGW): ffmpeg's configure pkg-config resolution is non-deterministic on Windows — different deps report <lib> not found using pkg-config across runs (aom → dav1d → fdk-aac → freetype2, which passed in the prior run). Because it intermittently hits essential libs (freetype), disabling libraries can't converge. Full analysis + suggested fixes in zackees/forge#10. This needs a Windows+conan local repro to fix, so it's out of reach for more CI iteration here. (The static-linking requirement is satisfied by #13 for whenever the build completes.)

musl still needs a musl C++ toolchain in forge (forge#10).

Net state

target	8.1.2
macOS x64 / arm64	✅ shipped (full GPL+nonfree)
glibc-Linux x64 / arm64	✅ shipped (full GPL+nonfree)
Windows x64 / arm64	⏳ forge#10 (flaky configure pkg-config) — falls back to legacy
musl x64 / arm64	⏳ forge#10 (musl C++ toolchain) — falls back to legacy

Old installs unaffected throughout; ffmpeg_bins frozen.