perf(macos): convert colour through Accelerate/vImage

[wagenet]

Stacked on #731 (which is itself stacked on #726). Review the last commit only; the first two belong to those PRs. Merge them first.

What this adds

stromvimageconvert — a GStreamer element that converts colour through Accelerate/vImage on macOS. It lives in backend/src/gst/vimage/, registers itself statically the first time detect_gpu_capabilities() runs, and is selected by a new macOS-only VideoConvertMode::VImage.

No call site changed. The element carries the same n-threads property as videoconvert, so configure_video_convert() from #726 reaches it unchanged and all five sites pick it up through element_name() / convert_scale_element_name().

Four modules, one job each: accelerate.rs (FFI), plan.rs (decide, once per negotiation, whether vImage can do the pair), convert.rs (execute, per frame), imp.rs (the element and its caps negotiation).

Fallback

The element never fails a conversion videoconvert would have managed. Anything vImage has no path for — an unlisted format pair, a resize, interlaced content, a colour matrix outside BT.601/BT.709 — is built as a GstVideoConverter, which is the code videoconvert and videoconvertscale are themselves built on, configured with the same thread count. A new read-only conversion-path property reports vimage / fallback / unnegotiated.

vImage paths: packed RGB32 (RGBA/BGRA/ARGB/ABGR and their x variants) to and from NV12, I420, YV12, UYVY, YUY2; RGB32 to RGB32; and NV12 to and from I420/YV12.

Which pairs Strom actually negotiates

Checked against a running backend with GST_DEBUG=stromvimageconvert:5, driving videotestsrc → Video Format → Video Encoder → fakesink:

Configuration	Negotiated	Path
Video Format with an output format set	RGBA→NV12, RGBA→I420, RGBA→UYVY, BGRA→NV12	vImage
Video Format left unset, vtenc_h264_hw downstream	RGBA→UYVY (was RGBA→ARGB64_BE)	vImage
Second converter, format already pinned	NV12→NV12 etc.	passthrough

So the headline RGBA→NV12 is real, but only when an output format is pinned. Unpinned, the encoder's own preference decides, and that turned out to be worth reporting on its own: vtenc_h264_hw offers { AYUV64, UYVY, NV12, I420, P010_10LE, ARGB64_BE, RGBA64_LE }, and stock videoconvert picks AYUV64 — sixteen bits per component for an eight-bit source, which the encoder then converts again. That is pre-existing behaviour, not something this PR introduced.

The one deliberate divergence from videoconvert

My first fixation scored the same menu and picked ARGB64_BE, also 16-bit. Measured end to end that cost 15.7% against picking an 8-bit target (t = −163, n = 40). So a bit-depth increase now outweighs every other penalty combined: a same-or-shallower format always wins when one exists, and a deeper one is still chosen when it is the only option. Widening cannot add information and it doubles the bytes touched per frame. The element picks UYVY there, the regression goes to zero, and that path now reaches vImage too.

fixate_caps is otherwise a smaller model than gst_video_convert's score_value — colour space, alpha, depth, chroma subsampling, ties to the peer's order. The case that matters most, "the input format is on the menu, take it", is exact.

Numbers

M4 Max (12P+4E, 64 GB), 1080p RGBA→NV12, 300 frames, 40 counterbalanced pairs per experiment (half each order, shuffled — not alternating), run in chunks of fresh processes so macOS App Nap cannot reach them. Host load average 2.5–9 across the run, recorded per chunk in the CSV. Debug build, which if anything favours the C-heavy videoconvert arm.

Experiment	Arm A	Arm B	Net of source	t
baseline (identity vs identity)	237.9 ms	238.0 ms	—	−0.20
A/A control (videoconvert vs itself)	472.0 ms	444.7 ms	1.01×	0.60 †
A/B (videoconvert n-threads=12 vs vImage)	186.2 ms	54.3 ms	3.43×	33.6
vs stock (videoconvert n-threads=1 vs vImage)	738.2 ms	53.1 ms	13.91×	791.8
unpinned (vtenc_h264_hw downstream)	444.5 ms	444.2 ms	1.00×	5.96

† The A/A figure shown is from the clean run (+0.4%). The repeat run caught a load spike and returned +5.8%, t = 1.26 — also not significant, which is the point. Both A/B runs agreed at 3.43× and 3.44×.

Against the figures this work was scoped from: 3.4× vs a predicted 3.6× over the threaded path, and 13.9× vs a predicted ~14× over stock. Different machine (M4 Max, not M2), same conclusion.

The harness is committed as #[ignore]d tests in bench.rs with the method documented, so the numbers are reproducible. It asserts nothing about timing.

Tests

cargo test — 566 lib tests + all integration binaries pass, run in full. cargo clippy --all-targets -- -D warnings and cargo fmt --check clean.

The load-bearing test is vimage_output_matches_videoconvert: for all eleven claimed pairs it pushes one frame through this element and through stock videoconvert and compares pixel for pixel, tolerating ±2 for chroma rounding. It asserts via conversion-path that the vImage path was actually taken — without that the comparison would pass vacuously whenever negotiation quietly chose the fallback.

I verified it bites, rather than assuming it: swapping the red and blue entries of the RGBA permute map fails it by 149; treating limited-range Y'CbCr as full range fails it by 20. Both restored afterwards.

Also covered: unsupported pairs and resizes take the fallback and match videoconvert exactly; n-threads is exposed and configure_video_convert moves it; identical caps negotiate without converting; the depth-increase rule and its "only option" escape.

Where these tests run

Locally on macOS only. None of the 14 new tests execute in PR CI. The module is #[cfg(target_os = "macos")] throughout, and build-macos in ci.yml is gated on github.event_name == 'workflow_dispatch', so the Linux jobs compile the whole thing out. They will run on a manual dispatch with platforms: macos (or both), and they need no new package — everything used is in gst-plugins-base.

This is a pre-existing property of the repo's CI rather than something this PR introduces: the macOS-only tests #726 added to gpu.rs are in the same position. Worth deciding separately whether a macOS test job should run on PRs; I have not changed CI here.

What CI did verify on this PR: Linux x86_64 and ARM64 builds, Check (Linux) (clippy plus cargo test with the module cfg'd out), the WASM build, and the API contract check — all green. Windows is built by no job on a PR either, so it is unverified by anyone; the non-macOS code paths it shares with Linux are identical.

gstreamer-base is a macOS-only dependency and already transitive via gstreamer-video, so nothing new compiles anywhere.

Not done

Scaling has no vImage path — vImageScale plus a convert would be two passes, so a resize stays on GstVideoConverter. NV21, 10-bit and v210 are not covered. Worth revisiting if a flow turns out to need them.

🤖 Generated with Claude Code