fix(vision mixer): keep per-pixel alpha on keyed pads with an alpha-less output_format

[wagenet]

The bug

The vision mixer's output_format property offers alpha-less formats (I420, NV12, RGB, ...). Setting one on the CPU path destroyed the per-pixel alpha of every keyed pad:

• a DSK graphic's transparent areas composited as the graphic's own black instead of the program picture — and DSK is a downstream keyer;
• the multiview overlay covers the whole canvas, so the multiview went entirely black except for the overlay's own graphics.

There is a standing incentive to set the property: unpinned RGBA into vtenc_h264_hw negotiates AYUV64 rather than NV12.

Cause

Two places, one mechanism.

The builder inserted output_format capsfilters on the DSK chains and on the overlay appsrc chain, flattening alpha before the compositor saw it. Removing those alone does not fix it — measured, not assumed. compositor blends in its own output format and converts every sink pad to that format first, so pinning an alpha-less format on the compositor's src caps makes its sink pads alpha-less too. Directly:

compositor ! video/x-raw,format=NV12   # RGBA input: not-negotiated
compositor ! videoconvert ! video/x-raw,format=NV12   # RGBA input: key survives

The hazard was already known in this file — the border underlays pin RGBA with a comment saying an alpha-less output_format "would silently drop the alpha of #RRGGBBAA border colors" — but the conclusion was never applied to the compositor's own output.

Fix

When a compositor has keyed pads and output_format cannot carry alpha, blend in the alpha-carrying counterpart of the requested format (A420, A422, A422_10LE, or RGBA — matching the requested chroma subsampling) and convert to output_format after the mixer. output_format still pins what leaves the block; it no longer dictates the blend space.

The dist compositor gets this when DSK inputs or PiP border underlays exist; the multiview always does, since its overlay is a full-canvas RGBA pad.

The DSK and overlay capsfilters are dropped. The video-input capsfilters stay — their tee feeds two independently negotiating compositors, which is what they are for.

builder/pipeline_gpu.rs never referenced output_format and is unchanged.

Tests

New backend/tests/vision_mixer_keyed_alpha_test.rs builds a real flow through PipelineManager (following vision_mixer_fx_test.rs): a white program input, a DSK graphic that is opaque red on the left half and fully transparent on the right, output_format=NV12, CPU backend. It asserts the transparent half still reads the white background, that the multiview shows picture once the overlay renderer has actually pushed frames, and that PGM still negotiates NV12. A second test runs the same flow with output_format=Auto as the never-broken baseline.

Two things the measurement has to get right, both of which produced a green test that guarded nothing until fixed. The multiview reading is gated on a buffer probe counting overlay appsrc pushes, so it cannot pass by measuring a multiview the overlay has not reached yet. And readiness is detected by colour, not brightness: an all-black startup frame is dark in exactly the way an opaque-keyed failure is dark, so the run waits for the graphic's opaque half to actually be red before measuring.

Verified by reverting the fix and re-running — the guard goes red on both halves:

	fix reverted	with fix
DSK transparent half, fraction white	0.000	1.000
multiview picture, fraction bright	0.000	0.144
PGM negotiated format	NV12	NV12

Run locally on macOS (Apple silicon, GStreamer 1.28.6): the full cargo test suite, all green, including the new test and vision_mixer_fx_test. Nothing was skipped in the new test — it needs compositor (gst-plugins-base) and videobox (gst-plugins-good), both already in the CI package list on every platform, and it has no skip path. Not run: anything requiring hardware absent from this machine (NVIDIA/CUDA paths).

🤖 Generated with Claude Code