Skip to content

perf(video): convert and scale in one pass in the Video Format block - #731

Draft
wagenet wants to merge 2 commits into
Eyevinn:mainfrom
wagenet:wagenet/videoconvertscale
Draft

perf(video): convert and scale in one pass in the Video Format block#731
wagenet wants to merge 2 commits into
Eyevinn:mainfrom
wagenet:wagenet/videoconvertscale

Conversation

@wagenet

@wagenet wagenet commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Stacked on #726 — this branch is cut from wagenet/macos-convert-mode, so the diff includes that commit. Only the second commit belongs to this PR.

The Video Format block built videoscale -> videoconvert -> capsfilter, so a flow changing both resolution and pixel format walked every frame twice with an intermediate buffer between. videoconvertscale does both in one walk.

autovideoconvert already covers both passes — it is Bin/Colorspace/Scale/Video/Converter and autoplugs a scaler when the caps ask for one (verified with gst-launch-1.0: 1080p RGBA in, 720p I420 negotiated out). So the collapse applies in both convert modes, not just software. The new VideoConvertMode::convert_scale_element_name() picks the element per mode; element_name() is untouched, so the convert-only sites (NDI, videoenc, devicesrc, CPU vision mixer) don't gain a scaler.

Measurements

M2, 4 performance cores, 1080p RGBA, 300 frames, within-pair order alternated. The noise floor here is ±11% on paired runs, and a fixed order has produced sign-flipping "wins" before.

Isolated stage, gst-launch-1.0, 20 pairs: 1.558 s → 0.589 s, +62%, 20/20 pairs, order-symmetric (+62.2% A-first, +62.0% B-first).

Full Strom flow ending in x264enc, fresh backend per measurement, 12 pairs: 2.104 s → 1.292 s, +38% (range 34–41%), 12/12 pairs, order-symmetric (+38.4%, +38.7%).

Format only, scaling passthrough (1080p RGBA → 1080p I420), 20 pairs: median −0.9%, 10/20 pairs. No measurable difference — a passthrough videoscale is free.

Most of that is not pass-merging. A third arm, videoscale n-threads=4 ! videoconvert n-threads=4, splits it: merging the passes is worth ~10% (median +10.5%, 17/20 pairs). The rest is that the scaling half is now threaded — videoscale has the same n-threads property and the same default of 1, but nothing ever set it, and the merged element goes through configure_video_convert().

Scaling method

Both elements share GstVideoScaleMethod and default to 1, "bilinear" (GStreamer 1.28.6), so scaled output is unchanged and no method pin is needed. scaling_method_default_matches_videoscale fails if upstream diverges.

External pads

Saved flows store links as block_id:external_pad_name and resolve them through the block definition at build time. VideoFormat doesn't override get_external_pads(), so it never persists computed_external_pads and nothing stale is baked into saved JSON. The risk is a definition naming an element the builder no longer creates, so the surviving element keeps the ID videoscale and external_pads is unchanged. That matches the convention already in the file: the ID names the role, not the factory, which is why the convert element was called videoconvert while being autovideoconvert.

Verified end to end — a flow linking vfmt:video_in and vfmt:video_out was created and started against the real backend, built videoconvertscale -> capsfilter, played, and reached EOS.

Other videoscale sites

None. videoformat.rs was the only place building one, and thumbnail_tap.rs already uses videoconvertscale. A stale doc comment in thumbnail.rs naming elements that module no longer references is corrected here.

Tests

  • builds_one_conversion_element_not_two — two elements, one internal link, and the conversion element both converts and scales.
  • one_element_both_scales_and_converts — 1080p RGBA into the block, 1280x720 I420 out.
  • conversion_element_is_threaded (macOS) — n-threads reaches the merged element and is above the default of 1.
  • external_pads_resolve_to_built_elements — every external pad resolves to an element the builder creates.
  • scaling_method_default_matches_videoscale
  • gpu::convert_scale_element_name_scales — the mode mapping, plus a guard that autovideoconvert still advertises Scale.

The first three were verified to fail against a temporary revert of build() to the old chain. The last three pass under that revert; they guard a future rename, future upstream drift, and a future mode-mapping mistake rather than this change.

All six use core and gst-plugins-base elements plus autovideoconvert from gst-plugins-bad, all installed in CI, so none skip.

Ran locally: full cargo test (552 lib tests plus all integration tests, including pipeline_lifecycle_test) green, with one pre-existing unrelated ignore (test_jitterbuffer_stalls_without_drop_on_latency); cargo build and cargo clippy --all-targets -- -D warnings clean.

Not run: Linux or Windows. macOS always resolves to Software, so the autovideoconvert branch of convert_scale_element_name() is never exercised here — it rests on the gst-launch check and the klass assertion, not a real GPU flow.

🤖 Generated with Claude Code

@wagenet
wagenet force-pushed the wagenet/videoconvertscale branch 2 times, most recently from b656eb2 to d856a8a Compare August 31, 2026 19:32

@wagenet wagenet left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks reasonable.

wagenet and others added 2 commits August 31, 2026 21:46
…ersion

`detect_gpu_capabilities()` set the process-wide `VideoConvertMode` by asking
whether `nvh264enc` exists. On a Mac it never does, so every Mac fell through to
`Software`: a right answer reached by asking about hardware the machine does not
have, and one that hid the question that does matter.

Split the decision per platform. Linux and Windows keep the existing WSL /
NVENC / CUDA-GL interop path untouched. macOS answers for itself and still picks
`Software`, for two reasons that apply to the hardware:

- `autovideoconvert` has no GPU path to offer. For the system-memory frames our
  blocks feed it, it selects `videoconvertscale` and never creates a GL context
  (checked with `GST_DEBUG=gl*:5`, zero GL lines). It measured within 1% of
  plain single-threaded `videoconvert`, the same CPU work wrapped in extra bin
  and negotiation machinery.
- It is a bin, so it exposes no `n-threads` and cannot forward one to the
  converter it picks.

The second point is what the NVENC probe was hiding. `videoconvert` ships with
`n-threads=1` and the string appeared nowhere in the tree, so every colour
conversion on an 8-core M2 ran on one core. `configure_video_convert()` now sets
the pool at all five call sites (ndi, devicesrc, videoenc, videoformat and the
vision mixer's CPU pipeline).

1080p RGBA->I420 on a 4+4 M2, paired runs alternating the within-pair order:

    concurrent converts   n-threads 1 -> 4   pairs won
    1                     -23.8%             20/20
    2                     -18.4%             16/16
    4                     -8.3%              14/16

Isolating the convert stage against a source-only baseline, its cost drops from
1.05 s to 0.24 s over 300 frames. The win narrows as the cores saturate and
stays positive.

The pool counts every performance tier macOS reports except the efficiency one.
Reading tier 0 alone held only while each part had a single fast tier, which the
M5 Ultra ends: its 36-core CPU is 12 super cores plus 24 performance cores with
no efficiency tier, so tier 0 there is 12 and would discard 24 fast cores.
Naming the tier to exclude is the durable form, since "Efficiency" has stayed
stable while the fast tiers gain names, and an unrecognised name fails open at a
cost of ~2%.

Excluding the efficiency tier is a mechanism call, because measurement could not
settle it: 4 threads against 8 landed within 2% and disagreed about the sign
across the three contention levels, well inside the machine's +/-11% noise.
Equal-stripe splitting makes an efficiency core the straggler that the whole
frame waits on.

A bound of 32 guards against a bogus syscall result and logs when it binds. It
stays loose because the errors are asymmetric: 16, 32 and 64 threads cost 1.2%,
1.7% and 1.4% against the 4-thread baseline, with no cliff even at 17-row
stripes, while 1 thread instead of 4 costs 24%. `STROM_VIDEOCONVERT_THREADS`
(1-64) overrides it, and the resolution logic is split out from the environment
so the wide-machine paths are covered by tests.

`n-threads` is scoped to macOS. No Linux or Windows machine was available to
measure, and a Linux container's visible CPU count routinely overstates its
cgroup quota, so sizing a pool there unmeasured risks oversubscribing shared
hosts. Linux and Windows behaviour is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The block built `videoscale -> videoconvert -> capsfilter`, so a flow that
changed both resolution and pixel format walked every frame twice with an
intermediate buffer in between. `videoconvertscale` does both in one walk.

`autovideoconvert` already covers both passes on the GPU path — it is
`Bin/Colorspace/Scale/Video/Converter` and autoplugs a scaler when the caps
ask for one — so the collapse applies in both convert modes. The new
`VideoConvertMode::convert_scale_element_name()` names the right element for
each, leaving `element_name()` alone for the convert-only call sites.

Two effects, measured separately on an M2 (4 performance cores), 1080p RGBA
source, 300 frames, 20 paired runs with the within-pair order alternated:

- Merging the two passes is worth ~10% on its own.
- The larger part is that the scale half is now threaded. `videoscale` has
  the same `n-threads` property and the same default of 1, but nothing ever
  set it; the merged element goes through `configure_video_convert()`.

Together: the isolated conversion stage runs ~62% faster (20/20 pairs), and
a full Strom flow ending in x264enc runs 34-41% faster, median 38%, 12/12
pairs. Both order-symmetric. Where only the format changes and scaling is
passthrough there is no measurable difference (median -0.9%, 10/20 pairs),
well inside this machine's noise floor.

The element ID stays "videoscale" so the block's external input pad still
resolves and saved flows that link into this block keep working. Both
elements default to bilinear scaling, so the output is unchanged; a test
pins that and fails if upstream ever diverges.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant