perf(video): convert and scale in one pass in the Video Format block - #731
Draft
wagenet wants to merge 2 commits into
Draft
perf(video): convert and scale in one pass in the Video Format block#731wagenet wants to merge 2 commits into
wagenet wants to merge 2 commits into
Conversation
wagenet
force-pushed
the
wagenet/videoconvertscale
branch
2 times, most recently
from
August 31, 2026 19:32
b656eb2 to
d856a8a
Compare
wagenet
commented
Aug 31, 2026
wagenet
left a comment
Contributor
Author
There was a problem hiding this comment.
Code looks reasonable.
…ersion
`detect_gpu_capabilities()` set the process-wide `VideoConvertMode` by asking
whether `nvh264enc` exists. On a Mac it never does, so every Mac fell through to
`Software`: a right answer reached by asking about hardware the machine does not
have, and one that hid the question that does matter.
Split the decision per platform. Linux and Windows keep the existing WSL /
NVENC / CUDA-GL interop path untouched. macOS answers for itself and still picks
`Software`, for two reasons that apply to the hardware:
- `autovideoconvert` has no GPU path to offer. For the system-memory frames our
blocks feed it, it selects `videoconvertscale` and never creates a GL context
(checked with `GST_DEBUG=gl*:5`, zero GL lines). It measured within 1% of
plain single-threaded `videoconvert`, the same CPU work wrapped in extra bin
and negotiation machinery.
- It is a bin, so it exposes no `n-threads` and cannot forward one to the
converter it picks.
The second point is what the NVENC probe was hiding. `videoconvert` ships with
`n-threads=1` and the string appeared nowhere in the tree, so every colour
conversion on an 8-core M2 ran on one core. `configure_video_convert()` now sets
the pool at all five call sites (ndi, devicesrc, videoenc, videoformat and the
vision mixer's CPU pipeline).
1080p RGBA->I420 on a 4+4 M2, paired runs alternating the within-pair order:
concurrent converts n-threads 1 -> 4 pairs won
1 -23.8% 20/20
2 -18.4% 16/16
4 -8.3% 14/16
Isolating the convert stage against a source-only baseline, its cost drops from
1.05 s to 0.24 s over 300 frames. The win narrows as the cores saturate and
stays positive.
The pool counts every performance tier macOS reports except the efficiency one.
Reading tier 0 alone held only while each part had a single fast tier, which the
M5 Ultra ends: its 36-core CPU is 12 super cores plus 24 performance cores with
no efficiency tier, so tier 0 there is 12 and would discard 24 fast cores.
Naming the tier to exclude is the durable form, since "Efficiency" has stayed
stable while the fast tiers gain names, and an unrecognised name fails open at a
cost of ~2%.
Excluding the efficiency tier is a mechanism call, because measurement could not
settle it: 4 threads against 8 landed within 2% and disagreed about the sign
across the three contention levels, well inside the machine's +/-11% noise.
Equal-stripe splitting makes an efficiency core the straggler that the whole
frame waits on.
A bound of 32 guards against a bogus syscall result and logs when it binds. It
stays loose because the errors are asymmetric: 16, 32 and 64 threads cost 1.2%,
1.7% and 1.4% against the 4-thread baseline, with no cliff even at 17-row
stripes, while 1 thread instead of 4 costs 24%. `STROM_VIDEOCONVERT_THREADS`
(1-64) overrides it, and the resolution logic is split out from the environment
so the wide-machine paths are covered by tests.
`n-threads` is scoped to macOS. No Linux or Windows machine was available to
measure, and a Linux container's visible CPU count routinely overstates its
cgroup quota, so sizing a pool there unmeasured risks oversubscribing shared
hosts. Linux and Windows behaviour is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The block built `videoscale -> videoconvert -> capsfilter`, so a flow that changed both resolution and pixel format walked every frame twice with an intermediate buffer in between. `videoconvertscale` does both in one walk. `autovideoconvert` already covers both passes on the GPU path — it is `Bin/Colorspace/Scale/Video/Converter` and autoplugs a scaler when the caps ask for one — so the collapse applies in both convert modes. The new `VideoConvertMode::convert_scale_element_name()` names the right element for each, leaving `element_name()` alone for the convert-only call sites. Two effects, measured separately on an M2 (4 performance cores), 1080p RGBA source, 300 frames, 20 paired runs with the within-pair order alternated: - Merging the two passes is worth ~10% on its own. - The larger part is that the scale half is now threaded. `videoscale` has the same `n-threads` property and the same default of 1, but nothing ever set it; the merged element goes through `configure_video_convert()`. Together: the isolated conversion stage runs ~62% faster (20/20 pairs), and a full Strom flow ending in x264enc runs 34-41% faster, median 38%, 12/12 pairs. Both order-symmetric. Where only the format changes and scaling is passthrough there is no measurable difference (median -0.9%, 10/20 pairs), well inside this machine's noise floor. The element ID stays "videoscale" so the block's external input pad still resolves and saved flows that link into this block keep working. Both elements default to bilinear scaling, so the output is unchanged; a test pins that and fails if upstream ever diverges.
wagenet
force-pushed
the
wagenet/videoconvertscale
branch
from
September 1, 2026 04:46
d856a8a to
60dc59a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Video Format block built
videoscale -> videoconvert -> capsfilter, so a flow changing both resolution and pixel format walked every frame twice with an intermediate buffer between.videoconvertscaledoes both in one walk.autovideoconvertalready covers both passes — it isBin/Colorspace/Scale/Video/Converterand autoplugs a scaler when the caps ask for one (verified withgst-launch-1.0: 1080p RGBA in, 720p I420 negotiated out). So the collapse applies in both convert modes, not just software. The newVideoConvertMode::convert_scale_element_name()picks the element per mode;element_name()is untouched, so the convert-only sites (NDI, videoenc, devicesrc, CPU vision mixer) don't gain a scaler.Measurements
M2, 4 performance cores, 1080p RGBA, 300 frames, within-pair order alternated. The noise floor here is ±11% on paired runs, and a fixed order has produced sign-flipping "wins" before.
Isolated stage,
gst-launch-1.0, 20 pairs: 1.558 s → 0.589 s, +62%, 20/20 pairs, order-symmetric (+62.2% A-first, +62.0% B-first).Full Strom flow ending in x264enc, fresh backend per measurement, 12 pairs: 2.104 s → 1.292 s, +38% (range 34–41%), 12/12 pairs, order-symmetric (+38.4%, +38.7%).
Format only, scaling passthrough (1080p RGBA → 1080p I420), 20 pairs: median −0.9%, 10/20 pairs. No measurable difference — a passthrough
videoscaleis free.Most of that is not pass-merging. A third arm,
videoscale n-threads=4 ! videoconvert n-threads=4, splits it: merging the passes is worth ~10% (median +10.5%, 17/20 pairs). The rest is that the scaling half is now threaded —videoscalehas the samen-threadsproperty and the same default of 1, but nothing ever set it, and the merged element goes throughconfigure_video_convert().Scaling method
Both elements share
GstVideoScaleMethodand default to1, "bilinear"(GStreamer 1.28.6), so scaled output is unchanged and nomethodpin is needed.scaling_method_default_matches_videoscalefails if upstream diverges.External pads
Saved flows store links as
block_id:external_pad_nameand resolve them through the block definition at build time.VideoFormatdoesn't overrideget_external_pads(), so it never persistscomputed_external_padsand nothing stale is baked into saved JSON. The risk is a definition naming an element the builder no longer creates, so the surviving element keeps the IDvideoscaleandexternal_padsis unchanged. That matches the convention already in the file: the ID names the role, not the factory, which is why the convert element was calledvideoconvertwhile beingautovideoconvert.Verified end to end — a flow linking
vfmt:video_inandvfmt:video_outwas created and started against the real backend, builtvideoconvertscale -> capsfilter, played, and reached EOS.Other videoscale sites
None.
videoformat.rswas the only place building one, andthumbnail_tap.rsalready usesvideoconvertscale. A stale doc comment inthumbnail.rsnaming elements that module no longer references is corrected here.Tests
builds_one_conversion_element_not_two— two elements, one internal link, and the conversion element both converts and scales.one_element_both_scales_and_converts— 1080p RGBA into the block, 1280x720 I420 out.conversion_element_is_threaded(macOS) —n-threadsreaches the merged element and is above the default of 1.external_pads_resolve_to_built_elements— every external pad resolves to an element the builder creates.scaling_method_default_matches_videoscalegpu::convert_scale_element_name_scales— the mode mapping, plus a guard thatautovideoconvertstill advertisesScale.The first three were verified to fail against a temporary revert of
build()to the old chain. The last three pass under that revert; they guard a future rename, future upstream drift, and a future mode-mapping mistake rather than this change.All six use core and gst-plugins-base elements plus
autovideoconvertfrom gst-plugins-bad, all installed in CI, so none skip.Ran locally: full
cargo test(552 lib tests plus all integration tests, includingpipeline_lifecycle_test) green, with one pre-existing unrelated ignore (test_jitterbuffer_stalls_without_drop_on_latency);cargo buildandcargo clippy --all-targets -- -D warningsclean.Not run: Linux or Windows. macOS always resolves to
Software, so theautovideoconvertbranch ofconvert_scale_element_name()is never exercised here — it rests on thegst-launchcheck and the klass assertion, not a real GPU flow.🤖 Generated with Claude Code