Skip to content

feat: serve Wan2.2-T2V-A14B on TRT-LLM VisualGen (dual guidance, cache_dit, software-encode, thread caps) - #25

Merged
Johan-de-R merged 10 commits into
deep-main-v1.1.1-videogenfrom
johan/wan22-t2v-visualgen
Jul 30, 2026
Merged

feat: serve Wan2.2-T2V-A14B on TRT-LLM VisualGen (dual guidance, cache_dit, software-encode, thread caps)#25
Johan-de-R merged 10 commits into
deep-main-v1.1.1-videogenfrom
johan/wan22-t2v-visualgen

Conversation

@Johan-de-R

@Johan-de-R Johan-de-R commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

What

Make the TRT-LLM diffusion (VisualGen) backend serve Wan-AI/Wan2.2-T2V-A14B on a 1×B200 at the benchmarked latency/quality. Five related changes:

  1. Dual guidance — Wan 2.2 A14B's two-expert MoE guidance (e.g. 4.0 / 3.0 at boundary 0.875). Adds default_guidance_scale_2 / default_boundary_ratio (Wan extra_param_specs), default None → single guidance, so Flux/others are unaffected.
  2. Default the video response to inline b64_json — deepapi reads the clip from data[0].b64_json; the old "url" default returned an unreadable file reference (ERR_MODEL "No video data").
  3. Wire cache_dit (--cache-backend cache_dit) — the ~2× step-skip Wan 2.2 supports (it REJECTS teacache, which hard-crashes worker init). Without it, gen runs the full 50 evals (~2× slower). Builds a CacheDiTConfig(); VisualGen's defaults are already tuned for few-step.
  4. Software mp4 encode (PyAV/libx264) — datacenter GPUs (B200/H100/A100) have no NVENC, and the in-tree imageio ffmpeg has no software h264, so encode_to_video_bytes (h264_nvenc) failed at encode on every request. Rewrote to PyAV/libx264 (self-contained wheel) → temp file → BT.709 tags. ~1s for 720p/81f.
  5. Cap video-worker CPU threads — one shared dynamo.common helper (limit_video_worker_threads) called once at worker startup, with three env-overridable knobs (DI_VIDEO_TORCH_THREADS=12, DI_VIDEO_ENCODE_THREADS=12, DI_VIDEO_COMPILE_THREADS=32) + OMP_WAIT_POLICY=PASSIVE/KMP_BLOCKTIME=0, so a video worker doesn't grab every core on a shared GPU node. Intended as the general pattern for all video workers (torch pool, libx264 encoder, Inductor compile pool — the three CPU consumers, which peak in disjoint phases so the budgets don't sum).

Validation (1×B200, di-slc-37, rc14 runtime image)

  • Reproduces the bench: ~269s/clip (bf16, 720p, 81f, 50 steps, goff 4.0/3.0 @ 0.875, cache_dit), verified 4× back-to-back under full node load (7 neighbor GPUs at ~98%). Direct-engine control = 268s = the 258–264s bench.
  • cache_dit skip confirmed active (Executed Steps 16/34) through the full --cache-backend CLI → DiffusionConfig.from_config flow (previously only the hand-built-config path was validated).
  • Software-encode: valid BT.709 h264 720p/81f mp4, no NVENC device needed.
  • py_compile clean; the dynamo pytest suite runs in CI (not runnable in the backend di-main env). Existing diffusion tests updated (b64 default) + guidance-forwarding coverage added.

Paired backend PRs (deepinfra/backend)

🤖 Generated with Claude Code

@Johan-de-R
Johan-de-R temporarily deployed to external_collaborator July 23, 2026 16:29 — with GitHub Actions Inactive
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 24, 2026 21:46 — with GitHub Actions Failure
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 25, 2026 03:57 — with GitHub Actions Failure
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 28, 2026 00:43 — with GitHub Actions Failure
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 28, 2026 17:02 — with GitHub Actions Failure
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 28, 2026 21:44 — with GitHub Actions Failure
@Johan-de-R Johan-de-R changed the title trtllm diffusion: dual guidance (guidance_scale_2 / boundary_ratio) for Wan 2.2 trtllm diffusion: serve Wan2.2-T2V-A14B (dual guidance, cache_dit, software-encode, thread caps) Jul 28, 2026
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 28, 2026 21:56 — with GitHub Actions Failure
@Johan-de-R Johan-de-R changed the title trtllm diffusion: serve Wan2.2-T2V-A14B (dual guidance, cache_dit, software-encode, thread caps) feat: serve Wan2.2-T2V-A14B on TRT-LLM VisualGen (dual guidance, cache_dit, software-encode, thread caps) Jul 28, 2026
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 28, 2026 23:09 — with GitHub Actions Failure
@github-actions github-actions Bot added the feat label Jul 28, 2026
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 28, 2026 23:15 — with GitHub Actions Failure
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 28, 2026 23:27 — with GitHub Actions Failure
Johan-de-R and others added 10 commits July 29, 2026 00:02
@Johan-de-R @claude
trtllm diffusion: add guidance_scale_2 / boundary_ratio to DiffusionC…
c7abb53 
…onfig

Wan 2.2 A14B is a two-expert MoE that needs dual guidance (e.g. 4.0/3.0 with a
0.875 timestep boundary). The diffusion worker only exposed a single
--default-guidance-scale; guidance_scale_2 / boundary_ratio (Wan extra_param_specs)
defaulted to None => single guidance. Add:

- DiffusionConfig.default_guidance_scale_2 / default_boundary_ratio + the CLI flags
  --default-guidance-scale-2 / --default-boundary-ratio (backend_args).
- DiffusionEngine.generate() injects them into req.params.extra_params BEFORE the
  spec-default merge so they take effect.
- video_handler forwards the config defaults (config-only; the /v1/videos nvext
  doesn't carry them).

Both default to None, so existing models (Flux, Wan single-guidance) are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Johan-de-R @claude
trtllm diffusion: default video response to inline b64_json
bbac1c1 
The video handler defaulted response_format to "url", which uploads the clip to
media_output_fs_url and returns a file reference. deepinfra's deepapi reads the
video from data[0].b64_json and never sends response_format, so it received no
video (ERR_MODEL "No video data"). Default to b64_json so the clip is returned
inline; callers can still pass response_format="url" explicitly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Johan-de-R @claude
trtllm diffusion tests: fix default-response-format test for b64 + co…
fb342e4 
…ver guidance forwarding

The b64_json default change flipped VideoGenerationHandler's default from "url"
to "b64_json", which broke test_default_response_format_is_url (it asserted
upload_to_fs was called and data[0].url was set). Rewrote it to
test_default_response_format_is_b64_json (no upload; data[0].b64_json set) so the
new default is pinned.

Also added test_guidance_defaults_forwarded_to_engine: the handler must forward
config.default_guidance_scale_2 / default_boundary_ratio to engine.generate()
(dual guidance isn't carried in the /v1/videos nvext) -- the core of this PR had
no test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Johan-de-R @claude
trtllm diffusion: wire cache_dit backend (Wan 2.2 ~2x step-skip)
416c11e 
VisualGen supports cache_dit (DBCache/TaylorSeer/SCM) and Wan 2.2 requires it
(it rejects teacache), but the dynamo backend only ever built a TeaCacheConfig.
Add a --cache-backend selector; when "cache_dit", build a CacheDiTConfig and
pass it as VisualGenArgs.cache (defaults are tuned for few-step -> ~2x skip).
teacache path preserved via --enable-teacache / --cache-backend teacache.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Johan-de-R @claude
trtllm diffusion: software (PyAV/libx264) mp4 encode for NVENC-less GPUs
2a514d3 
encode_to_video_bytes drove h264_nvenc (hardware), which has no capable
device on datacenter compute GPUs (B200/H100/A100) -> every /v1/videos
request failed at encode ("No capable devices found"). The in-tree imageio
ffmpeg has no software h264 fallback either (nvenc + libvpx-vp9 only).

Encode via PyAV/libx264 (self-contained, bundled) to a temp file instead:
works on B200 (CPU software), ~0.7s for 720p/81f, BT.709-tagged, and no
in-memory ffmpeg output pipe to stall on. Defensive batch-dim squeeze for
MediaOutput.video's (B,T,H,W,C) shape. Worker image adds `pip install av`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Johan-de-R @claude
trtllm diffusion: cap video-worker CPU threads (good neighbor on shar…
1b4c824 
…ed GPU nodes)

Video workers left uncapped size torch's intra-op pool to the physical core count
(~112 on a 224-thread box), libx264 to every core, and the Inductor compile pool
to 32 — all of which hog a shared GPU node and thrash under contention (a 720p/81f
encode took 244s vs ~1s). Add one shared helper in dynamo.common with three
independent, env-overridable knobs, called once at each video worker's startup:

  - DI_VIDEO_TORCH_THREADS   (default 12) torch.set_num_threads; gen maxes useful
                             CPU parallelism at ~12 (12 == 32 threads -> same 268s),
                             so >12 buys no speed. 12 is well under the per-GPU fair
                             share (224/8 = 28), so full speed AND neighborly.
  - DI_VIDEO_ENCODE_THREADS  (default 12) libx264 thread_count; measured PyAV knee.
  - DI_VIDEO_COMPILE_THREADS (default 32) TORCHINDUCTOR_COMPILE_THREADS (boot-only).

Also sets OMP_WAIT_POLICY=PASSIVE / KMP_BLOCKTIME=0 so idle pools sleep between the
sequential compile -> generate -> encode phases and hand their cores to the active
phase instead of spinning. The three phases never overlap, so the budgets don't sum.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Johan-de-R @claude
trtllm diffusion tests: realign encode tests to PyAV + cover thread caps
112d0d4 
The PyAV rewrite (51d380b) replaced imageio/BytesIO with av/tempfile in
encode_to_video_bytes but left the whole TestEncodeToVideoBytes class mocking
imageio.v3 + io.BytesIO -- stale against the real code (and asserting the wrong
exception type: unsupported format now raises ValueError, not RuntimeError).

Rewrite them to exercise the real PyAV encoder (pytest.importorskip("av")):
decode the output and assert h264 / vp9, BT.709 color tags, the (B,T,H,W,C)->
(T,H,W,C) squeeze, and ValueError on an unsupported container. Add coverage for
the new thread-cap helper: DI_VIDEO_ENCODE_THREADS default/override, and
limit_video_worker_threads setting OMP_WAIT_POLICY / KMP_BLOCKTIME /
TORCHINDUCTOR_COMPILE_THREADS + torch.set_num_threads (with DI_VIDEO_TORCH_THREADS
override).

Verified: 9 passed under pytest 9.0.3 in the rc14 runtime image.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Johan-de-R @claude
style: pre-commit formatting (drop unused io import, black wraps)
e785a5e 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Johan-de-R @claude
style: pre-commit formatting for the guidance/cache/test commits (bla…
0f913c3 
…ck/isort)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Johan-de-R @claude
fix: address adversarial review of the video-diffusion changes
a9f16fe 
- Make guidance mode + cache backend observable so goff can't silently degrade
  to single guidance (the flags are config-only; a missing --default-guidance-
  scale-2 falls back to single with no error): log DUAL/SINGLE (+ values,
  cache_backend) at worker init, add guidance to the per-request handler log,
  and include guidance_scale/guidance_scale_2/boundary_ratio/cache_backend in
  DiffusionConfig.__str__.
- backend_args: constrain --cache-backend to choices [none, teacache, cache_dit]
  (a typo previously fell through to no-cache -> ~2x slower, no error).
- video_handler: replace the exact-5D assert (AssertionError on 4D, stripped
  under -O) with explicit (T,H,W,C)/(1,T,H,W,C) handling + a clear RuntimeError.
- video_utils: warn if limit_video_worker_threads() runs after torch is already
  imported (OMP_WAIT_POLICY/KMP_BLOCKTIME would not bind) + honest docstring;
  fix the stale imageio -> PyAV Raises doc.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Johan-de-R
Johan-de-R force-pushed the johan/wan22-t2v-visualgen branch from 5d1070e to a9f16fe Compare July 29, 2026 00:02
@Johan-de-R
Johan-de-R had a problem deploying to external_collaborator July 29, 2026 00:02 — with GitHub Actions Failure
@Johan-de-R
Johan-de-R marked this pull request as ready for review July 30, 2026 03:53
@Johan-de-R
Johan-de-R merged commit b569cc2 into deep-main-v1.1.1-videogen Jul 30, 2026
13 of 16 checks passed
@sopwg612 sopwg612 mentioned this pull request Aug 12, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backend::trtllm feat

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Johan-de-R