Skip to content

Add Omnidream interactive upsampler client#350

Draft
gtong-nv wants to merge 1 commit into
mainfrom
dev/gtong/omnidream-upsample
Draft

Add Omnidream interactive upsampler client#350
gtong-nv wants to merge 1 commit into
mainfrom
dev/gtong/omnidream-upsample

Conversation

@gtong-nv

@gtong-nv gtong-nv commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds an optional async FlashVSR uplift stream for the OmniDreams interactive-drive world-model backend.

When upsampling_enabled is set in the world-model manifest, generated chunks are queued to a FlashVSR gRPC server without blocking the interactive rendering loop. The client defaults to display_only, so the FlashVSR HTTP viewer owns presentation while interactive-drive continues to show the original model output plus an on-screen uplift status message.

Changes

  • Adds UpliftStreamClient and UpliftStreamConfig for best-effort chunk streaming to FlashVSR.
  • Wires uplift lifecycle into WorldModelRenderBackend:
    • starts the uplift client from manifest settings,
    • resets the stream on scene/model reset,
    • closes the stream with the backend,
    • submits first and steady-state generated chunks asynchronously.
  • Adds manifest knobs for uplift server, scale, sparse ratio, input encoding, JPEG quality, return-frame mode, queue depth, and message size.
  • Extends example world-model manifests with the new upsampling settings.
  • Allows the FlashDreams adapter to select normal recipe slugs when upsampling is enabled.
  • Adds manifest and adapter tests for the new upsampling config surface.
  • Adds an uplift optional dependency extra for flashdreams-flashvsr, which provides the FlashVSR gRPC protos/client surface only when upsampling is enabled.

Notes

  • The PR does not yet route upsampled frames back into interactive-drive. upsampling_return_frames is present as a future hook, but the default path drains responses only to observe server errors/backpressure.

  • The uplift queue is intentionally bounded. If the remote server falls behind, chunks are dropped and the status message reports the drop count instead of stalling rendering.

  • Normal interactive-drive usage does not require FlashVSR. To run with upsampling_enabled=true, sync/run with both extras:

    uv sync --package flashdreams-omnidreams --extra interactive-drive --extra uplift
    uv run --package flashdreams-omnidreams --extra interactive-drive --extra uplift interactive-drive
  • Installing the uplift extra installs the FlashVSR Python package surface needed by the gRPC client. It does not download FlashVSR checkpoints; those are fetched only when the FlashVSR server/pipeline is instantiated.

@copy-pr-bot

copy-pr-bot Bot commented Jun 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant