Skip to content

Support Multi-Backend for Demo Scripts#5938

Merged
ooctipus merged 12 commits into
isaac-sim:developfrom
YizeWang:yizew/support-multi-backends
Jun 7, 2026
Merged

Support Multi-Backend for Demo Scripts#5938
ooctipus merged 12 commits into
isaac-sim:developfrom
YizeWang:yizew/support-multi-backends

Conversation

@YizeWang

@YizeWang YizeWang commented Jun 3, 2026

Copy link
Copy Markdown

Description

This PR updates the demo scripts to support running across the default KitViz/PhysX path and NewtonViz/Newton-MJWarp backend combinations.

The affected demos now use consistent CLI parsing, import ordering, usage comments, default Kit visualizer behavior, and visualizer-aware simulation loops. It also tunes selected Newton MJWarp solver parameters for demos that need more stable articulation/contact behavior, avoids resolving InteractiveScene before simulation launch in bin_packing.py, and adds Newton visualizer support for SimulationContext.set_camera_view().

Dependencies

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Documentation update

Validation

Validated the demo matrix for:

  • scripts/demos/arl_robot_1.py
  • scripts/demos/arms.py
  • scripts/demos/bin_packing.py
  • scripts/demos/bipeds.py
  • scripts/demos/hands.py
  • scripts/demos/quadcopter.py
  • scripts/demos/quadrupeds.py

with:

  • default Kit visualizer + PhysX
  • Newton visualizer + PhysX
  • default Kit visualizer + Newton MJWarp
  • Newton visualizer + Newton MJWarp

Screenshots

arms bin_packing bipeds hands_fixed quadcopter quadrupeds

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Jun 3, 2026

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

This PR migrates demo scripts from the legacy AppLauncher pattern to the new launch_simulation context-manager approach, adding multi-backend support (PhysX / Newton MJWarp) and multi-visualizer support (Kit / Newton). The structural changes are consistent and well-organized.

Below are some findings to consider:


Findings

# Severity File Finding
1 🟡 Medium bin_packing.py string_to_callable used instead of direct import
2 🟡 Medium simulation_context.py is_headless_or_exist_active_visualizer() naming & semantics
3 🟠 Low Multiple demos Duplicated MJWarp solver tuning blocks
4 🟠 Low newton_visualizer.py set_camera_view silently ignores calls before initialization
5 🟠 Low bin_packing.py Prim path change may break existing saved scenes
6 🟢 Nit arl_robot_1.py Unused import from isaaclab_contrib.assets import Multirotor removed but Multirotor class usage replaced silently

Overall this is a solid, well-tested migration. The inline comments below provide more detail on the actionable items.---

Update (2eeea78): Reviewed incremental changes (1 commit, 6 files).

Changes observed:

  • arl_robot_1.py reverted to legacy AppLauncher pattern (removed from multi-backend migration scope)
  • Added TYPE_CHECKING guards for runtime-unused type annotations (Articulation, RigidObjectCollection) in arms.py, bipeds.py, hands.py, quadrupeds.py, bin_packing.py
  • Import ordering: moved isaaclab_newton.physics imports above isaaclab.sim imports

Previous findings status:

  • Finding 1 (string_to_callable in bin_packing.py): Still present — awaiting maintainer decision (discussion ongoing)
  • Finding 2 (is_headless_or_exist_active_visualizer naming): No change — resolved in discussion
  • Finding 3 (Duplicated MJWarp tuning): No change — accepted as intentional for demo clarity
  • Finding 4 (set_camera_view early return): No change — resolved in discussion
  • Finding 5 (Prim path change): No change — resolved in discussion
  • Finding 6 (arl_robot_1.py): ✅ Moot — script reverted to AppLauncher pattern, no longer part of multi-backend migration

New observations:

  • The TYPE_CHECKING guards are a clean approach to avoid circular/heavy imports at runtime while maintaining type safety for static analysis. 👍
  • Import reordering of isaaclab_newton.physics above isaaclab.sim is consistent across all modified demo scripts.

No new issues found. The incremental changes are clean.


Update (f8d7e25): Reviewed incremental changes (1 commit, 3 files).

Changes observed:

  • Import reordering fix in arms.py, bipeds.py, hands.py: moved isaaclab_newton.physics import below isaaclab.sim / isaaclab.physics imports and added # isort:skip annotation
  • This corrects the import ordering from the previous commit (which had placed the newton import above isaaclab imports) to follow the project's import convention: first-party (isaaclab.*) before extension packages (isaaclab_newton.*, isaaclab_assets.*), with # isort:skip to prevent automated reordering

Previous findings status:

  • Finding 1 (string_to_callable in bin_packing.py): No change — awaiting maintainer decision
  • Findings 2–6: Previously resolved in discussion

No new issues. The import ordering fix is clean and consistent across all three files. ✅


Update (5ebc064): Reviewed incremental changes (1 commit, 1 file).

Changes observed:

  • hands.py: MJWarp solver tuning parameters updated — njmax 128→200, impratio 100→10.0, removed ls_iterations/ls_parallel/integrator, added update_data_interval=2, ccd_iterations=50, debug_mode=False
  • These are physics solver configuration refinements (likely for improved simulation stability/performance)

Previous findings status:

  • Finding 1 (string_to_callable in bin_packing.py): No change — awaiting maintainer decision
  • Findings 2–6: Previously resolved in discussion

No new issues. The solver parameter changes are straightforward configuration tuning. ✅


Update (44473f5): Reviewed incremental changes (1 commit, 1 file).

Changes observed:

  • hands.py: Added Shadow Hand Newton MJWarp-specific robot configuration
    • New import: ImplicitActuatorCfg from isaaclab.actuators
    • New import: ShadowHandRobotCfg from isaaclab_tasks.core.shadow_hand.shadow_hand_env_cfg
    • Created SHADOW_HAND_NEWTON_MJWARP_CFG that overrides actuator settings:
      • fingers actuator: stiffness=20.0, damping=2.0
      • New distal_passive actuator for tendon-coupled joints ((FF|MF|RF)J4, LFJ5) with stiffness=10.0, damping=0.1, friction=1e-2, armature=2e-3
    • Scene now conditionally uses SHADOW_HAND_NEWTON_MJWARP_CFG when --physics newton_mjwarp is specified
    • Camera position adjusted: eye [0.0, -0.35, 1.1][0.0, -0.5, 1.5] (pulled back for better view)

Previous findings status:

  • Finding 1 (string_to_callable in bin_packing.py): No change — awaiting maintainer decision
  • Findings 2–6: Previously resolved in discussion

New observations:

  • The distal_passive actuator correctly targets the tendon-coupled distal joints (J4/J5) that require different control parameters than the active finger joints
  • Using ShadowHandRobotCfg().newton_mjwarp as a base and then customizing actuators is a clean pattern that reuses existing validated configurations
  • The camera adjustment improves visibility for the dexterous hand demo

No new issues found. The tendon configuration fix is well-structured. ✅


Update (439a91d): Reviewed incremental changes (1 commit, 1 file).

Changes observed:

  • quadcopter.py: Method call renamed from set_forces_and_torques() to set_forces_and_torques_index() — likely aligning with a renamed API method in the wrench composer class.

Previous findings status:

  • Finding 1 (string_to_callable in bin_packing.py): No change — awaiting maintainer decision
  • Findings 2–6: Previously resolved in discussion

No new issues. Single method name correction to match updated API. ✅


Update (444f582): Reviewed incremental changes (1 commit, 3 files in isaaclab_newton package).

Changes observed:

  • newton_manager.py: Fixed deferred CUDA graph capture to preserve first-step external wrenches during warmup. State is now snapshot/restored around the warmup step so staged writes (e.g., external forces) aren't consumed before the user-visible graph replay.
  • New test test_deferred_relaxed_capture_preserves_first_step_external_wrench: Validates that applying an upward force exactly cancelling gravity results in zero net velocity after one step with deferred graph capture.
  • New changelog entry documenting the fix.

Previous findings status:

  • Finding 1 (string_to_callable in bin_packing.py): No change — awaiting maintainer decision
  • Findings 2–6: Previously resolved in discussion

No new issues. The state snapshot/restore pattern is correct and the test coverage is thorough. ✅


Update (656fd2b): Reviewed incremental changes (1 commit, 1 file).

Changes observed:

  • hands.py: Removed the custom SHADOW_HAND_NEWTON_MJWARP_CFG configuration block (custom actuator stiffness/damping/friction/armature overrides) and the ImplicitActuatorCfg import. Now uses ShadowHandRobotCfg().newton_mjwarp directly in the scene design function — a simplification indicating the upstream robot config has been updated with appropriate defaults.

Previous findings status:

  • Finding 1 (string_to_callable in bin_packing.py): No change — awaiting maintainer decision
  • Findings 2–6: Previously resolved in discussion

No new issues. The simplification is clean — removing demo-local actuator overrides in favor of using the canonical robot config is the right direction. ✅


Update (8527930): Previous concern addressed. No new issues.

  • Fixed: quadrupeds.py now uses scene_origins.to(sim.device) instead of wrapping with torch.tensor() — exactly as suggested.
  • Finding 1 (string_to_callable in bin_packing.py): No change — still awaiting maintainer decision.
  • Findings 2–6: Previously resolved in discussion.

Clean single-line fix, no new issues introduced.


Update (4090883): PR rebased/force-pushed. Reviewed incremental changes vs previous HEAD.

Changes observed:

  • quadrupeds.py: Added parser.set_defaults(visualizer=["kit"]) for consistency with other demos; define_origins now returns torch.Tensor directly (avoids the .tolist()torch.tensor() round-trip); removed unused sim_time variable; split joint pos/vel writes for clarity
  • scripts/demos/sensors/cameras.py: Fixed save_images_grid subtitle lists — added missing "TiledCamera" label for RGB and depth grids
  • All demos: rebased onto latest develop (commit timestamps updated); no content changes to the multi-backend migration logic

Previous findings status:

  • Finding 1 (string_to_callable in bin_packing.py): No change — still present, awaiting maintainer decision
  • Findings 2–6: Previously resolved in discussion

No new issues found. The incremental changes are clean housekeeping (type annotation improvement, missing subtitle fix, unused variable removal). ✅---

Update (47b38b8):Finding 1 resolved! All previous concerns now addressed.

Changes observed:

  • bin_packing.py: Removed string_to_callable import and replaced string_to_callable("isaaclab.scene:InteractiveScene")(scene_cfg) with scene_cfg.class_type(scene_cfg) — clean, type-safe, no dynamic string resolution at runtime
  • interactive_scene_cfg.py: Added class_type: type[InteractiveScene] | str = "{DIR}.interactive_scene:InteractiveScene" attribute to InteractiveSceneCfg with proper TYPE_CHECKING guard for the type annotation

Previous findings status:

  • Finding 1 (string_to_callable in bin_packing.py): ✅ Resolved — replaced with class_type pattern from the cfg class, which is the idiomatic IsaacLab approach
  • Findings 2–6: Previously resolved in discussion

Assessment: The class_type pattern on the cfg dataclass is consistent with how other IsaacLab components (e.g., AssetBaseCfg, SensorBaseCfg) handle class resolution — much better than runtime string_to_callable. All originally flagged issues are now resolved. This PR is ready from a code quality perspective. ✅


Update (d96bf14): New commits only add changelog entries (source/isaaclab/changelog.d/yizew-support-multi-backends.rst and source/isaaclab_visualizers/changelog.d/yizew-support-multi-backends.rst). No code changes. No new issues. All previously flagged concerns remain resolved. ✅

sim.reset()
# Now we are ready!
print("[INFO]: Setup complete...")
# Run the simulator

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium: Indirect construction via string_to_callable

Using string_to_callable("isaaclab.scene:InteractiveScene")(scene_cfg) here is unusual compared to the other demos which import and call classes directly. This pattern:

  1. Obscures the dependency — readers won't immediately see what's being constructed.
  2. Bypasses static analysis / IDE navigation.
  3. Differs from the approach in every other demo file in this PR.

Is there a specific reason InteractiveScene cannot be imported directly here (e.g., circular import when Newton backend is active)? If so, a brief comment explaining why would help future maintainers. If not, consider using the direct import for consistency:

from isaaclab.scene import InteractiveScene
scene = InteractiveScene(scene_cfg)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ooctipus Could you please take a look here? I tried to avoid local imports. Not sure if there exists a better way to achieve.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising this @YizeWang — happy to share my thoughts while you wait for @ooctipus.

Looking at the file, InteractiveScene is actually already directly imported at the top:

from isaaclab.scene import InteractiveScene, InteractiveSceneCfg

So unless there's a runtime reason the import needs to be deferred (e.g., the class identity changes depending on the active physics backend), you could simply use:

scene = InteractiveScene(scene_cfg)

…which would be consistent with the existing import and easier to follow.

If there is a backend-aware dispatch happening (where "isaaclab.scene:InteractiveScene" might resolve to a different class at runtime depending on the physics context), then string_to_callable makes sense — but a short inline comment explaining why would really help future readers. Something like:

# Use string_to_callable to allow backend-specific scene class resolution at runtime.
scene = string_to_callable("isaaclab.scene:InteractiveScene")(scene_cfg)

Deferring to @ooctipus on whether the multi-backend machinery requires this indirection here.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improved in the latest commit "Remove string_to_callable"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks for removing it! 👍

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, using scene_cfg.class_type(scene_cfg) is even cleaner than my suggestion — it keeps the construction tied to the config definition. Thanks for addressing this! 👍

self.get_setting("/isaaclab/video/auto_start_kit")
)

def is_headless_or_exist_active_visualizer(self) -> bool:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium: Method naming and semantics

The method name is_headless_or_exist_active_visualizer is a bit ambiguous — it reads as a query about the state but is actually used as a loop continuation predicate ("should we keep stepping?").

Consider:

  • should_continue_stepping() — clearer intent for the loop usage pattern
  • Or at minimum, update the docstring to clarify: "Returns True when the simulation loop should continue (either no visualizers are attached, or at least one visualizer window is still open)."

Also: the condition not self._visualizers returns True when _visualizers is an empty list. If a visualizer is removed from the list mid-simulation (rather than being marked closed), this would still return True and the loop would run indefinitely. Is that the intended behavior?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the visualizer is closed in the mid way, the second condition will be False.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that makes sense — thanks for clarifying! So the logic is:

  • not self._visualizers → no visualizers attached, headless mode, keep stepping
  • any(viz.is_running() and not viz.is_closed for viz in self._visualizers) → at least one visualizer is still alive

If the user closes the visualizer window mid-run, is_closed becomes True (or is_running() returns False), the any(...) evaluates to False, and the loop exits. That's a clean shutdown path 👍

The naming concern is minor — the current name does reflect what's checked. If you feel it's clear enough in context, happy to leave it as-is.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying! That makes sense — so the intended behavior is:

  • No visualizers attached (not self._visualizers) → keep stepping (headless mode)
  • Visualizer(s) attached but all closed → stop the loop (the any() returns False)
  • At least one visualizer still open → keep stepping

This is reasonable semantics for a user-facing loop guard. My earlier concern about "running indefinitely" was misplaced since you do handle the closed-visualizer case correctly.

The naming suggestion (should_continue_stepping or similar) is still a mild preference for clarity, but I understand if you want to keep the current name — it's not blocking.

Comment thread scripts/demos/arms.py
physics_cfg.solver_cfg.nconmax = 70
physics_cfg.solver_cfg.ls_iterations = 40
physics_cfg.solver_cfg.cone = "elliptic"
physics_cfg.solver_cfg.impratio = 100

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Low: Duplicated solver tuning configuration

This identical MJWarp solver configuration block (njmax, nconmax, ls_iterations, cone, impratio, ls_parallel, integrator, num_substeps) is repeated verbatim across arms.py, bipeds.py, and hands.py.

Consider extracting a shared helper or preset:

# In a shared module, e.g. isaaclab_assets.newton_presets
ARTICULATION_MJWARP_PRESET = dict(
    njmax=70, nconmax=70, ls_iterations=40,
    cone="elliptic", impratio=100,
    ls_parallel=False, integrator="implicitfast",
)

This reduces the maintenance burden when solver defaults change and makes it obvious which demos share physics characteristics.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we want to expose the way to tune parameters.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense — for demo scripts, having the tuning parameters explicit and self-contained is more instructive for users learning how to configure the solver. Keeping each demo standalone avoids an extra indirection that could obscure the workflow.

No objection from me. Thanks for clarifying!

def set_camera_view(
self, eye: tuple[float, float, float] | list[float], target: tuple[float, float, float] | list[float]
) -> None:
"""Set active viewer camera eye/target.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Low: Silent early return may surprise callers

set_camera_view() silently returns (with only a debug log) if the visualizer isn't initialized. Since this is called in main() before sim.reset(), the timing depends on when the Newton visualizer initializes.

If the visualizer initializes lazily (e.g., on first sim.step()), this camera pose will be lost entirely. Consider:

  1. Storing the requested pose and applying it once initialization completes (deferred apply pattern), or
  2. Raising a warning-level log so users notice the no-op during debugging.

The existing SimulationContext.set_camera_view() likely already handles this dispatch — is there a case where this method is called directly on the visualizer instance?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set_camera_view is implemented in the same way as in kit_visualizer.py.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method was previously missing for the Newton visualizer.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! You're right — I can confirm the pattern matches kit_visualizer.py (early return with a debug log when not initialized). Keeping both implementations consistent makes sense.

The suggestion about a deferred-apply pattern was more of a general improvement idea that would apply to both visualizers equally, not something specific to this PR. No action needed here. 👍

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense — thanks for adding it! The implementation looks clean.

My comment was less about whether to add it and more about the early-return behavior: if set_camera_view() is called before the visualizer initializes (which can happen in main() before sim.reset()), the requested pose is silently dropped. That said, if the call site ordering guarantees initialization has already happened by then, this is fine as-is. Not a blocker.

rigid_objects={
f"Object_{label}_Layer{layer}": RigidObjectCfg(
prim_path=f"/World/envs/env_.*/Object_{label}_Layer{layer}",
prim_path=f"/World/envs/env_.*/Groceries/Object_{label}_Layer{layer}",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Low: Prim path hierarchy change

The prim path changed from /World/envs/env_.*/Object_{label}_Layer{layer} to /World/envs/env_.*/Groceries/Object_{label}_Layer{layer} (added Groceries/ intermediate Xform).

This is fine for a fresh run, but could break:

  • Saved USD stages that reference the old paths
  • Any downstream tooling or tests that pattern-match on prim paths

Is there a corresponding change that creates the /Groceries Xform prim? I don't see one in this diff — the InteractiveScene may handle it automatically, but it's worth confirming.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Newton visualizer needs this change. Otherwise, objects cannot be properly visualized.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! That makes sense — if Newton visualizer requires this hierarchy for proper object rendering, then this is the right approach. The InteractiveScene should handle creating the intermediate Xform automatically based on the prim path.

Good to go 👍

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! That makes sense — the Newton visualizer likely needs the intermediate /Groceries Xform to correctly traverse and render the child prims in its scene graph.

Since this is a new demo script (not modifying an existing one), there are no pre-existing saved stages or path references to worry about. My concern about backward compatibility doesn't apply here. 👍

No further concerns on this change.

@greptile-apps

greptile-apps Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR updates seven demo scripts to support a 2×2 matrix of physics backends (PhysX and Newton MJWarp) and visualizer frontends (Kit and Newton), while also adding NewtonVisualizer.set_camera_view() support and a shared SimulationContext.is_headless_or_exist_active_visualizer() helper that unifies the simulation-loop exit condition across all scripts.

  • All demo scripts are migrated from AppLauncher/simulation_app.is_running() to launch_simulation + sim.is_headless_or_exist_active_visualizer(), with robot instantiation moved to cfg.class_type(cfg) for backend-agnostic construction.
  • newton_visualizer.py gains set_camera_view() so camera positioning set before reset() is replayed correctly after Newton visualizer initialization.
  • bin_packing.py changes the InteractiveScene instantiation path via string_to_callable and fixes a prim-path inconsistency in the scene config.

Confidence Score: 4/5

Safe to merge; the changes are additive demo-script updates with one missing import and one redundant wrapper that don't affect runtime correctness.

The core simulation-context and Newton-visualizer changes are clean and well-scoped. The demo scripts are correctly migrated to the new multi-backend launch pattern. The only substantive issues are in bin_packing.py: RigidObjectCollection was dropped from the import but is still used in type annotations (harmless at runtime due to PEP 563, but breaks static analysis), and string_to_callable is applied to a class that is already imported, making the workaround ineffective and confusing. The quadrupeds.py wrapping of an already-tensor return value in torch.tensor() is a minor code-quality concern.

scripts/demos/bin_packing.py — missing RigidObjectCollection import and redundant string_to_callable usage.

Important Files Changed

Filename Overview
scripts/demos/bin_packing.py RigidObjectCollection removed from imports but still used as a type annotation; string_to_callable workaround is redundant since InteractiveScene is still imported at top-level.
scripts/demos/quadrupeds.py define_origins() now returns torch.Tensor but main() still wraps it with torch.tensor(), which is redundant and may emit a UserWarning.
source/isaaclab/isaaclab/sim/simulation_context.py Adds is_headless_or_exist_active_visualizer() helper that consolidates the visualizer-aware loop condition; logic is correct and consistent with existing visualizer state checks.
source/isaaclab_visualizers/isaaclab_visualizers/newton/newton_visualizer.py Adds set_camera_view() override that defers to _apply_camera_pose; guard on _is_initialized ensures pending camera views are replayed correctly after initialization.
scripts/demos/arms.py Migrated to launch_simulation with Newton MJWarp solver tuning; direct imports split per robot file, no issues found.
scripts/demos/bipeds.py Migrated to launch_simulation with Newton MJWarp solver tuning; robot instantiation via class_type, no issues found.
scripts/demos/hands.py Migrated to launch_simulation with Newton MJWarp solver tuning; camera view updated, no issues found.
scripts/demos/quadcopter.py Migrated to launch_simulation; no Newton MJWarp solver tuning needed for quadcopter, no issues found.
scripts/demos/arl_robot_1.py Migrated to launch_simulation; PXR dome-light creation replaced with DomeLightCfg, no issues found.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Script entry point] --> B[parse args: --physics / --visualizer]
    B --> C{launch_simulation context manager}
    C --> D[PhysX backend]
    C --> E[Newton MJWarp backend]
    D --> F{Visualizer frontend}
    E --> G{Tune MJWarp solver params?}
    G -->|arms/bipeds/hands/bin_packing| H[Set njmax, nconmax, ls_iterations, etc.]
    G -->|quadcopter/quadrupeds/arl_robot_1| I[Use defaults]
    H --> F
    I --> F
    F --> J[Kit visualizer]
    F --> K[Newton visualizer]
    J --> L[SimulationContext created]
    K --> L
    L --> M[sim.set_camera_view - pending replayed after init]
    M --> N[scene / robots spawned via cfg.class_type]
    N --> O[sim.reset - visualizers initialized]
    O --> P{sim.is_headless_or_exist_active_visualizer}
    P -->|True| Q[step simulation]
    Q --> P
    P -->|False| R[exit loop / context cleanup]
Loading

Comments Outside Diff (1)

  1. scripts/demos/bin_packing.py, line 551 (link)

    P2 Redundant string_to_callable wrapper

    InteractiveScene is already imported at module level (from isaaclab.scene import InteractiveScene, InteractiveSceneCfg), so string_to_callable("isaaclab.scene:InteractiveScene") resolves to the same class object. The wrapper doesn't defer or avoid any PXR loading — the top-level import already triggers that. If the intent is to instantiate the scene lazily (inside launch_simulation), that is already achieved by the with block; a direct InteractiveScene(scene_cfg) call here would be equivalent and much clearer.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Reviews (1): Last reviewed commit: "Improve Viz" | Re-trigger Greptile

Comment thread scripts/demos/bin_packing.py
Comment thread scripts/demos/quadrupeds.py Outdated
@isaaclab-review-bot

Copy link
Copy Markdown

Update (f8d7e25): Reviewed incremental changes (1 commit, 3 files).

Changes observed:

  • arms.py, bipeds.py, hands.py: Moved isaaclab_newton.physics import back to the # isort:skip section (after isaaclab.* imports)
  • Added # isort:skip comment to the Newton import to match the existing pattern for external asset imports

Assessment: Purely stylistic fix to maintain consistent import ordering with the isort configuration. No functional changes. ✅

./isaaclab.sh -p scripts/demos/quadcopter.py --visualizer newton

# Usage with Newton (MJWarp) physics and default kit visualizer.
./isaaclab.sh -p scripts/demos/quadcopter.py --physics newton_mjwarp

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The quadcopter will fall with KitViz + Newton. No issues in other combinations. Still investigating.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue has been filed in #5961. The fix is beyond this PR.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #5983

Comment thread scripts/demos/hands.py
./isaaclab.sh -p scripts/demos/hands.py --visualizer newton

# Usage with Newton (MJWarp) physics and default kit visualizer.
./isaaclab.sh -p scripts/demos/hands.py --physics newton_mjwarp

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shadow hand does not behave properly with the Newton backend. I tried tuning the MJWarp parameters but had no luck yet. Still investigating.

@YizeWang YizeWang Jun 4, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same tendon issue in #5437. This issue has been fixed in my latest commit: "Fix Shadow Hand Tendons".

Comment thread scripts/demos/hands.py Outdated
if TYPE_CHECKING:
from isaaclab.assets import Articulation

_SHADOW_HAND_NEWTON_MJWARP_CFG = ShadowHandRobotCfg().newton_mjwarp

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default configuration for the shadow hand does not work in the Newton backend. The analysis is in #5437. These lines are mimicking the fix in the quoted PR.

I think we should create a newton config for shadow hand as a new assest instead of hardcoding the actuations in the user scripts, such as https://github.com/isaac-sim/IsaacLab/pull/5437/changes#diff-1e58eac15480c1da1cba788a3a40401333d0fd9919923b9777a4fa94b6f39d95 and here.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to test if #5522 asset was enough and if the additional changes are still needed?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have confirmed that #5522 fixes the tendon issue and the shadow hand works without overriding.

@ooctipus ooctipus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thank @YizeWang

@YizeWang YizeWang force-pushed the yizew/support-multi-backends branch from 444f582 to 656fd2b Compare June 5, 2026 14:12
./isaaclab.sh -p scripts/demos/quadcopter.py --visualizer newton

# Usage with Newton (MJWarp) physics and default kit visualizer.
./isaaclab.sh -p scripts/demos/quadcopter.py --physics newton_mjwarp

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #5983

Comment thread scripts/demos/hands.py Outdated
if TYPE_CHECKING:
from isaaclab.assets import Articulation

_SHADOW_HAND_NEWTON_MJWARP_CFG = ShadowHandRobotCfg().newton_mjwarp

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have confirmed that #5522 fixes the tendon issue and the shadow hand works without overriding.

Comment thread scripts/demos/quadrupeds.py Outdated
Comment thread scripts/demos/bin_packing.py Outdated
# Design scene
scene_cfg = MultiObjectSceneCfg(num_envs=args_cli.num_envs, env_spacing=1.0, replicate_physics=True)
with Timer("[INFO] Time to create scene: "):
scene = string_to_callable("isaaclab.scene:InteractiveScene")(scene_cfg)

@YizeWang YizeWang Jun 5, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ooctipus Could you please take a look here? This is how I tried to avoid local imports. Not sure if there exists a better way to achieve.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can class_type in Cfg. and used cfg.class_type(cfg) pattern. This allow maximum extensibility, flexibility while protect us from bad import chain

@YizeWang YizeWang force-pushed the yizew/support-multi-backends branch from 8527930 to 4090883 Compare June 6, 2026 03:07
@ooctipus

ooctipus commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator

@YizeWang really nice work!

Yize Wang added 12 commits June 7, 2026 13:27
Signed-off-by: Yize Wang <yizew@nvidia.com>
Signed-off-by: Yize Wang <yizew@nvidia.com>
Signed-off-by: Yize Wang <yizew@nvidia.com>
Signed-off-by: Yize Wang <yizew@nvidia.com>
Signed-off-by: Yize Wang <yizew@nvidia.com>
Signed-off-by: Yize Wang <yizew@nvidia.com>
Signed-off-by: Yize Wang <yizew@nvidia.com>
Signed-off-by: Yize Wang <yizew@nvidia.com>
@YizeWang YizeWang force-pushed the yizew/support-multi-backends branch from d96bf14 to b0847bc Compare June 7, 2026 05:28
@ooctipus ooctipus merged commit 0a0dc1b into isaac-sim:develop Jun 7, 2026
60 of 61 checks passed
@YizeWang YizeWang deleted the yizew/support-multi-backends branch June 8, 2026 04:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants