[Dynamo] managed generation backend cleanup tracking issue

[terrykong]

Tracking issue for loose ends in the managed Dynamo generation backend introduced in #3391.
Items will be checked off or split into separate issues as they land.

Scope: the policy.generation.backend: dynamo path — the driver-owned etcd / NATS / Dynamo-frontend
runtime and its Ray-scheduled vLLM worker fleet. Not Kubernetes, DGD, or externally-deployed Dynamo.

Each item has a stable LE-n identifier so review comments can point at a specific one. Identifiers are never reused or renumbered; a completed item keeps its number.

Known missing items

• [LE-1] Multi-node inference (TP/PP spanning nodes) is unsupported — important for large models.
  FixedDynamoWorkerPool.start() hard-rejects any engine group whose GPUs land on more than one node
  (worker_pool.py#L124-L128):
  "A managed Dynamo engine group spans multiple nodes. Multi-node TP/PP is not supported in the fixed-fleet milestone."
  This caps the backend at models whose TP×PP fits inside one node — for the shipped 4-GPU-per-node
  topology that is TP≤4. The vLLM backend already supports this: it derives
  needs_cross_node = model_parallel_size > cluster.num_gpus_per_node, builds a unified placement group,
  and sets NCCL_NVLS_ENABLE=0 for the non-colocated cross-node case
  (vllm_generation.py#L74-L78,
  #L178-L221).
  Two sub-parts: (a) route Dynamo through init_cluster_placement_groups so it gets use_unified_pg
  instead of the catch-all else branch at
  grpo.py#L906-L916; (b) verify dynamo.vllm can be
  driven with a cross-node TP group at all. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-2] Reuse WeightSynchronizer instead of a sixth refit mechanism.
  The refit is wired as isinstance(policy_generation, DynamoGeneration) branches inside the legacy
  path of refit_policy_generation (grpo.py#L2315), bypassing the
  abstraction that feat: add delta-compressed collective refit #2444, feat: add checkpoint-engine refit interface and integrate NIXL #2608 and feat: NCCL-Reshard refit merge PR (final) #2971 all landed through. DynamoGeneration already implements
  every method CollectiveWeightSynchronizer calls, with matching signatures — see the review thread
  for the concrete port. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)
  (edit: asked for in PR feat(dynamo): add managed Slurm generation with W&B telemetry #3391 — the wrong shape here is hard to unwind once dynamo is spread across the algorithm layer)

• [LE-3] AREAL-style cache invalidation (recompute_kv_cache_after_weight_updates: true) is unsupported and fails silently.
  resume_after_refit calls policy_generation.invalidate_kv_cache() from inside AsyncTrajectoryCollector,
  which holds a pickled copy whose _managed_runtime is None, so the call raises and is swallowed by a
  blanket except Exception (trajectory_collector.py#L567-L575).
  The run silently falls back to Magistral-style (keep stale caches) — which is what the shipped recipe selects
  anyway, so this is latent today. Either carry refit_workers() through __getstate__ so the copy can POST
  flush_cache itself, or assert at setup that the flag is false for backend: dynamo.
  (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-4] Support the single-controller path (single_controller.py, the v2 async GRPO loop).
  backend: dynamo is currently rejected there — _generation_max_len raises
  ValueError: Unknown generation backend 'dynamo'
  (single_controller_utils/setup.py#L237) — which
  fails loud, correctly, but means the v2 loop cannot use Dynamo at all. Note that wiring it requires the
  pickled-copy fix above first: unlike trajectory_collector.py, the SC loop calls
  self._gen.invalidate_kv_cache() unguarded
  (single_controller.py#L615-L616), so the same
  _managed_runtime is None failure would hard-crash the run rather than degrade silently.
  SC also builds its synchronizer through create_weight_synchronizer
  (setup.py#L410), so the WeightSynchronizer
  item above is a prerequisite too. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-5] Unit-test both cache-invalidation modes. There is no test that AREAL-style
  (recompute_kv_cache_after_weight_updates: true) actually invalidates, nor that Magistral-style skips it.
  test_pickle_roundtrip_drops_driver_owned_runtime asserts restored._managed_runtime is None but never calls
  restored.invalidate_kv_cache(); the invalidation test covers the driver instance, which production never uses.
  TRT-LLM has the precedent — test_trtllm_generation.py#L296-L318
  parametrizes (in_flight, recompute_kv, expected_drain) across all three reachable combinations.
  (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-6] Fault tolerance on Slurm — and a way to simulate it.
  The managed runtime adds four driver-side single points of failure no other backend has (etcd,
  nats-server, the Dynamo frontend, the token-wrapper uvicorn thread). All liveness checks are
  startup-only — _wait_for_frontend/_start_etcd/_wait_for_port
  (managed_runtime.py#L300) never run again,
  so a frontend that dies mid-run takes the job with it. Worker-side,
  FixedDynamoWorkerPool.validate()
  detects dead workers and reservations before each refit but only raises — no restart, quarantine, or
  re-admission. On a 6-node / 240-minute recipe every one of those losses is currently fatal.

  feat: FT & R #3454 ("FT & R") is building this for the single-controller path (engine_supervisor.py,
  fleet_health.py, policy_router.py, weight_sync/membership.py, shard quarantine,
  recreate_worker()) and validates it with fault injection. Two requirements when Dynamo adopts it:
  (a) reuse feat: FT & R #3454's config schema — async_rl.watchdog.*, async_rl.rollout_failure.*,
  fleet_health.* — rather than a parallel dynamo_cfg-scoped one, so the same knobs mean the same
  thing across backends; (b) make the failure simulation backend-agnostic, able to target a
  dynamo.vllm subprocess, the frontend, etcd or NATS — otherwise Dynamo's four extra SPOFs are exactly
  the ones never exercised. Killing the frontend is the interesting case; it has no analog elsewhere.
  (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-7] Write a usage guide, separate from the design doc.
  docs/design-docs/dynamo-integration.md covers the "why"
  well, but there is no "how do I run this". Add docs/guides/dynamo-generation.md that links back to the
  design doc and then goes straight into: support status (vLLM engine only, non-colocated only, single-node
  engine groups, async rollouts only), the derived-image prerequisite, the dynamo_cfg + vllm_cfg config
  layout with the honoured/moved/ignored key table, the 2-GPU smoke run then the 6-node recipe, how to verify
  the refit landed (token_mult_prob_error), and the backend-specific failure modes. Template:
  checkpoint-engine-refit.md for a feature guide,
  models/qwen/qwen3-5.md for the support-status/recipe shape.
  Register in docs/index.md. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)
  (edit: asked for in PR feat(dynamo): add managed Slurm generation with W&B telemetry #3391)

• [LE-8] Make ignored vllm_cfg keys loud. 12 of the 22 VllmSpecificArgs keys are silently dropped under
  backend: dynamo. Most are genuinely N/A (in-process llm.generate() knobs, the HTTP/ZMQ refit transports,
  async_engine), and enable_return_routed_experts is already guarded by validate_router_replay_config —
  but tool_parser_plugin and reasoning_parser_plugin moved to dynamo_cfg.worker_args.*, so migrating
  a working vLLM config silently changes tool/reasoning parsing. Raise for the moved keys, warn for the
  genuinely unsupported ones (skip_tokenizer_init, is_mx), stay silent for N/A. Typing vllm_cfg as a
  model rather than dict[str, Any] is what makes this enumerable. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)
  (edit: asked for in PR feat(dynamo): add managed Slurm generation with W&B telemetry #3391 — the guard and the four-line recipe edit land together)

• [LE-9] Router replay (MoE routing replay) is unsupported with Dynamo.
  validate_router_replay_config raises
  "router_replay.enabled requires vLLM generation." for any non-vLLM backend, so this fails loud today —
  no correctness issue. Two follow-ups if it is ever wired: (a) the token wrapper overwrites
  nvext.extra_fields with ["engine_data"]
  (token_wrapper.py#L387) rather than merging,
  so Gym's routed_experts request would be silently dropped — ai-dynamo ignores unrecognised
  extra_fields names, so nothing upstream would complain either; (b) the guard is Megatron-only and its
  message says "requires vLLM generation" without mentioning that Dynamo is vLLM but not backend: vllm.
  Worth stating in the usage guide's support-status table regardless. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-10] Ship a recipe on a public model. The 6-node SWE recipe targets Nemotron Nano v3.5, which is not
  public, and model_name defaults to a filesystem placeholder (/path/to/nemotron-nano-v3.5-checkpoint)
  rather than a HuggingFace ID — custom_jinja_template then reads the chat template from inside that same
  private directory. Combined with the recipe being in disabled.txt and having no L1 functional test, nobody
  outside the team can execute the 6-node path at all. The repo already standardises on
  nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 (10 recipes), which is also MoE so it still exercises expert
  parallelism and the packed NCCL refit. The 2-GPU smoke is fine — it uses public Qwen/Qwen2.5-1.5B.
  (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)
  (edit: asked for in PR feat(dynamo): add managed Slurm generation with W&B telemetry #3391 — same one-line edit as the oc.env cleanup)

• [LE-11] Externally-deployed / unmanaged Dynamo. Explicitly out of scope for this milestone
  (dynamo-integration.md#L4). DynamoGeneration
  constructs ManagedDynamoRuntime unconditionally, so this needs a runtime seam.
  (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-12] Engine backends other than vLLM. The engine is the hardcoded literal "dynamo.vllm"
  (dynamo_worker.py#L120); ai-dynamo v1.3.0
  also ships sglang
  and trtllm workers.
  Needs an engine field on DynamoCfg rather than another string literal. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-13] expert_parallel_size is accepted and silently ignored. validate_managed_vllm_config
  checks positivity only (arguments.py#L297-L306),
  and a bare --enable-expert-parallel makes vLLM compute EP = DP×TP = TP. The vLLM backend rejects
  ep != tp outright (vllm_generation.py#L125-L136).
  (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-14] Sync (non-async) rollouts. _should_use_async_rollouts returns True unconditionally for
  dynamo (grpo.py#L1947), so DynamoGeneration.generate() is
  unreachable. Tracking in case a sync entrypoint ever needs it. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)
  (edit: dropped — no known need for a sync entrypoint; deleting the unreachable generate() is part of LE-17)

• [LE-18] Speculative decoding is accepted but never refit. vllm_kwargs.speculative_config
  forwards generically to --speculative-config, so dynamo.vllm starts with a drafter — but the drafter
  refit path lives in VllmInternalWorkerExtension
  (_get_drafter_model, _maybe_refit_mtp_drafter), which the managed path has no equivalent of. Draft weights
  stay at step 0 for the whole run: rejection sampling keeps outputs correct, so nothing errors — acceptance
  rate just decays and throughput falls back toward non-speculative. For MTP the co-trained layer never reaches
  the engine. Supporting this needs a drafter-refit route over Dynamo's admin endpoints, alongside the main
  weight transfer. A guard is requested in this PR; the feature is not. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-19] Low-precision generation (FP8 / MXFP8 / NVFP4) is unsupported. Both mechanisms the vLLM
  backend uses are unavailable out-of-process: init_fp8
  returns hf_overrides and monkeypatches Fp8LinearMethod / ModelOptMxFp8FusedMoE inside NeMo RL's own vLLM
  process, and NVFP4 swaps the worker class via
  resolve_generation_worker_cls →
  VllmQuantGenerationWorker, whereas FixedDynamoWorkerPool hardcodes DynamoVllmWorker. Today precision: fp8
  dies on an unrelated --dtype argparse error and NVFP4 is silently ignored. Real support means passing
  quantization through dynamo.vllm's own CLI rather than patching the process. A guard is requested in this PR;
  the feature is not. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

Known cleanup items

• [LE-15] Automated coverage. The only acceptance recipe ships in tests/test_suites/disabled.txt, the
  five new unit-test files are collected by no CI shard, and docker/dynamo/Dockerfile is never built in
  CI. Needs an L1 functional test (the 2-GPU smoke config is the right size) plus shard registration.
  (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• [LE-16] Port bands. system_port_base: 29000 and the recipes' 25000-28000 / 15001-20000 ranges sit
  above the 9000 GB200 ephemeral floor that fix: configure port ranges to avoid TOCTOU port contention #2380 and fix: Port contention issues between Ray/vLLM/Gym and sandbox workers #3103 established; VLLM_PORT is not forwarded to
  the engine subprocess. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)

• ~~[LE-17] Dead code. ~225 removable lines: unreachable generate(), three TypedDict twins duplicating
  their BaseModel counterparts, _owns_managed_runtime, a dead uv branch, three unreachable guards in
  token_wrapper.py. (Flagged during feat(dynamo): add managed Slurm generation with W&B telemetry #3391 review)~~
  (edit: asked for in PR feat(dynamo): add managed Slurm generation with W&B telemetry #3391 — deletions only)

Add anything else here as it comes up.

[Abubakar-rashid]

Hi, I would like to contribute to this tracking issue. I am interested in taking the task of making ignored vllm_cfg keys explicit, so users are warned when unsupported configuration keys are silently ignored. Would it be okay for me to work on this?

[terrykong]

hi @Abubakar-rashid. thanks for the interest. this is currently just a scratch pad for issues that may or may not need implementing so just a word of caution to implementing anything from this list as the main PR also has not even landed.

i have created this as part of this review https://github.com/NVIDIA-NeMo/RL/pull/3391/changes#diff-8e393d7963fd936eaf1ab5399e4cfe1935b941f87b9d5497fe5f90a3bf62fb34 so much of this is still in flux. feel free to follow along in the review to watch.