feat(sc): NeMo-Gym path - #3267
Conversation
6afeaae to
62947e5
Compare
8025950 to
5aba114
Compare
62947e5 to
7f1d6ce
Compare
7f1d6ce to
4278c73
Compare
cba2389 to
4d159fb
Compare
850b583 to
faf877c
Compare
63931ad to
3f42aa5
Compare
dd3cb53 to
47b0970
Compare
Update the PR doc describe. |
|
/ok to test 1d6a09c |
RayenTian
added
the
CI:Lfast
|
/ok to test 1d6a09c |
1d6a09c to
b2bd34d
Compare
|
/ok to test b2bd34d |
|
/ok to test 522ab8e |
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
This reverts commit 0eeab08. Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: ruit <ruit@nvidia.com>
Signed-off-by: ruit <ruit@nvidia.com>
Signed-off-by: ruit <ruit@nvidia.com>
The entrypoint restored the NeMo-Gym setup call, which reads
config.env.get("should_use_nemo_gym") in main(). The main_context fixture's
SimpleNamespace config had no `env` attribute, so
test_cleanup_is_best_effort_and_preserves_run_error (and the other main()
tests) raised AttributeError: 'types.SimpleNamespace' object has no
attribute 'env'. Add env={} so the gym branch is skipped in these tests.
Signed-off-by: Terry Kong <terryk@nvidia.com>
…m smoke grpo_async_gym_single_controller.sh overrode loss_fn.reference_policy_kl_penalty=0.01 but inherited grpo.skip_reference_policy_logprobs_calculation=true from the workplace-assistant recipe base, so the loss demanded reference_policy_logprobs that the SC train pump was told to skip -> KeyError: 'reference_policy_logprobs' mid-training. Set skip_reference_policy_logprobs_calculation=false in the smoke so the reference-logprob path is exercised end-to-end on the gym route (the non-gym grpo_dp smoke already trains with the KL penalty). Also harden validate_single_controller_config: a non-zero reference_policy_kl_penalty combined with skip_reference_policy_logprobs_calculation=true now raises at setup with a clear message, instead of failing deep in the loss with an opaque KeyError after the whole cluster/model build. Signed-off-by: Terry Kong <terryk@nvidia.com>
522ab8e to
bad63f0
Compare
|
/ok to test bad63f0 |
Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: ruit <ruit@nvidia.com> Signed-off-by: Terry Kong <terryk@nvidia.com> Co-authored-by: ruit <ruit@nvidia.com> Co-authored-by: Terry Kong <terryk@nvidia.com>
3 participants
Part 4/4 of the #2819 split. Stack on #3266.
#3219 (1/4) -> #3220 (2/4) -> #3266 (3/4) -> #3267 (4/4)
Summary
Wires NeMo-Gym rollouts into the SC (Single Controller) entrypoint by introducing an SC-facing gym-actor spinup helper and wiring it into the SC setup path. The existing
grpo.pypath retains its local_spinup_nemo_gymimplementation for now because it also supplies GRPO-specific timing, routed-experts dtype, and fastokens configuration.spinup_nemo_gym_actor(nemo_rl/environments/nemo_gym.py): factors the NeMo-Gym actor construction needed by SC (config assembly, uv/venv cache reuse, image-baked venv reuse, and softNodeAffinitySchedulingStrategywhennum_gpu_nodes > 0) into a reusable helper. Consolidating the remaining GRPO-specific startup path is left for follow-up work.nemo_rl/algorithms/single_controller_utils/setup.py):setup_single_controllerbranches on_should_use_nemo_gym— skips env spinup insetup_response_data, brings up generation first, then callsspinup_nemo_gym_actorwith the vLLM OpenAI URLs. RaisesNotImplementedErroron non-vllm backends.examples/run_grpo_single_controller.py: restores thesetup_nemo_gym_configcall (removed in feat(sc): setup + entrypoint #3266 while the SC path had no gym support).tests/unit/single_controller/test_single_controller_setup.pygains env-handle wiring + non-vllm backend guard coverage (replaces the previous "not supported yet" trip-wire). Newtests/functional/grpo_async_gym_single_controller.shcovers SC + NeMo-Gym end-to-end.