Skip to content

feat(megatron): expose training CUDA Graph config - #3483

Open
seonjinn wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
seonjinn:sj/nemorl-megatron-cuda-graph-config
Open

feat(megatron): expose training CUDA Graph config#3483
seonjinn wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
seonjinn:sj/nemorl-megatron-cuda-graph-config

Conversation

@seonjinn

@seonjinn seonjinn commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

What

  • Expose cuda_graph_modules and cuda_graph_warmup_steps in the NeMo-RL Megatron config.
  • Forward them to Megatron-Core before TransformerConfig.__post_init__().
  • Add disabled-by-default example configs and focused tests.

Why

NeMo-RL already forwards cuda_graph_impl, but ignored these two existing Megatron-Core options when they were set in YAML.

Configuration

policy:
  megatron_cfg:
    cuda_graph_impl: transformer_engine
    cuda_graph_modules: [attn, mlp]
    cuda_graph_warmup_steps: 3

With cuda_graph_impl: "none", module scopes and warmup are inactive; 3 matches the Megatron-Core default.

Signed-off-by: seonjinn <sna@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@seonjinn seonjinn added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Aug 4, 2026
@seonjinn

seonjinn commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn
seonjinn marked this pull request as ready for review August 4, 2026 02:25
@seonjinn
seonjinn requested review from a team as code owners August 4, 2026 02:25
@seonjinn
seonjinn requested a review from terrykong August 4, 2026 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant