feat (recipe): add Nemotron 3 Ultra MXFP8 GB200 pretraining perf-recipe

[ghadiaravi13]

What does this PR do ?

Add a pre-training perf-recipe for Nemotron-3 Ultra MXFP8 on 256xGB200.

Changelog

• Recipe is similar to the GB300 recipe, with the only difference being TP2 + SP due to smaller HBM size of GB200.

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

• Related to # (issue)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ghadiaravi13]

/ok to test f526067

[ghadiaravi13]

/claude review

[claude]

Light review: feat(recipe) add Nemotron 3 Ultra 256xGB200 MXFP8 perf recipe

Summary: new nemotron_3_ultra_pretrain_256gpu_gb200_fp8mx_config, a GB200 port of the existing GB300 recipe. Differences vs GB300: TP2+SP (smaller GB200 HBM) instead of TP1; min_offloaded_tensor_size lowered 500M to 350M (accounting for the /2 seq split under SP); adds fine_grained_offloading_max_inflight_offloads=1 and graph_capture_record_stream_reuse:True in PYTORCH_CUDA_ALLOC_CONF. Export added to nemotronh/init.py. No functional-logic bugs found; config field names all match existing usage in the deepseek/gb300 recipes.

Minor points (non-blocking):

• The shared _apply_nemotron_3_ultra_fsdp_hsdp helper docstring still says 'on GB300' and its _GB300_NVLINK_DOMAIN_GPUS = 64 constant is GB300-specific; it is reused verbatim for GB200 (256 // 64 = 4 optimizer instances). Please confirm the 64-GPU HSDP sharding domain is correct for GB200 and consider generalizing the name/docstring. (see inline)
• Unlike GB300, the GB200 recipe unrolls the config inline rather than sharing a _nemotron_3_ultra builder. Fine, just noting the divergence between the two files for future maintainers.

Suggested test cases:

• test_perf_recipe_factory_builds_config[nemotron_3_ultra_pretrain_256gpu_gb200_fp8mx_config]
• test_all_perf_recipe_factories_are_exported

No perf launcher/functional tests impacted (256-GPU recipes exceed the 2-GPU CI budget; scripts/performance/configs/ has no nemotron entries).

[claude]

The GB200 recipe drives HSDP via _apply_nemotron_3_ultra_fsdp_hsdp, whose docstring still says "for Nemotron 3 Ultra on GB300", and which computes num_distributed_optimizer_instances from _GB300_NVLINK_DOMAIN_GPUS = 64 (documented as GB300 "16 nodes x 4 GPUs"). For this 256-GPU GB200 recipe that yields 256 // 64 = 4 optimizer instances. Since this recipe advertises NVLINK_DOMAIN_SIZE=72 while sharding on a 64-GPU domain, please confirm the 64-GPU sharding domain is the intended HSDP grouping for GB200 (it does line up with EP64 / NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN=64). If so, consider renaming/generalizing the GB300-specific helper name and constant, or at least updating the docstring so the GB200 reuse isn't surprising.

[malay-nagda]

/ok to test ab1ae70

[claude]

Light review - LGTM overall.

This adds nemotron_3_ultra_pretrain_256gpu_gb200_fp8mx_config, a GB200 twin of the existing nemotron_3_ultra_pretrain_256gpu_gb300_fp8mx_config, differing intentionally in TP2+SP (smaller GB200 HBM) and min_offloaded_tensor_size (350M vs 500M elems, for the SP-halved seq). The export is wired into init.py, and the AST-based env-var test is extended to resolve local-constant references (needed because this recipe references locals like hybrid_ep_ranks_per_nvlink_domain inside env_vars).

Correctness checks pass:

• Auto-covered by test_all_perf_recipe_factories.py (offline build) and test_perf_recipe_environment.py (env invariants). NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN (64) <= NVLINK_DOMAIN_SIZE (72) holds.
• The test change is backward-compatible: non-Name values still go through literal_eval, and any Name value must be a known local constant (asserted).

Minor (non-blocking):

• _apply_nemotron_3_ultra_fsdp_hsdp and the _GB300_NVLINK_DOMAIN_GPUS constant it divides by are now shared with this GB200 recipe, but the helper docstring still reads 'for Nemotron 3 Ultra on GB300'. Functionally fine (256 // 64 = 4 optimizer instances is the intended HSDP layout for both) but the GB300-specific naming/docstring is now slightly stale. Consider generalizing to GB200/GB300.

Suggested test cases:

• nemotron_3_ultra_pretrain_256gpu_gb200_fp8mx_config (new: offline build via test_all_perf_recipe_factories.py)
• nemotron_3_ultra_pretrain_256gpu_gb200_fp8mx_config env invariants via test_perf_recipe_environment.py (exercises new local-constant resolution)
• nemotron_3_ultra_pretrain_256gpu_gb300_fp8mx_config (regression: shares _apply_nemotron_3_ultra_fsdp_hsdp / _GB300_NVLINK_DOMAIN_GPUS)

No scripts/performance/configs/ perf launcher configs are touched, so no GPU perf launch cases are impacted.

[malay-nagda]

/ok to test 38cfd0d