Skip to content

Fix GTP prefetch parameter readiness - #6232

Open
JF-D wants to merge 1 commit into
NVIDIA:mainfrom
JF-D:jiangfeid/gtp_ddp_prefetch_readiness
Open

Fix GTP prefetch parameter readiness#6232
JF-D wants to merge 1 commit into
NVIDIA:mainfrom
JF-D:jiangfeid/gtp_ddp_prefetch_readiness

Conversation

@JF-D

@JF-D JF-D commented Aug 4, 2026

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

Summary

  • attach a GTP-only readiness handle to each DDP parameter-gather bucket used by GTP weights
  • finish the required DDP gather before eager GTP weight reads
  • record capture-local readiness dependencies in gtp_cuda_graphs.py and finish them before replaying each local forward CUDA graph
  • keep the existing DDP pre-forward gather policy shared by both paths

Root cause

GTP can prefetch the next module's local weight shard before that module executes. With overlapped DDP parameter gathering, the ordinary module pre-forward hook therefore runs too late: GTP may read the distributed-optimizer parameter buffer while its bucket all-gather is still in flight. CUDA graph capture also suppresses the DDP hook, so the dependency must be retained and replayed explicitly.

Each forward graph capture now creates its own GTPCaptureCommState. The transient active state is used only while capturing; the yielded state is stored on that graph's _CudaGraphRunner, preventing handle sets from aliasing across graph instances.

Signed-off-by: Jiangfei Duan <jiangfeid@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@JF-D
JF-D marked this pull request as ready for review August 4, 2026 00:15
@JF-D
JF-D requested review from a team as code owners August 4, 2026 00:15
@JF-D JF-D self-assigned this Aug 4, 2026
@JF-D

JF-D commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

@claude strict-review

@JF-D JF-D added the nemotron label Aug 4, 2026
@JF-D

JF-D commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 0f18dfe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant