Support intranode DeepEP without IMEX fabric#7
Draft
rchalamala wants to merge 59 commits into
Draft
Conversation
Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Retain the validated local-shard pre-copy for recognized rank-major and concatenated LOAD gather layouts, but pass any unrecognized output layout through to the original collective unchanged instead of raising, preserving the delegate's return value and exceptions. Invalid rank bounds still fail fast. Log which optional gather seams were actually installed. Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Reject corrupt VMM2 IPC chunk descriptors in cuIpcOpenMemHandle before deriving the interior pointer: chunk_base/chunk_size must be nonzero together, and the requested carve must be fully contained within the advertised chunk. All range arithmetic is overflow-safe because the descriptor is untrusted. Invalid metadata returns CUDA_ERROR_INVALID_VALUE before any fd fetch, import, reservation, or mapping, so existing cleanup paths are unaffected. Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Add set -euo pipefail to both experimental serve scripts (guarding the optional second argument to preserve defaults), remove the duplicate VLLM_USE_V2_MODEL_RUNNER export, pin --seed 42 / --random-seed 42, and set the validated SGLang static memory fraction default to 0.65. Document the 0.65 rationale in the recipe README. Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Explain the 8-byte legacy and current 16-byte dependency record layouts, how current readers accept both by stored section size, and why FORMAT_VERSION stays at 1. Note that compatibility is one-directional: legacy readers cannot parse the 16-byte edge-metadata records this writer emits. Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
The malformed-chunk regression's in-process positive control opened a self-exported handle, which fails at cuMemSetAccess (CUDA IPC is inter-process; a second same-process mapping of a self-imported handle returns CUDA_ERROR_INVALID_VALUE) independent of the new range validation. Drop that unsupported same-process open and keep the six invalid-metadata rejection assertions, which are the regression's purpose. Valid cross-process chunk import plus interior-pointer derivation stays covered by the two-H100 nvl_ipc_prealloc DeepEP matrix case. Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
…findings Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
The SGLang experimental serve script now threads --deepep-mode through a DEEPEP_MODE env override (default low_latency) instead of hardcoding low_latency. DEEPEP_MODE=auto is the supported nonzero-num_nvl_bytes mode that keeps low-latency decode graphs Foundry captures (verified against pinned sglang@4cfcc25: DeepEPMode.AUTO enables both normal and low_latency so get_deepep_buffer computes a nonzero NVL buffer, and server_args only sets disable_cuda_graph for deepep_mode=normal, not auto). README documents the override and the normal-mode limitation. Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Add the LOAD-only logits pre-copy patch to the install-order list and a dedicated section describing both wrapped seams (attention-TP and regular TP gather), renumbering the CudaGraphRunner and spawn-site sections and fixing the internal cross-references accordingly. Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Rewrite _verify_dispatch_result to decode every valid received token (bounded by the per-expert recv_count DeepEP returns, since padding slots are not zeroed) and assert the decoded (src_rank, token_id) is in range and that this rank's global expert id is in the token's deterministic top-k selection. Route the LOAD post-replay check through the same helper. The token payload still encodes rank*1000+token here, which is not exactly representable in bfloat16 for rank>0 (>=1000 rounds to a multiple of 4/8), so the new assertions expose the rounding: the decoded token id no longer matches the expert it landed on. This is the RED baseline for the encoding fix. Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Switch the token payload stride/offset so every rank*token id maps to an integer in [1, 128], which bfloat16 represents exactly (integers <= 256). The previous rank*1000 encoding rounded rank>0 payloads to nearby multiples of 4/8, which the new routing assertions correctly reject. With the exact encoding the decoded (src_rank, token_id) round-trips, and the +1 offset keeps real tokens non-zero so they are distinguishable from DeepEP's non-zeroed padding slots. Makes the standalone dispatch matrix pass with the assertions in place. Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Add a 2-GPU cross-process regression where rank 1 launches two importer threads that contend on the SAME peer chunk's first open. Both miss the mapping cache and import a temporary mapping; the winner establishes a narrow self-consistent view while the insertion-race loser claims the full chunk and carves past the winner's mapping. The loser must be rejected against the actual cached mapping (CUDA_ERROR_INVALID_VALUE) rather than returning an out-of-mapping interior pointer, and a subsequent valid open/close must still work. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Rework the concurrent regression: two importer threads contend on the same peer chunk's first open so one lands on the insertion-race-loser path and must adopt the winner's whole-chunk mapping (same interior). Because the driver cannot map a sub-range of an imported handle, an inconsistent handle cannot itself reach the loser path, so inconsistent same-key geometry is presented against the live cached mapping (a cache hit) and must be rejected with CUDA_ERROR_INVALID_VALUE and no out-of-mapping pointer, sharing the validator the loser path also uses as defense-in-depth. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
The mapping cache key omits chunk_size, so a same-key handle can carry a different (untrusted) chunk_size and carve than the live mapping. The cache-hit fast path already re-validated the request against the actual cached mapping, but the insertion-race-loser path adopted the winner's mapping without the same check. Factor a single vmm_ipc_chunk_request_matches_mapping() validator (chunk-size equality + overflow-safe carve containment against VmmIpcChunkMapping.size) and call it in BOTH paths before any refcount bump, subpointer registration, or pointer return. The loser path releases its losing temporary mapping/handle first and only then re-validates, so all cleanup and lock/refcount behavior is preserved on the reject path. The driver cannot map a sub-range of an imported handle, so in practice every handle reaching the loser path has mapped the full chunk and is already consistent; the loser-path check is defense-in-depth that keeps the two adopt paths symmetric. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
Strengthen _verify_dispatch_result beyond routing-only: the deterministic
top-k selection depends only on the token id, so each local expert must
receive exactly {(src_rank, token) : rank in ranks, token selects expert},
once each. Assert recv_count[e] equals the expected delivery count and that
the decoded (src_rank, token_id) multiset equals the expected set with no
duplicate, missing, or spurious deliveries.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Rahul Chalamala <rchalamala@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
deepep_transport = "fabric" | "nvl_ipc"configuration for vLLM and SGLang, preserving fabric as the defaultValidation
Validated on Modal
rahul-devwith CUDA 13.0, PyTorch 2.11/cu130, and H100 GPUs:ll_nofabric,nvl_ipc, andnvl_ipc_preallocall passed with six successful graph replays and exact dispatch deliveryFinal whole-branch review found no remaining Critical or Important code issues.
Administrative follow-up
Seven early commits predate
git commit -sin this workspace and still need DCO trailers. Rewriting those already-pushed commits requires explicit approval before force-pushing.