fix(legacy): order work after ready-index publication

[youchengsong]

Summary

Restore the post-publication ordering point used by NVSHMEM after
ibgda_submit_requests advances ready_idx.

DeepEP already uses __threadfence() before the CAS to order WQE payload
writes before ready-index publication. This change adds the separate fence
after the successful CAS so subsequent post-send work remains ordered after
that publication.

The added memory_fence_cta() emits the same fence.acq_rel.cta
instruction used by NVSHMEM's IBGDA_MFENCE() at the corresponding point:

https://github.com/NVIDIA/nvshmem/blob/v3.7.2-0/src/include/non_abi/device/pt-to-pt/ibgda_device.cuh#L1648-L1670

Scope

The functional change is one fence in the legacy IBGDA submission helper.
It does not change:

• WQE construction or the existing pre-publication system fence;
• ready-index selection or CAS behavior;
• synchronous versus asynchronous post-send handling;
• CQ polling, QP selection, batching, or public APIs.

This PR is protocol-ordering hardening and does not claim that this omitted
fence alone explains every observed legacy progress failure.

Validation

• full build of this commit for sm_103 with CUDA 13.0.88,
  NVSHMEM 3.7.2, NCCL 2.30.7, and PyTorch 2.11;
• generated extension contains an sm_103 cubin;
• bash ./format.sh;
• git diff --check upstream/main...HEAD;
• verified that the helper's memory_fence_cta() is
  fence.acq_rel.cta, matching NVSHMEM's ordering primitive.

The change also merges cleanly with the current head of #677; the two
patches address independent parts of the submission path.

[ds-review-bot]

🔵 suggestion: Consider adding a brief note in the comment that this new fence pairs with the pre-CAS __threadfence() to bracket the ready-index publication (acquire/release semantics). This makes the ordering contract explicit for future readers and clarifies why a CTA-scope fence is used here versus the system-scope fence above.

_{🤖 v3}

[ds-review-bot]

🤖 ds-review-bot Code Review

v6

The added CTA fence appropriately prevents subsequent post-send work from being reordered before ready-index publication. No actionable regressions were found.

v4

⚠️ 未完成评审（no_result_file：模型未产出结果文件）

v3

This PR adds a single memory_fence_cta() (emitting fence.acq_rel.cta) immediately after the successful CAS that advances ready_idx in ibgda_submit_requests (csrc/kernels/legacy/ibgda_device.cuh). The intent is to restore the post-publication ordering point used by NVSHMEM's IBGDA_MFENCE(), ensuring subsequent post-send work stays ordered after the ready-index publication. The change is minimal, well-scoped, and matches the described intent.

Correctness assessment (verified):

• The pre-publication __threadfence() (system-scope) still orders WQE payload writes before the CAS, unchanged.
• The added memory_fence_cta() is placed correctly: after the CAS spin-loop and before the post-send branch (line 161).
• memory_fence_cta() is confirmed to emit fence.acq_rel.cta (csrc/kernels/legacy/utils.cuh:77-78), matching NVSHMEM's ordering primitive referenced in the PR.
• The fence usage is consistent with other uses in the same file (lock acquire/release, doorbell paths).
• Note the intentional scope asymmetry: the pre-publication fence is __threadfence() (device/system scope) while the new post-publication fence is CTA-scope; this matches the stated NVSHMEM reference and DeepEP's existing patterns, and given ibgda_post_send/doorbell writes have their own __threadfence/relaxed-store ordering, CTA scope appears adequate.

The change is low-risk protocol-ordering hardening with no API, batching, CQ polling, or WQE-construction changes. It looks correct and ready to merge.

Files reviewed: 1
Issues found: 🔵 1 suggestion
Inline comments posted: 1

⚠️ Parse warning: [v4] no_result_file：模型未产出结果文件