feat(ascend): add MiniMax-H3 fused RoPE and MindIE SD compile backend - #1471
Open
blian6 wants to merge 1 commit into
Open
feat(ascend): add MiniMax-H3 fused RoPE and MindIE SD compile backend#1471blian6 wants to merge 1 commit into
blian6 wants to merge 1 commit into
Conversation
## Summary
Enable an optional MindIE-SD compile backend and a fused partial
split-half RoPE for MiniMax-H3 on Ascend NPU (USP4). On a 5s t2av 768p
run this measures -12%~-15% wall-clock vs the torch-native baseline
(rank0): rms/rope runtime fusion plus MindIE compile-side adaln/swiglu/
gate fusion, and Ulysses A2A kept on the eager path.
## Changes
- Add `minimax_h3_npu_rope`: fuses the 96/128 rotate-half rotary part
through MindIE-SD `rotary_position_embedding` (input guards live in
mindiesd); falls back to `TorchRealRope` when mindiesd is absent
- `BaseTransformerInfer`: support `compile_backend` ("default"|"mindie")
and `compile_dynamic` config keys; reuse ONE MindieSDBackend instance
to avoid BACKEND_MATCH recompilation and silent eager fallback
- Keep `TorchUlyssesA2A.exchange` out of the compiled graph
(`torch._dynamo.disable`) so HCCL keeps the fast fixed-size alltoall.
This changes compiled-graph behavior for all Ulysses + compile users
across platforms (the collective now runs eager)
- Add ascend_npu configs: `minimax_h3_t2av_sp_compile_5s.json`,
`minimax_h3_t2av_sp_compile_15s.json`
## Usage
mindiesd is an optional out-of-tree dependency. Without it, both the
compile backend and the fused rope degrade gracefully to the default
torch.compile / TorchRealRope paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enable an optional MindIE-SD compile backend and a fused partial split-half RoPE for MiniMax-H3 on Ascend NPU (USP4). The MindIE compile fuses the remaining DiTBlock ops (adaln / swiglu / residual-gate) on top of the runtime rms/rope fusion, and keeping the Ulysses A2A on the eager path removes an HCCL alltoallv regression. Kernel-level (CANN profiler, rank0): kernel count -15.3%, kernel time -9.9%, communication -35%.
Performance (kernel-level)
Measured with CANN profiler (warmup 5 + 1 profiled step, rank0, 768p t2av, USP4, 4x Ascend950PR). Compile vs the runtime-fused config (
npu_rms_norm+minimax_h3_npu_rope, no compile):The DiTBlock op chains are replaced by three fused kernels, and the Ulysses A2A regression is removed:
swiglu50x32ms,gather_scale_shift200x19ms,gather_residual_gate100x11mshcom_alltoallveliminated: tracingdist.all_to_all_singleinferssplit_sizes=[1,1,...]and degrades HCCL to the variable-length alltoallv path (200-270ms/call); with the collective kept on the eager path it stays on the fixed-size alltoall and overlaps better with the compiled compute (-35% communication)15s:
swiglu93.7ms +gather_scale_shift55.5ms +gather_residual_gate32.3ms replace the 99.4ms old Silu/split/Add/Mul chain; kernel count -550 (-15.3%), kernel time -333ms (-0.7%). Communication dominates the 15s kernel time (57.6%, 28002ms/48584ms) — a Ulysses-level bottleneck that grows linearly with sequence length and is outside compile's reach.Known residual: the gate-msa residual (
residual + gate * attn_out) spans a graph-break boundary (itsindex_selectlands in subgraph 1, the attention output in subgraph 6) and cannot be fused by a single pattern; a dedicated 2D residual-gate kernel was prototyped and reverted (its contiguous-copy and launch overhead outweighed the ~15ms/step fusion saving). All other DiTBlock ops are fused.Changes
minimax_h3_npu_rope(Ascend): fuses the 96/128 rotate-half rotary part through MindIE-SDrotary_position_embedding(input guards live in mindiesd); falls back toTorchRealRopewhen mindiesd is absentBaseTransformerInfer: supportcompile_backend("default" | "mindie") andcompile_dynamicconfig keys; reuse ONE MindieSDBackend instance to avoid BACKEND_MATCH recompilation and silent eager fallbackTorchUlyssesA2A.exchangeout of the compiled graph (torch._dynamo.disable) so HCCL keeps the fast fixed-size alltoall. This changes compiled-graph behavior for all Ulysses + compile users across platforms (the collective now runs eager)minimax_h3_t2av_sp_compile_5s.json,minimax_h3_t2av_sp_compile_15s.jsonUsage
mindiesdis an optional out-of-tree dependency. Without it, both the compile backend and the fused rope degrade gracefully to the defaulttorch.compile/TorchRealRopepaths. Enable the feature in the model config:Verification
TorchRealRopebitwiseruff checkandruff format --checkpass on all changed files