Process abort (`cudaErrorIllegalAddress`) on consumer dual-GPU when P2P is unavailable and `cudaMallocAsync` is the active allocator

[TaoGunner]

Summary

On a 2x RTX 3090 system, GeForce cards have PCIe P2P disabled at driver level, so can_access_peer correctly reports False and the CPU-staging path engages. That path then produces CUDA error: an illegal memory access was encountered.

Critically, the error surfaces from a C++ destructor, so it is not catchable as a Python exception — it calls abort() and kills the whole ComfyUI process:

terminate called after throwing an instance of 'c10::AcceleratorError'
  what(): CUDA error: an illegal memory access was encountered
Exception raised from free_impl at /__w/pytorch/pytorch/c10/cuda/CUDAMallocAsyncAllocator.cpp:206
Fatal Python error: Aborted

ComfyUI runs with the cudaMallocAsync backend (visible at startup: Device: cuda:0 NVIDIA GeForce RTX 3090 : cudaMallocAsync). That allocator uses per-device memory pools and is far stricter about foreign pointers than the native caching allocator — I suspect the CPU-staged tensor ends up freed against the wrong device's pool.

Two distinct reproductions

Both on MiniMax H3 (ComfyUI 0.30.0 native nodes), text encoder = Qwen3-VL 32B int8_convrot.

A. Quantized text encoder on cuda:1 — fails at load

CLIPLoaderMultiGPU (and CLIPLoaderDisTorch2MultiGPU, identical result), device cuda:1:

[MultiGPU P2P] can_access_peer(1, 0) = False
...
comfy_kitchen/tensor/int8.py:172 in dequantize
  torch.ops.comfy_kitchen.dequantize_int8_convrot_weight_dtype(...)
comfy_kitchen/backends/cuda/__init__.py:1531 in dequantize_int8_convrot_weight_dtype
  _wrap_for_dlpack(q_2d)
→ torch.AcceleratorError: CUDA error: an illegal memory access was encountered

B. fp16 video VAE on cuda:1 — fails during decode

VAELoaderMultiGPU, device cuda:1. Sampling completes fine (20 steps, 3.82 s/it, diffusion model on cuda:0). CPU-staging logs look healthy:

[MultiGPU P2P] can_access_peer(1, 0) = False
[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)
[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))

…repeated several times, then hard abort inside the VAE's attention block:

Stack (most recent call first):
  File "/root/ComfyUI/comfy/ops.py", line 93 in scaled_dot_product_attention
  File "/root/ComfyUI/comfy/ldm/modules/attention.py", line 526 in attention_pytorch
  File "/root/ComfyUI/comfy/ldm/minimax/vae.py", line 239 in forward
  File "/root/ComfyUI/comfy/ldm/minimax/vae.py", line 499 in tiled_decode
  File "/root/ComfyUI/comfy/ldm/minimax/vae.py", line 690 in decode

Frame #8 in the C++ trace is at::native::_flash_attention_forward, so a staged tensor reaches flash-attention and dies on free.

Since B involves no quantization at all, the trigger is cross-device pointers in general, not the int8_convrot path specifically.

Working configuration

Everything on cuda:0 with stock loaders works. ComfyUI's built-in DynamicVRAM streams the 26 GB text encoder through host RAM without issue (Model MiniMaxH3TEModel_ prepared for dynamic VRAM loading. 25882MB Staged). So the second GPU is currently unusable on this hardware.

Questions / suggestions

1. Is the CPU-staging path tested against the cudaMallocAsync allocator? A note in the README about PYTORCH_CUDA_ALLOC_CONF=backend:native would help if it is a known incompatibility.
2. Should the staged tensor be materialized as a real copy owned by the destination device's pool, rather than wrapped via DLPack across the boundary?
3. Consider refusing to place models on a non-default device when can_access_peer is False and the allocator is cudaMallocAsync — a clear error at load time beats a process abort mid-decode.

Happy to test patches; the setup reproduces both cases reliably in under two minutes.

Environment

• ComfyUI 0.30.0 (rev 5693, b1693ecb)
• PyTorch 2.13.0+cu130, Python 3.13.14
• comfy-kitchen 0.2.26, comfy-aimdo 0.4.11
• 2x NVIDIA RTX 3090 (no NVLink bridge), PCIe P2P disabled by driver
• Allocator: cudaMallocAsync
• Launch args: --listen --port 8188 --reserve-vram 1.0 --disable-smart-memory
• Docker, image yanwk/comfyui-boot:cu130-slim
• Note: issue #<номер первого> had to be worked around first to reach this point

[andypotato]

This issue is still present in the current version. I'm encountering the exact same error on a Dual 5060TI configuration.