Failes on 1xRTX4090 and 1xRTX6000 Ada

[wlarsong]

NVIDIA Open GPU Kernel Modules Version

610.57.04-p2p

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Description: Ubuntu 26.04 LTS

Kernel Release

Linux definitely-not-a-banana 7.0.0-29-generic #29-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 17 20:52:35 UTC 2026 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA RTX 6000 Ada Generation (UUID: GPU-00a1e30c-c092-d1da-65a8-5732c0b37891) GPU 1: NVIDIA GeForce RTX 4090 (UUID: GPU-8c42a17f-068b-943a-329e-e30d56449953)

Describe the bug

Driver is installed Libcuda is modified simplep2p passes:

./cuda-samples/build/cpp/0_Introduction/simpleP2P/simpleP2P
[./cuda-samples/build/cpp/0_Introduction/simpleP2P/simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...

Peer access from NVIDIA RTX 6000 Ada Generation (GPU0) -> NVIDIA GeForce RTX 4090 (GPU1) : Yes
Peer access from NVIDIA GeForce RTX 4090 (GPU1) -> NVIDIA RTX 6000 Ada Generation (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
CUDA error at /home/will/cuda-samples/cpp/0_Introduction/simpleP2P/simpleP2P.cu:130 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)"

p2pLatenencyTest Fails

will@definitely-not-a-banana:~$ ./cuda-samples/build/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX 6000 Ada Generation, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: 4b, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 806.66 21.64
1 21.77 950.43
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
Cuda failure /home/will/cuda-samples/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu:280: 'mapping of buffer object failed'

Thu Aug 20 00:51:43 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.57.04 KMD Version: 610.57.04 CUDA UMD Version: 13.3 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX 6000 Ada Gene... Off | 00000000:01:00.0 Off | Off |
| 30% 44C P0 66W / 300W | 0MiB / 49140MiB | 2% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 Off | 00000000:4B:00.0 Off | Off |
| 30% 37C P0 69W / 450W | 0MiB / 49140MiB | 2% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

will@definitely-not-a-banana:~$ cat /etc/modprobe.d/nvidia-gdr.conf
options nvidia NVreg_ExperimentalDmaBufP2P=1

To Reproduce

Install driver for 610 per directions.

Install the modified version of cuda via the python script.

Add the : /etc/modprobe.d/nvidia-gdr.conf
options nvidia NVreg_ExperimentalDmaBufP2P=1

Run p2pLatenecyTest

Bug Incidence

Once

nvidia-bug-report.log.gz

===

More Info

===

[wlarsong]

will@definitely-not-a-banana:$ sudo dmesg | grep "amd_"
[sudo: authenticate] Password:
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-7.0.0-29-generic root=UUID=5c38a63a-66b3-43d8-a8e2-9dde48e5ab5a ro amd_iommu=on iommu=pt pci=realloc pcie_acs_override=downstream,multifunction crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-7.0.0-29-generic root=UUID=5c38a63a-66b3-43d8-a8e2-9dde48e5ab5a ro amd_iommu=on iommu=pt pci=realloc pcie_acs_override=downstream,multifunction crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
[ 0.577075] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 0.577099] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[ 0.577122] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[ 0.577146] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[ 7.603877] amd_atl: AMD Address Translation Library initialized
will@definitely-not-a-banana:$ nvidia-smi -q | grep -A 4 "BAR1 Memory Usage"
BAR1 Memory Usage
Total : 65536 MiB
Used : 5 MiB
Free : 65531 MiB
Conf Compute Protected Memory Usage

BAR1 Memory Usage
    Total                                          : 32768 MiB
    Used                                           : 1 MiB
    Free                                           : 32767 MiB
Conf Compute Protected Memory Usage

will@definitely-not-a-banana:~$ ./cuda-samples/build/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX 6000 Ada Generation, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: 21, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 798.82 21.78
1 21.89 950.43
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
Cuda failure /home/will/cuda-samples/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu:280: 'mapping of buffer object failed'

[aikitoria]

please try this version https://github.com/aikitoria/open-gpu-kernel-modules/tree/610.57.04-p2p-v2

[wlarsong]

Doesn't work:
will@definitley-not-a-banana:~/cuda-samples/cpp/0_Introduction/simpleP2P$ nvidia-smi
Mon Aug 24 17:56:38 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.57.04 KMD Version: 610.57.04 CUDA UMD Version: 13.3 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX 6000 Ada Gene... Off | 00000000:01:00.0 Off | Off |
| 30% 39C P8 29W / 300W | 18MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 Off | 00000000:4B:00.0 Off | Off |
| 30% 35C P8 20W / 450W | 1MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
will@definitley-not-a-banana:~/cuda-samples/cpp/0_Introduction/simpleP2P$ ./simpleP2P
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...

Peer access from NVIDIA RTX 6000 Ada Generation (GPU0) -> NVIDIA GeForce RTX 4090 (GPU1) : Yes
Peer access from NVIDIA GeForce RTX 4090 (GPU1) -> NVIDIA RTX 6000 Ada Generation (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
CUDA error at simpleP2P.cu:130 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)"
will@definitley-not-a-banana:/cuda-samples/cpp/0_Introduction/simpleP2P$ cd nccl-tests/
-bash: cd: nccl-tests/: No such file or directory
will@definitley-not-a-banana:/cuda-samples/cpp/0_Introduction/simpleP2P$ CUDA_VISIBLE_DEVICES=0,1 NCCL_CUMEM_ENABLE=1 NCCL_DEBUG=INFO NCCL_P2P_LEVEL=SYS ./build/sendrecv_perf -b 8M -e 1G -f 2 -g 1
-bash: ./build/sendrecv_perf: No such file or directory
will@definitley-not-a-banana:/cuda-samples/cpp/0_Introduction/simpleP2P$ cd $HOME/nccl-tests/
will@definitley-not-a-banana:/nccl-tests$ CUDA_VISIBLE_DEVICES=0,1 NCCL_CUMEM_ENABLE=1 NCCL_DEBUG=INFO NCCL_P2P_LEVEL=SYS ./build/sendrecv_perf -b 8M -e 1G -f 2 -g 1

nccl-tests version 2.19.7 nccl-headers=23102 nccl-library=23102

Collective test starting: sendrecv_perf

nThread 1 nGpus 1 minBytes 8388608 maxBytes 1073741824 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0

Using devices

Rank 0 Group 0 Pid 2274 on definitley-not-a-banana device 0 [0000:01:00] NVIDIA RTX 6000 Ada Generation

definitley-not-a-banana:2274:2274 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
definitley-not-a-banana:2274:2274 [0] NCCL INFO Bootstrap: Using enp67s0:192.168.157.54<0>
definitley-not-a-banana:2274:2282 [0] NCCL INFO Socket handshake magic 0x564ab9f2fc4b9d6c (built-in default)
definitley-not-a-banana:2274:2274 [0] NCCL INFO cudaDriverVersion 13030
definitley-not-a-banana:2274:2274 [0] NCCL INFO NCCL version 2.31.2+cuda13.3
definitley-not-a-banana:2274:2274 [0] NCCL INFO NCCL git version unknown unknown
definitley-not-a-banana:2274:2274 [0] NCCL INFO Comm config nvlinkCentricSched set to 1
definitley-not-a-banana:2274:2274 [0] NCCL INFO Comm config maxP2pPeers set to 2
definitley-not-a-banana:2274:2274 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
definitley-not-a-banana:2274:2274 [0] NCCL INFO NET/IB : No device found.
definitley-not-a-banana:2274:2274 [0] NCCL INFO NET/IB : Using [RO]; OOB enp67s0:192.168.157.54<0>
definitley-not-a-banana:2274:2274 [0] NCCL INFO Failed to initialize NET plugin IB
definitley-not-a-banana:2274:2274 [0] NCCL INFO NET/Socket : Using [0]enp67s0:192.168.157.54<0> [1]enp68s0:192.168.157.53<0>
definitley-not-a-banana:2274:2274 [0] NCCL INFO Initialized NET plugin Socket
definitley-not-a-banana:2274:2274 [0] NCCL INFO Assigned NET plugin Socket to comm
definitley-not-a-banana:2274:2274 [0] NCCL INFO RMA/Plugin: Could not find: libnccl-rma.so
definitley-not-a-banana:2274:2274 [0] NCCL INFO RMA/Plugin: Failed to initialize any plugin
definitley-not-a-banana:2274:2274 [0] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
definitley-not-a-banana:2274:2274 [0] NCCL INFO GIN/Plugin: Failed to initialize any GIN plugin
definitley-not-a-banana:2274:2274 [0] NCCL INFO Using network Socket
definitley-not-a-banana:2274:2274 [0] NCCL INFO Loaded NVML from libnvidia-ml.so.1
definitley-not-a-banana:2274:2274 [0] NCCL INFO [Rank 0] ncclCommInitRankConfig comm 0x5c1a4938b050 rank 0 nranks 1 cudaDev 0 nvmlDev 0 busId 1000 commId 0x7901e3bd7c6be1fe - Init START
definitley-not-a-banana:2274:2274 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
definitley-not-a-banana:2274:2274 [0] NCCL INFO Bootstrap timings total 0.000522 (create 0.000035, send 0.000106, recv 0.000131, ring 0.000001, delay 0.000001)
definitley-not-a-banana:2274:2274 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 1.
definitley-not-a-banana:2274:2274 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to SYS
definitley-not-a-banana:2274:2274 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is empty, ignoring. (GPU affinity = ; CPU affinity = 0-63).
definitley-not-a-banana:2274:2274 [0] NCCL INFO Max P2P Peers 2 is too high, capping to communicator size 1
definitley-not-a-banana:2274:2274 [0] NCCL INFO Rank 0: 2 Net devices
definitley-not-a-banana:2274:2274 [0] NCCL INFO Rank 0: 0 CollNet devices
definitley-not-a-banana:2274:2274 [0] NCCL INFO Local Net device counts across ranks: min 2 max 2
definitley-not-a-banana:2274:2274 [0] NCCL INFO Local CollNet device counts across ranks: min 0 max 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO comm 0x5c1a4938b050 rank 0 nRanks 1 nNodes 1 localRanks 1 localRank 0 MNNVL 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 00/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 01/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 02/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 03/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 04/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 05/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 06/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 07/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 08/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 09/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 10/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 11/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 12/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 13/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 14/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 15/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 16/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 17/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 18/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 19/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 20/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 21/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 22/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 23/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 24/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 25/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 26/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 27/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 28/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 29/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 30/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 31/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 32/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 33/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 34/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 35/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 36/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 37/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 38/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 39/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 40/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 41/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 42/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 43/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 44/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 45/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 46/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 47/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 48/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 49/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 50/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 51/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 52/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 53/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 54/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 55/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 56/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 57/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 58/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 59/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 60/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 61/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 62/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Channel 63/64 : 0
definitley-not-a-banana:2274:2274 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 [32] -1/-1/-1->0->-1 [33] -1/-1/-1->0->-1 [34] -1/-1/-1->0->-1 [35] -1/-1/-1->0->-1 [36] -1/-1/-1->0->-1 [37] -1/-1/-1->0->-1 [38] -1/-1/-1->0->-1 [39] -1/-1/-1->0->-1 [40] -1/-1/-1->0->-1 [41] -1/-1/-1->0->-1 [42] -1/-1/-1->0->-1 [43] -1/-1/-1->0->-1 [44] -1/-1/-1->0->-1 [45] -1/-1/-1->0->-1 [46] -1/-1/-1->0->-1 [47] -1/-1/-1->0->-1 [48] -1/-1/-1->0->-1 [4
definitley-not-a-banana:2274:2274 [0] NCCL INFO P2P Chunksize set to 524288
definitley-not-a-banana:2274:2274 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
definitley-not-a-banana:2274:2274 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
definitley-not-a-banana:2274:2287 [0] NCCL INFO [Proxy Service] Device 0 CPU core 44
definitley-not-a-banana:2274:2287 [0] NCCL INFO proxy listening socket at 192.168.157.54<57203>
definitley-not-a-banana:2274:2288 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 10
definitley-not-a-banana:2274:2274 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
definitley-not-a-banana:2274:2274 [0] NCCL INFO threadThresholds 8/8/64 | 8/8/64 | 512 | 512
definitley-not-a-banana:2274:2274 [0] NCCL INFO 64 coll channels, 64 collnet channels, 0 nvls channels, 64 p2p channels, 64 p2p channels per peer
definitley-not-a-banana:2274:2274 [0] NCCL INFO symmetricSupport 1, cuMemEnable 1, globalGinSupport 0, cuMemGdrSupport 1, contiguousRanksPerHost 1, crossNicSupport 1
definitley-not-a-banana:2274:2274 [0] NCCL INFO CC Off, workFifoBytes 1048576
definitley-not-a-banana:2274:2274 [0] NCCL INFO ncclCommInitRankConfig comm 0x5c1a4938b050 rank 0 nranks 1 cudaDev 0 nvmlDev 0 busId 1000 commId 0x7901e3bd7c6be1fe - Init COMPLETE
definitley-not-a-banana:2274:2274 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 1 total 0.23 (kernels 0.19, alloc 0.03, bootstrap 0.00, allgathers 0.00, topo 0.00, graphs 0.00, connections 0.01, rest 0.00)

[2026-08-24 17:57:25] definitley-not-a-banana:2274:2274 [0] allocator.cc:89 (ncclMemAlloc) NCCL WARN Cuda failure 205 'mapping of buffer object failed'
definitley-not-a-banana: Test NCCL failure common.cu:1107 'unhandled cuda error (run with NCCL_DEBUG=INFO for details) / '
.. definitley-not-a-banana pid 2274: Test failure common.cu:1636
.. definitley-not-a-banana pid 2274: Test failure common.cu:1454
(failed reverse-i-search)`bar': cat .^Cshrc
will@definitley-not-a-banana:~/nccl-tests$ nvidia-smi -q | grep -A 4 "BAR1 Memory Usage"
BAR1 Memory Usage
Total : 65536 MiB
Used : 17 MiB
Free : 65519 MiB
Conf Compute Protected Memory Usage

BAR1 Memory Usage
    Total                                          : 32768 MiB
    Used                                           : 1 MiB
    Free                                           : 32767 MiB
Conf Compute Protected Memory Usage

will@definitley-not-a-banana:/nccl-tests$ cd ..
will@definitley-not-a-banana:$ cd
610/ .config/ cuda-samples/ .local/ .nv/ .ssh/ .triton/
.cache/ cuda-auto-patch/ .humming/ nccl-tests/ open-gpu-kernel-modules/ .tilelang/ vllm-env/
will@definitley-not-a-banana:$ cd cuda-samples/
will@definitley-not-a-banana:/cuda-samples$ cd ./cpp/5_Domain_Specific/p2pBandwidthLatencyTest/
will@definitley-not-a-banana:~/cuda-samples/cpp/5_Domain_Specific/p2pBandwidthLatencyTest$ ./p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX 6000 Ada Generation, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: 4b, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 805.41 21.68
1 22.15 950.43
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
Cuda failure p2pBandwidthLatencyTest.cu:280: 'mapping of buffer object failed'