Enabling P2P reduces bandwidth on Arrow Lake / Z890?

[petertiedemann]

On my systems equipped with 4x 5060TI 16gb GPUs, i noticed that p2pBandwidthLatencyTest indicates that P2P bandwidth is lower than non-P2P. Is that expected, and if so, is it specific to the specific CPU, or determined by chipset etc?

In case it matters, performance is identical with iommu off completely as well as with intel_iommu=on iommu=pt.

Here is an example:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5060 Ti, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5060 Ti, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5060 Ti, pciBusID: 4, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5060 Ti, pciBusID: 5, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3
     0       1     1     1     1
     1       1     1     1     1
     2       1     1     1     1
     3       1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3
     0 470.29  12.24  12.13  11.80
     1  12.56 473.35  13.76  13.00
     2  12.49  14.25 473.05  12.94
     3  12.47  14.17  13.03 473.77
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3
     0 472.49   5.47   5.48   5.47
     1   5.47 475.08   5.48   5.47
     2   5.47   5.48 473.63   5.47
     3   5.48   5.47   5.48 473.49
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3
     0 470.56  15.23  15.35  15.39
     1  14.24 471.83  17.06  16.29
     2  14.19  19.06 471.26  16.33
     3  14.15  18.13  16.33 471.91
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3
     0 470.06  10.94  10.94  10.94
     1  10.94 472.12  10.94  10.95
     2  10.96  10.94 472.55  10.94
     3  10.94  10.92  10.95 471.83
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3
     0   1.49  14.72  14.81  14.72
     1  15.27   0.97  14.48  14.70
     2  14.77  14.41   0.97  14.40
     3  15.22  14.41  14.42   0.97

   CPU     0      1      2      3
     0   1.51   5.38   5.32   5.32
     1   5.23   1.50   5.28   5.25
     2   5.26   5.25   1.52   5.25
     3   5.17   5.24   5.26   1.52
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3
     0   0.96   0.49   0.49   0.41
     1   0.48   0.95   0.41   0.47
     2   0.41   0.41   0.97   0.40
     3   0.42   0.41   0.41   0.97

   CPU     0      1      2      3
     0   1.61   1.22   1.22   1.23
     1   1.33   1.51   1.24   1.24
     2   1.29   1.25   1.51   1.28
     3   1.30   1.27   1.27   1.53

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

## System Information Summary

┌────────────────┬─────────────────────────────────────────────────────────────────────────┐
│ Item           │ Detail                                                                  │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ CPU            │ Intel Core Ultra 5 235 (Arrow Lake-HX), 14 cores, 1 socket              │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Motherboard    │ MSI MEG Z890 UNIFY-X (MS-7E20), BIOS 1.A90                              │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Chipset        │ Intel Z890 (Arrow Lake-HX / 800 Series PCH)                             │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Kernel         │ 7.0.0-22-generic (gcc 15.2.0, Ubuntu)                                   │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ NVIDIA Driver  │ 610.43.02 (Open Kernel Module, source-built), CUDA 13.3                 │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Kernel cmdline │ intel_iommu=on iommu=pt  (also tried off entirely)                      │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ GPUs           │ 4x NVIDIA GeForce RTX 5060 Ti 16GB (GB206, rev a1, device ID 10de:2d04) │
└────────────────┴─────────────────────────────────────────────────────────────────────────┘

## PCIe Topology

CPU Host Bridge [8086:7d2a]
├── 00:01.0 [8086:7ecc] Meteor Lake-H PCIe Root Port #12  ── x4 @ 32GT/s
│   └── 01:00.0  GPU 0  ── x4 @ 32GT/s  (downgraded from x16 capable)
│
├── 00:06.0 [8086:ae4d] Arrow Lake-HX PCIe Root Port #13  ── x8 @ 32GT/s
│   └── 02:00.0  GPU 1  ── x8 @ 32GT/s  (downgraded from x16 capable)
│
├── 00:06.1 [8086:7eca] ── NVMe SSD (03:00.0)
│
├── 00:06.3 [8086:ae4e] Arrow Lake-HX Device             ── x4 @ 32GT/s
│   └── 04:00.0  GPU 2  ── x4 @ 32GT/s  (downgraded from x8 capable)
│
└── 00:06.4 [8086:ae4f] Arrow Lake-HX Device             ── x4 @ 32GT/s
    └── 05:00.0  GPU 3  ── x4 @ 32GT/s  (downgraded from x16 capable)

[badnikhil]

@petertiedemann Short answer: yes, that's expected on this kind of platform and it's not a bug.

With P2P off, cudaMemcpyPeer stages the copy through host RAM (GPU→RAM→GPU), which is a well-optimized, posted-write DMA path. With P2P on, the copy goes GPU→GPU directly, but on a consumer Z890 board that traffic has to route up through the CPU root complex (and across the DMI link to the chipset for the PCH-attached cards). Client root complexes are poor P2P forwarders NVIDIA's own GPUDirect docs note peer-to-peer read bandwidth is "severely limited" through the CPU. Add your reduced link widths (x4/x8) and the shared DMI uplink, and direct P2P ends up slower than host staging for bulk transfers. The latency improvement you saw is the real P2P benefit on this hardware; bandwidth isn't.

So: for throughput, you're better off leaving P2P disabled here. Good P2P bandwidth needs full x16 CPU-direct lanes (Threadripper Pro / EPYC) or a board with a PCIe switch.

[mratsim]

X870e boards have decent P2P perf from reports online, even in x8/x8. The issue seems to be Intel or at least Arrow Lake specific.

[ikirsh]

I am experiencing the same issue with a dual RTX 5060 Ti 16GB setup on an MSI PRO Z890P motherboard paired with an Intel Core Ultra 265K (Arrow Lake) processor.

Below are the results from the connectivity and bandwidth matrix:

P2P Connectivity Matrix
     D\D      0      1
     0        1      1
     1        1      1

Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D      0      1 
     0 385.94  13.36 
     1  14.22 388.49 

Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D      0      1 
     0 354.63   5.47 
     1   5.47 388.30 

Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D      0      1 
     0 368.82  16.48 
     1  18.23 387.95 

Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D      0      1 
     0 369.13  10.92 
     1  10.94 387.09 

P2P=Disabled Latency Matrix (us)
   GPU      0      1 
     0   1.01  14.36 
     1  14.32   1.04 

   CPU      0      1 
     0   1.50   5.31 
     1   5.32   1.53 

P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU      0      1 
     0   0.99   0.47 
     1   0.49   1.04 

   CPU      0      1 
     0   1.54   1.25 
     1   1.27   1.52 

Given these metrics, I have two questions:

1. Does AMD offer better P2P bandwidth performance than Intel in this context?
2. Are there any cost-effective CPU and motherboard combinations that avoid these P2P bandwidth bottlenecks?

[Nero10578]

How did you even enable P2P on Z890? Disabling IOMMU and setting intel_iommu=on and iommu=pt I can't even get P2P to work with my RTX A6000 GPUs on an Asus Z890 Apex.

[ikirsh]

I no longer have that setup available to verify, but if I recall correctly, in the end it had intel_iommu=off (following AI advice), though I believe it performed identically with intel_iommu=on iommu=pt.

I'd suggest opening a new issue with your setup details, as that seems to be a distinct problem. However, even once you resolve it, you will likely still encounter the limited bandwidth bottleneck, which currently appears to be a hardware limitation.

I believe I now have answers to my two original questions above. This really does seem to be an Intel Arrow Lake platform issue.

I am now running 4 x RTX 5060 Ti 16GB cards on a much lower-cost AMD setup:

• Motherboard: ASUS PRIME B850-PLUS WIFI
• CPU: AMD Ryzen 5 7600X
• Same passive AliExpress adapters/risers, allowing PCIe 5.0 x4 to each card.

Here are my updated p2pBandwidthLatencyTest results on the AMD system:

`[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5060 Ti, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5060 Ti, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5060 Ti, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5060 Ti, pciBusID: 4, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3
     0       1     1     1     1
     1       1     1     1     1
     2       1     1     1     1
     3       1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 386.21  13.29  13.29  13.29 
     1  13.28 389.27  13.30  13.31 
     2  13.30  13.30 388.59  13.30 
     3  13.29  13.29  13.31 388.97 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3 
     0 386.76  14.29  14.30  14.30 
     1  14.30 389.36  14.30  14.30 
     2  14.30  14.29 388.88  14.30 
     3  14.30  14.29  14.30 388.68 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 386.03  16.32  16.30  16.25 
     1  16.28 386.90  16.27  16.22 
     2  16.27  16.22 387.23  16.31 
     3  16.33  16.37  16.21 387.28 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 386.44  27.88  27.89  27.89 
     1  27.88 386.66  27.88  27.88 
     2  27.90  27.87 387.49  27.89 
     3  27.89  27.88  27.89 386.61 
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3 
     0   0.99  14.57  14.39  14.57 
     1  14.46   0.98  14.48  14.66 
     2  14.42  14.62   1.00  14.75 
     3  14.71  14.60  14.49   1.00 

   CPU     0      1      2      3 
     0   1.62   4.75   4.74   4.75 
     1   4.80   1.49   4.51   4.48 
     2   4.87   4.57   1.51   4.50 
     3   4.87   4.59   4.53   1.50 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3 
     0   0.99   0.52   0.39   0.51 
     1   0.39   0.97   0.50   0.50 
     2   0.41   0.51   0.99   0.40 
     3   0.42   0.40   0.51   1.00 

   CPU     0      1      2      3 
     0   1.67   1.32   1.33   1.37 
     1   1.27   1.54   1.24   1.20 
     2   1.25   1.21   1.53   1.19 
     3   1.21   1.21   1.20   1.54 

Switching to a budget AMD CPU and motherboard (roughly half the price of the Arrow Lake platform) completely resolved the P2P throughput issue.

This Arrow Lake P2P throughput bottleneck may share a root cause with another reported issue:

Intel Arrow Lake Processors Bottleneck PCIe 5.0 NVMe SSDs (Tom's Hardware)

That report highlights two key findings:

1. M.2 PCIe 5.0 x4 throughput drops: Speeds fail to hit expected peaks.
2. Small/Random Transaction Overhead: While the main PCIe 5.0 x16 GPU slot has better bulk transfers, "random performance still lags behind older Z790 boards".

Assuming P2P transfers over PCIe rely on small transaction packets (e.g., 256 bytes) transferred via the controller, they would likely hit the same latency bottleneck seen in random write operations, accumulating to low bandwidth.

[mratsim]

Have you run nvbandwidth (https://github.com/nvidia/nvbandwidth) or simpleP2P (https://github.com/NVIDIA/cuda-samples/tree/master/cpp/0_Introduction/simpleP2P) to confirm if P2P even works?

p2pBandwidthLatencyTest does not have a sanity check on the results.

In my case I get nan everywhere so there is no transfer at all. And yes similar perf figure reported in p2pBandwidthLatencyTest: very low latency and reduced bandwidth, but it doesn't really matter.