[GSD-13087] UR_RESULT_ERROR_OUT_OF_RESOURCES on 4 GPUs (B70) with large allocations.

[Finistere]

Pre-submission Checklist

• I am using the latest GPU driver version (releases)
• I have searched for similar issues and found none

GPU Hardware

Arc Pro B70

DRI Devices Information

$ ls -la /dev/dri/by-path/
lrwxrwxrwx 1 root root   8 Jul 13 14:51 pci-0000:03:00.0-card -> ../card1
lrwxrwxrwx 1 root root  13 Jul 13 14:51 pci-0000:03:00.0-render -> ../renderD129
lrwxrwxrwx 1 root root   8 Jul 13 14:51 pci-0000:23:00.0-card -> ../card2
lrwxrwxrwx 1 root root  13 Jul 13 14:51 pci-0000:23:00.0-render -> ../renderD130
lrwxrwxrwx 1 root root   8 Jul 13 14:51 pci-0000:c3:00.0-card -> ../card3
lrwxrwxrwx 1 root root  13 Jul 13 14:51 pci-0000:c3:00.0-render -> ../renderD131
lrwxrwxrwx 1 root root   8 Jul 13 14:51 pci-0000:f3:00.0-card -> ../card0
lrwxrwxrwx 1 root root  13 Jul 13 14:51 pci-0000:f3:00.0-render -> ../renderD128

GPU Detailed Information (lspci output)

$ for c in /dev/dri/by-path/pci-*-card; do a=${c##*/pci-}; sudo lspci -vvv -k -s "${a%-card}"; done
03:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
	Subsystem: Intel Corporation Device 1701
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 225
	IOMMU group: 47
	Region 0: Memory at 14004000000 (64-bit, prefetchable) [size=16M]
	Region 2: Memory at 13000000000 (64-bit, prefetchable) [size=32G]
	Expansion ROM at f7400000 [disabled] [size=2M]
	Capabilities: [40] Vendor Specific Information: Len=0c <?>
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
			EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			FRS- TPHComp- ExtTPHComp-
			AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
			AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fee00004  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [d0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [110 v1] Null
	Capabilities: [200 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable+, Smallest Translation Unit: 00
	Capabilities: [420 v1] Physical Resizable BAR
		BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [220 v1] Virtual Resizable BAR
		BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration- 10BitTagReq+ Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
		IOVSta:	Migration-
		Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 1, stride: 1, Device ID: e223
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000014000000000 (64-bit, prefetchable)
		Region 2: Memory at 0000013800000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [400 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Kernel driver in use: xe
	Kernel modules: xe

23:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
	Subsystem: Intel Corporation Device 1701
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 228
	IOMMU group: 71
	Region 0: Memory at 15804000000 (64-bit, prefetchable) [size=16M]
	Region 2: Memory at 14800000000 (64-bit, prefetchable) [size=32G]
	Expansion ROM at f0000000 [disabled] [size=2M]
	Capabilities: [40] Vendor Specific Information: Len=0c <?>
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
			EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			FRS- TPHComp- ExtTPHComp-
			AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
			AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fee00004  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [d0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [110 v1] Null
	Capabilities: [200 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable+, Smallest Translation Unit: 00
	Capabilities: [420 v1] Physical Resizable BAR
		BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [220 v1] Virtual Resizable BAR
		BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration- 10BitTagReq+ Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
		IOVSta:	Migration-
		Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 1, stride: 1, Device ID: e223
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000015800000000 (64-bit, prefetchable)
		Region 2: Memory at 0000015000000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [400 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Kernel driver in use: xe
	Kernel modules: xe

c3:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
	Subsystem: Intel Corporation Device 1701
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 231
	IOMMU group: 18
	Region 0: Memory at 12804000000 (64-bit, prefetchable) [size=16M]
	Region 2: Memory at 11800000000 (64-bit, prefetchable) [size=32G]
	Expansion ROM at f2800000 [disabled] [size=2M]
	Capabilities: [40] Vendor Specific Information: Len=0c <?>
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
			EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			FRS- TPHComp- ExtTPHComp-
			AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
			AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fee00004  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [d0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [110 v1] Null
	Capabilities: [200 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable+, Smallest Translation Unit: 00
	Capabilities: [420 v1] Physical Resizable BAR
		BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [220 v1] Virtual Resizable BAR
		BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration- 10BitTagReq+ Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
		IOVSta:	Migration-
		Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 1, stride: 1, Device ID: e223
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000012800000000 (64-bit, prefetchable)
		Region 2: Memory at 0000012000000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [400 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Kernel driver in use: xe
	Kernel modules: xe

f3:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
	Subsystem: Intel Corporation Device 1701
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 222
	IOMMU group: 10
	Region 0: Memory at 11004000000 (64-bit, prefetchable) [size=16M]
	Region 2: Memory at 10000000000 (64-bit, prefetchable) [size=32G]
	Expansion ROM at f4e00000 [disabled] [size=2M]
	Capabilities: [40] Vendor Specific Information: Len=0c <?>
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
			EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			FRS- TPHComp- ExtTPHComp-
			AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
			AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fee00004  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [d0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [110 v1] Null
	Capabilities: [200 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable+, Smallest Translation Unit: 00
	Capabilities: [420 v1] Physical Resizable BAR
		BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [220 v1] Virtual Resizable BAR
		BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration- 10BitTagReq+ Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
		IOVSta:	Migration-
		Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 1, stride: 1, Device ID: e223
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000011000000000 (64-bit, prefetchable)
		Region 2: Memory at 0000010800000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [400 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Kernel driver in use: xe
	Kernel modules: xe

Driver Version

26.22.38646.6-124.04ppa1

Installed GPU Driver Packages

$ sudo dpkg --list | grep -iE "igc|gmm|opencl|level-zero|fc|level_zero|ocloc|libze"
ii  bpfcc-tools                                      0.29.1+ds-1ubuntu7                               all          tools for BPF Compiler Collection (BCC)
ii  clinfo                                           3.0.23.01.25-1build1                             amd64        Query OpenCL system information
ii  intel-igc-core-2                                 2.34.4                                           amd64        Intel(R) Graphics Compiler for OpenCL(TM)
ii  intel-igc-opencl-2                               2.34.4                                           amd64        Intel(R) Graphics Compiler for OpenCL(TM)
ii  intel-ocloc                                      26.22.38646.6-1~24.04~ppa1                       amd64        Tool for managing Intel Compute GPU device binary format
ii  intel-opencl-icd                                 26.22.38646.6-1~24.04~ppa1                       amd64        Intel graphics compute runtime for OpenCL
ri  level-zero                                       1.28.2                                           amd64        oneAPI Level Zero
ii  level-zero-devel                                 1.28.2                                           amd64        oneAPI Level Zero
ii  libbpfcc:amd64                                   0.29.1+ds-1ubuntu7                               amd64        shared library for BPF Compiler Collection (BCC)
ii  libcbor0.10:amd64                                0.10.2-1.2ubuntu2                                amd64        library for parsing and generating CBOR (RFC 7049)
ii  libdebconfclient0:amd64                          0.271ubuntu3                                     amd64        Debian Configuration Management System (C-implementation library)
ii  libfile-fcntllock-perl                           0.22-4ubuntu5                                    amd64        Perl module for file locking with fcntl(2)
ii  libigc2                                          2.36.3-2~24.04                                   amd64        Core libraries for Intel(R) Graphics Compiler for OpenCL(TM)
ii  libigdfcl2                                       2.36.3-2~24.04                                   amd64        OpenCL library for Intel(R) Graphics Compiler for OpenCL(TM)
ii  libigdgmm12:amd64                                22.10.0-1~24.04~ppa1                             amd64        Intel Graphics Memory Management Library -- shared library
ii  libref-array1t64:amd64                           0.6.2-2.1build1                                  amd64        refcounted array for C
ii  libsigc++-3.0-0:amd64                            3.6.0-2                                          amd64        type-safe Signal Framework for C++ - runtime
ii  libze-intel-gpu1                                 26.22.38646.6-1~24.04~ppa1                       amd64        Intel oneAPI L0 support implementation for Intel GPUs -- shared library
ii  libze1:amd64                                     1.28.6-1~24.04~ppa1                              amd64        oneAPI Level Zero -- share libraries
ii  ocl-icd-libopencl1:amd64                         2.3.2-1build1                                    amd64        Generic OpenCL ICD Loader
ii  python3-bpfcc                                    0.29.1+ds-1ubuntu7                               all          Python 3 wrappers for BPF Compiler Collection (BCC)
ii  python3-idna                                     3.6-2ubuntu0.1                                   all          Python IDNA2008 (RFC 5891) handling (Python 3)

Driver Installation Details

Installed through apt on Ubuntu 24.04.

Linux Distribution

Ubuntu 24.04 LTS

Other Linux Distribution

No response

Kernel Version & Boot Parameters

$ uname -r
6.17.0-35-generic

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.17.0-35-generic root=UUID=cda26a52-4206-48e9-9beb-ed6d68441b9f ro splash vt.handoff=7

$ lsmod | grep -E 'i915|xe'
xe                   3874816  127
intel_vsec             24576  2 pmt_telemetry,xe
gpu_sched              65536  1 xe
drm_gpuvm              49152  1 xe
drm_gpusvm_helper      36864  1 xe
drm_buddy              28672  1 xe
drm_ttm_helper         16384  1 xe
ttm                   126976  2 drm_ttm_helper,xe
drm_exec               12288  2 drm_gpuvm,xe
drm_suballoc_helper    24576  1 xe
drm_display_helper    290816  1 xe
cec                    98304  2 drm_display_helper,xe
i2c_algo_bit           16384  1 xe
video                  77824  2 asus_wmi,xe

Actual Behavior

When using large allocations, the driver ends up reporting UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY on 4 GPUs (B70), but not on 2. The issue seems very similar to this one: #952, which was closed because the author thinks it's the same underlying issue as #953. It's unclear to me whether it's really the same problem, I couldn't clearly identify any metric on my kernel (6.17.0-35-generic) / xe driver that clearly indicated that host memory was the problem. However, disabling peer residency and avoiding P2P memcpy solves the issue for me.

Using NEOReadDebugKeys=1 EnableDeferBacking=0 as suggested in the #953 had no effect. I'll keep following the other issue.

Expected Behavior

The driver shouldn't fail with out of memory errors just for allocating huge memory chunks on each device.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

I have two reproductions in pure SYCL:

• The first one uses a single context and allocates a huge slab in each of the devices' memory. This fails for the third device. However, disabling peer residency with UR_LOADER_USE_LEVEL_ZERO_V2=0 SYCL_UR_USE_LEVEL_ZERO_V2=0 UR_L0_USM_RESIDENT=0x001 works.
• The second one uses a different context for each device. It manages to make the huge allocations, but fails during P2P memcpy.

Single context reproduction

#include <sycl/sycl.hpp>

#include <iostream>
#include <vector>

int main() {
  std::vector<sycl::device> gpus;
  for (const sycl::platform &platform : sycl::platform::get_platforms())
    if (platform.get_backend() == sycl::backend::ext_oneapi_level_zero)
      for (const sycl::device &device : platform.get_devices())
        if (device.is_gpu())
          gpus.push_back(device);

  if (gpus.size() < 4) {
    std::cerr << "need 4 Level Zero GPUs, found " << gpus.size() << "\n";
    return 2;
  }

  constexpr std::size_t num_gpus = 4;
  sycl::context context({gpus[0], gpus[1], gpus[2], gpus[3]});
  std::vector<sycl::queue> queues;
  std::vector<void *> slabs(num_gpus);
  std::vector<std::size_t> slab_bytes(num_gpus);
  queues.reserve(num_gpus);

  for (std::size_t i = 0; i < num_gpus; ++i) {
    queues.emplace_back(context, gpus[i], sycl::property::queue::in_order{});
  }

  for (std::size_t i = 0; i < num_gpus; ++i) {
    slab_bytes[i] =
        gpus[i].get_info<sycl::info::device::global_mem_size>() * 9 / 10;
    slabs[i] = sycl::malloc_device(slab_bytes[i], queues[i]);
    std::cout << "slab[" << i << "]=" << slabs[i] << " bytes=" << slab_bytes[i]
              << "\n";
    if (!slabs[i])
      return 1;
  }
}

Multiple contexts reproduction

#include <sycl/sycl.hpp>

#include <iostream>
#include <vector>

int main() {
  std::vector<sycl::device> gpus;
  for (const sycl::platform &platform : sycl::platform::get_platforms())
    if (platform.get_backend() == sycl::backend::ext_oneapi_level_zero)
      for (const sycl::device &device : platform.get_devices())
        if (device.is_gpu())
          gpus.push_back(device);

  if (gpus.size() < 4) {
    std::cerr << "need 4 Level Zero GPUs, found " << gpus.size() << "\n";
    return 2;
  }

  constexpr std::size_t num_gpus = 4;
  std::vector<sycl::context> contexts;
  std::vector<sycl::queue> queues;
  std::vector<void *> slabs(num_gpus);
  std::vector<std::size_t> slab_bytes(num_gpus);
  contexts.reserve(num_gpus);
  queues.reserve(num_gpus);

  for (std::size_t i = 0; i < num_gpus; ++i) {
    contexts.emplace_back(gpus[i]);
    queues.emplace_back(contexts.back(), gpus[i],
                        sycl::property::queue::in_order{});
  }

  for (std::size_t i = 0; i < num_gpus; ++i) {
    slab_bytes[i] =
        gpus[i].get_info<sycl::info::device::global_mem_size>() * 9 / 10;
    slabs[i] = sycl::malloc_device(slab_bytes[i], queues[i]);
    std::cout << "slab[" << i << "]=" << slabs[i] << " bytes=" << slab_bytes[i]
              << "\n";
    if (!slabs[i])
      return 1;
  }

  constexpr std::size_t copy_bytes = 1024;
  for (std::size_t src = 0; src < num_gpus; ++src) {
    for (std::size_t dst = 0; dst < num_gpus; ++dst) {
      if (src == dst)
        continue;

      queues[dst].memcpy(slabs[dst], slabs[src], copy_bytes).wait_and_throw();
      std::cout << "copied " << copy_bytes << " bytes from device " << src
                << " to device " << dst << "\n";
    }
  }
}

Is this a regression?

• Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

No response

First Known Failing Driver Version

No response

API Call Logs

No response

strace Logs

No response

System Logs / dmesg Output

No response

Backtrace (if crash or hang occurred)

No response

Source Code / Reproducer

No response

Command Line / Application Details

No response

oneAPI Version (if applicable)

$ icpx --version
Intel(R) oneAPI DPC++/C++ Compiler 2026.0.0 (2026.0.0.20260331)
Target: x86_64-unknown-linux-gnu
Thread model: posix

Screenshots / Video

No response

Additional Notes

No response

[neudinger]

Hello Intel team

@BartoszDunajski @narenbagria

I got similar issue using oneCCL but i got first UR_RESULT_ERROR_DEVICE_LOST error and then UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY error when i set CCL_SYCL_COPY_ENGINE=1.

I will also create an issue in the oneCCL repo to track this problem, but I wanted to share my findings here as well since it seems to be related to the same underlying problem.

I reproduced the multiple-context P2P copy failure from #957. I use the exact same configuration as the reproducer, four Arc Pro B70 GPUs using the released 26.22 Compute Runtime.

From what I understand the test creates one SYCL context per GPU, allocates 90% of each GPU’s reported memory, and runs all twelve directed 1 KiB peer copies.

The allocations succeed, but one of the later copies fails with:

sycl::_V1::exception: level_zero backend failed with error: 39
(UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)

I use the Level Zero Loader environment variable ZE_ENABLE_ALT_DRIVERS to select the locally built libze_intel_gpu.so.1.

ZE_ENABLE_ALT_DRIVERS=/home/kevin/compute-runtime-build-system-nolto/bin/libze_intel_gpu.so.1 \
LD_LIBRARY_PATH=/home/kevin/compute-runtime-system-deps/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} \
ONEAPI_DEVICE_SELECTOR=level_zero:* \ 
timeout -k 20s 120s \ 
./issue_957_multiple_contexts

slab[0]=0xffff82ac0bc00000 bytes=30818068070
slab[1]=0xffff83010bc00000 bytes=30818068070
slab[2]=0xffff83560bc00000 bytes=30818068070
slab[3]=0xffff83ab0bc00000 bytes=30818068070
copied 1024 bytes from device 0 to device 1
...
copied 1024 bytes from device 3 to device 2

With the experimental runtime and kernel, all twelve copies complete.

What seems to be happening

Although each copy is only 1 KiB, the current peer-import path exports and VM-binds the entire remote allocation.
In this test, each allocation is about 28.7 GiB.

With separate contexts importing each other’s allocations, the accumulated mappings eventually exhaust Xe driver resources.
Unified Runtime then reports this as UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY.

My experimental fix keeps the original GPU virtual address but imports only the part of the remote allocation needed for the copy.

It currently uses cached, 2 MiB-aligned windows and splits copies that cross a window boundary.

Prototype branch:

https://github.com/intel/compute-runtime/compare/releases/26.22...neudinger:compute-runtime:kevin/xe-range-bounded-peer-windows?expand=1

The relevant runtime changes are:

• support range-based Xe DMA-BUF exports;
• cache peer imports by allocation and 2 MiB window;
• split copy-engine transfers at window boundaries;
• preserve the BCS/copy-offload choice for segmented copies.

The last item is important because otherwise a copy can be encoded for BCS but submitted through CCS, which may hang or reset CCS.

Note:
Blitter Command Streamer (BCS): handles data movement, such as copying or filling GPU memory. It uses the GPU’s blitter/copy engine and usually does not execute compute kernels.
Compute Command Streamer (CCS): handles compute workloads. It sends instructions to the GPU execution units that run SYCL, OpenCL, or other compute kernels.

When the peer copy was split into 2 MiB segments, each segment was encoded for BCS.

However, the outer command list did not preserve that choice, so the BCS commands could be submitted through CCS instead.

This engine mismatch could hang or reset CCS and result in UR_RESULT_ERROR_DEVICE_LOST.

The fix propagates the copy-engine choice back to the outer operation:

memoryCopyParams.copyOffloadAllowed |=
    segmentParams.copyOffloadAllowed;

This ensures that commands encoded for BCS are also submitted through BCS.

The branch also includes two prototype-only commits:

• an experimental Xe kernel/UAPI change, which would need a separate DRM/Xe review;
• a local build workaround, which would be removed before submission.

Feedback requested

Does this general design make sense?

In particular:

1. Is a range-bounded DMA-BUF export with a 2 MiB peer-window cache a reasonable approach?
2. Should this use a new Xe DRM ioctl, or is there already a suitable kernel interface?
3. What test coverage would maintainers expect for window reuse, allocation lifetime, boundary-crossing copies, and BCS/CCS selection?

If the approach looks reasonable:

• What are the next steps for a focused Compute Runtime PR with the four runtime commits and test
• How to submit a separate Linux Xe DRM patch series for the range-export UAPI and DMA-BUF slice lifetime/offset handling.

Scope

This only addresses the multiple-context P2P copy failure in #957.

It does not fix:

• the single-context allocation failure from the same issue;
• the oneCCL UR_RESULT_ERROR_DEVICE_LOST or UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY problem completely because it also requires a fix in the oneCCL library.

The prototype requires both the patched Compute Runtime and the matching Xe kernel.
With an unpatched kernel, the runtime falls back to importing the full allocation, so the OOM may still occur.