You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
$ dpkg --list | grep -iE "igc|gmm|opencl|level-zero|level_zero|ocloc|libze"
ii clinfo 3.0.25.02.14-1build1
ii intel-opencl-icd 26.05.37020.3-1
ii intel-opencl-icd-legacy 24.35.30872.45-1
ii libigc1:amd64 1.0.17791.18+1-3
ii libigc2:amd64 2.28.4-4
ii libigdfcl1:amd64 1.0.17791.18+1-3
ii libigdfcl2:amd64 2.28.4-4
ii libigdgmm12:amd64 22.9.0+ds1-1
ii libze-intel-gpu-legacy1-1 24.35.30872.45-1
ii libze-intel-gpu1 26.05.37020.3-1
ii libze1:amd64 1.28.2-2
ii ocl-icd-libopencl1:amd64 2.3.4-1
Driver Installation Details:
Installation method: Ubuntu distribution repository (us.archive.ubuntu.com resolute/universe), standard apt install — no Intel PPA, no source builds, no DKMS
Version held at 26.05.37020.3-1 (see checklist note)
No GPU-related kernel boot parameters
Linux Distribution: Other — Ubuntu 26.04 LTS (resolute)
Kernel Version & Boot Parameters:
$ uname -r
7.0.0-27-generic # at the time of all incidents (host has since moved to 7.0.0-28-generic; driver unchanged)
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.0-28-generic root=UUID=... ro video=HDMI-A-5:1920x1080@60D crashkernel=2G-4G:320M,...
$ lsmod | grep -E '^xe|gpu_sched|drm_gpuvm'
xe 4419584 54
gpu_sched 69632 1 xe
drm_gpuvm 57344 1 xe
Host: AMD Threadripper 2950WX, 128 GB DDR4. Workloads run in Docker containers with a single render node mapped (--device /dev/dri/renderD130 --device /dev/dri/card2, ONEAPI_DEVICE_SELECTOR=level_zero:0).
Issue Description
Actual Behavior:
After a burst of UR_RESULT_ERROR_DEVICE_LOST / xe … exec queue reset detected events during Level-Zero GPU training (PyTorch 2.13.0+xpu) on the B70s, page-cache pages of ordinary files become silently corrupt host-wide: every subsequent reader — including pure-CPU processes in containers with no GPU device mounted — reads garbage for affected files (model weight shards, data caches). On-disk copies remain pristine (no mtime change; post-eviction reads hash correctly). The corruption survives container restarts, xe unbind/bind, FLR, and PCI bus reset — consistent with the damage living in host RAM (the page cache), which none of those touch.
Evidence chain from the incidents of 2026-08-02:
Training crashed with UR_RESULT_ERROR_DEVICE_LOST inside loss.backward(); kernel log shows repeated storms with the signature VM worker error: -16 (or -12) followed by 5–7× exec queue reset detected, on both B70s (see Diagnostics below).
From the first burst onward, every launch of the (deterministic) workload misbehaved identically: forward loss pinned at exactly ln(vocab) (all-zero logits), or NaN, or a nonsense-but-finite value — on both B70s, on a B60, and in a pure-CPU container with no GPU device mounted. The CPU reproduction rules out every GPU-execution explanation.
Conventional causes eliminated: all model params finite; CPU-vs-GPU layer-by-layer forward matched exactly (both reading the same corrupt bytes); no file mtimes changed; freshly rebuilt derived data matched cached copies for spot-checked entries.
Smoking gun — content-addressed files: Hugging Face hub blobs are named by their sha256. sha256sum of a resident 4.99 GB weights shard through the page cache → 3b3e9774…, which is not its filename hash f1dae409…. After posix_fadvise(POSIX_FADV_DONTNEED) (drop cached pages; no root needed) the same file hashes exactly to its name. Disk pristine; RAM pages poisoned.
Purging only the weights shard was not enough — the derived token-cache file's pages were also poisoned (in-range-but-wrong values). Purging all files the workload reads restored fully healthy behavior, reproducibly. Multiple independent poisoning events across one day, each traceable to a DEVICE_LOST burst.
Expected Behavior:
An exec-queue reset / DEVICE_LOST should terminate the offending context and, at worst, cost availability. It must never modify host memory outside the process's own mappings — in particular not file-backed page-cache pages belonging to arbitrary files and other processes.
Reproduction Rate: Sometimes reproduces (~50–74%) — the reset storms occur reliably within hours under the provoking workload; the page-cache corruption follows a subset of storms (sporadic per storm, depends on which physical pages the errant writes land in — one monitored storm left the checked files intact).
Steps to Reproduce:
Dual Arc Pro B70, driver 26.05.37020.3, xe KMD, workload in a container with one render node.
Read a large content-addressed file so it is resident in the page cache (e.g. sha256sum a multi-GB HF hub blob once — it will hash correctly).
Provoke exec-queue-reset storms: sustained PyTorch 2.13.0+xpu fp32 training with long sequences under VRAM oversubscription (GTT spill → >5 s kernels → resets → DEVICE_LOST). A single-op candidate trigger also exists (observed once, see Reproducer): fp32 torch.softmax fwd+bwd at shape (1,24,1400,1400) → VM worker error: -12 + reset burst with no memory pressure (~1 GB tensors on the 31 GB card).
After a DEVICE_LOST burst, re-run sha256sum on the resident file through the cache: on a poisoned run the hash no longer matches the filename.
Drop the pages (posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)) and hash again: it matches — proving RAM-only corruption, disk intact.
Regression Information
Unknown — 26.05.37020.3 is the only driver this hardware has run in production here, so there is no known-good/known-bad version pair. Not claimed as a regression.
Diagnostics & Logs
System Logs / dmesg Output:
Storm signature (three storms on 0000:44:00.0 and one on 0000:48:00.0 within ~15 minutes, kernel 7.0.0-27-generic):
Aug 02 02:58:04 core kernel: xe 0000:44:00.0: [drm] VM worker error: -16
Aug 02 02:58:09-11 core kernel: xe 0000:44:00.0: [drm] exec queue reset detected (x7)
Aug 02 03:02:57 core kernel: xe 0000:48:00.0: [drm] VM worker error: -16
Aug 02 03:03:02-05 core kernel: xe 0000:48:00.0: [drm] exec queue reset detected (x7)
Aug 02 03:06:43 core kernel: xe 0000:44:00.0: [drm] VM worker error: -12
Aug 02 03:06:50-52 core kernel: xe 0000:44:00.0: [drm] exec queue reset detected (x3)
Aug 02 03:10:04 core kernel: xe 0000:44:00.0: [drm] VM worker error: -16
Aug 02 03:10:09-11 core kernel: xe 0000:44:00.0: [drm] exec queue reset detected (x7)
The single-op candidate trigger produced the same signature (17:10, same day):
Aug 02 17:10:38 core kernel: xe 0000:44:00.0: [drm] VM worker error: -12
Aug 02 17:10:50-17:11:45 core kernel: xe 0000:44:00.0: [drm] exec queue reset detected (x5)
Recovery-path escalation when an FLR was issued on a wedged card (04:08–04:10) — note the TLB invalidation fence timeout, which seems directly relevant to the corruption mechanism (stale device-side translations surviving into reset):
Aug 02 04:08:14 core kernel: xe 0000:44:00.0: resetting
Aug 02 04:08:14 core kernel: xe 0000:44:00.0: reset done
Aug 02 04:08:26 core kernel: xe 0000:44:00.0: [drm] *ERROR* TLB invalidation fence timeout, seqno=17309 recv=17308
Aug 02 04:08:29 core kernel: xe 0000:44:00.0: [drm] *ERROR* TLB invalidation fence timeout, seqno=17309 recv=17308
Aug 02 04:08:34-44 core kernel: xe 0000:44:00.0: [drm] Tile0: GT0: Check job timeout: seqno=278069, lrc_seqno=278069, guc_id=0, not started (x3)
Aug 02 04:08:49 core kernel: xe 0000:44:00.0: [drm] Tile0: GT0: trying reset from guc_exec_queue_timedout_job [xe]
Aug 02 04:08:49 core kernel: xe 0000:44:00.0: [drm] *ERROR* Tile0: GT0: load failed: status = 0x40000056, time = 0ms, freq = 2800MHz (req 2800MHz)
Aug 02 04:08:49 core kernel: xe 0000:44:00.0: [drm] *ERROR* Tile0: GT0: load failed: status: Reset = 0, BootROM = 0x2B, UKernel = 0x00, MIA = 0x00, Auth = 0x01
Aug 02 04:08:49 core kernel: xe 0000:44:00.0: [drm] *ERROR* Tile0: GT0: firmware production part check failure
Aug 02 04:08:49 core kernel: xe 0000:44:00.0: [drm] *ERROR* Tile0: GT0: reset failed (-EPROTO)
Aug 02 04:08:49 core kernel: xe 0000:44:00.0: [drm] *ERROR* CRITICAL: Xe has declared device 0000:44:00.0 as wedged.
Aug 02 04:08:49 core kernel: xe 0000:44:00.0: [drm] device wedged, needs recovery
Aug 02 04:10:26 core kernel: xe 0000:44:00.0: resetting
Aug 02 04:10:27 core kernel: xe 0000:44:00.0: [drm] *ERROR* Tile0: GT0: Force wake domain 0: wake. MMIO unreliable (forcewake register returns 0xFFFFFFFF)!
Full journal excerpts for any window available on request (persistent journal covers all incidents).
API Call Logs: Not captured during the incidents (production runs). We can capture a unitrace/CLIntercept log of a provoking run on request.
strace Logs: Not applicable — the corruption manifests in unrelated reader processes; no syscall-level anomaly in the GPU process beyond the DEVICE_LOST return.
Backtrace: No UMD crash in the training process (it receives UR_RESULT_ERROR_DEVICE_LOST cleanly). One related breadcrumb: on the wedged card, the next UMD touch after FLR aborted at ioctl_helper_xe.cpp:1572; recovery required xe driver unbind/bind.
Reproducer Information
Source Code / Reproducer:
Candidate single-op storm trigger (PyTorch 2.13.0+xpu; observed once producing the VM worker error: -12 storm above; not re-run to confirm determinism because each storm risks corrupting resident pages of unrelated services on this production host):
Note: this size sits just past a range where the same softmax kernel silently returns wrong results (intel/torch-xpu-ops#4772) — the same kernel family escalates from wrong-math to device-kill as size grows.
Corruption detection/remediation recipe (usable by anyone):
# detect: any content-addressed file (e.g. HF hub blob) read THROUGH the cache
sha256sum /path/hub/models--*/blobs/<sha># mismatch => poisoned pages (or disk)# remediate: evict cached pages, then re-verify
python3 -c "import os; fd=os.open('<file>',os.O_RDONLY); os.posix_fadvise(fd,0,0,os.POSIX_FADV_DONTNEED)"
sha256sum <same file># now matches => RAM was bad, disk is fine
A post-purge mismatch would indicate true disk corruption — never observed across many incidents; it was page-cache damage every time.
Command Line / Application Details:
docker run --rm --device /dev/dri/renderD130 --device /dev/dri/card2 \
-e ONEAPI_DEVICE_SELECTOR=level_zero:0 <image with torch 2.13.0+xpu> python <script>
Additional Information
oneAPI Version: Not using a standalone oneAPI install; Level-Zero is exercised through the PyTorch 2.13.0+xpu wheel (SYCL_COMPILER_VERSION=20260000, oneDNN v3.12.0, Intel(R) oneAPI Unified Runtime over Level-Zero V2).
Impact: any host running sustained GPU compute on BMG with this driver can have arbitrary resident file data silently corrupted after a reset storm — including files belonging to unrelated services (in our case a CPU-only inference container served garbage weights until the cache was dropped). Data-at-rest is safe; anything cached is suspect after any DEVICE_LOST event.
We are happy to run debug builds, patches, or additional instrumentation on the affected host.
(Restructured per the Issue Submission Guide, following the Linux GPU Driver Issue template.)
Pre-submission Checklist
System Information
GPU Hardware: 2× Intel Arc Pro B70 (Battlemage G31) — the affected devices — plus 2× Arc Pro B60 (Battlemage G21) in the same host
DRI Devices Information:
GPU Detailed Information (lspci output):
(PCI IDs: B70 =
8086:e223, B60 =8086:e211. GuC firmwarexe/bmg_guc_70.binversion 70.58.0.)Driver Version: 26.05.37020.3
Installed GPU Driver Packages:
Driver Installation Details:
us.archive.ubuntu.com resolute/universe), standardapt install— no Intel PPA, no source builds, no DKMSLinux Distribution: Other — Ubuntu 26.04 LTS (resolute)
Kernel Version & Boot Parameters:
Host: AMD Threadripper 2950WX, 128 GB DDR4. Workloads run in Docker containers with a single render node mapped (
--device /dev/dri/renderD130 --device /dev/dri/card2,ONEAPI_DEVICE_SELECTOR=level_zero:0).Issue Description
Actual Behavior:
After a burst of
UR_RESULT_ERROR_DEVICE_LOST/xe … exec queue reset detectedevents during Level-Zero GPU training (PyTorch 2.13.0+xpu) on the B70s, page-cache pages of ordinary files become silently corrupt host-wide: every subsequent reader — including pure-CPU processes in containers with no GPU device mounted — reads garbage for affected files (model weight shards, data caches). On-disk copies remain pristine (no mtime change; post-eviction reads hash correctly). The corruption survives container restarts, xe unbind/bind, FLR, and PCI bus reset — consistent with the damage living in host RAM (the page cache), which none of those touch.Evidence chain from the incidents of 2026-08-02:
UR_RESULT_ERROR_DEVICE_LOSTinsideloss.backward(); kernel log shows repeated storms with the signatureVM worker error: -16(or-12) followed by 5–7×exec queue reset detected, on both B70s (see Diagnostics below).ln(vocab)(all-zero logits), or NaN, or a nonsense-but-finite value — on both B70s, on a B60, and in a pure-CPU container with no GPU device mounted. The CPU reproduction rules out every GPU-execution explanation.sha256sumof a resident 4.99 GB weights shard through the page cache →3b3e9774…, which is not its filename hashf1dae409…. Afterposix_fadvise(POSIX_FADV_DONTNEED)(drop cached pages; no root needed) the same file hashes exactly to its name. Disk pristine; RAM pages poisoned.Expected Behavior:
An exec-queue reset / DEVICE_LOST should terminate the offending context and, at worst, cost availability. It must never modify host memory outside the process's own mappings — in particular not file-backed page-cache pages belonging to arbitrary files and other processes.
Reproduction Rate: Sometimes reproduces (~50–74%) — the reset storms occur reliably within hours under the provoking workload; the page-cache corruption follows a subset of storms (sporadic per storm, depends on which physical pages the errant writes land in — one monitored storm left the checked files intact).
Steps to Reproduce:
sha256suma multi-GB HF hub blob once — it will hash correctly).torch.softmaxfwd+bwd at shape(1,24,1400,1400)→VM worker error: -12+ reset burst with no memory pressure (~1 GB tensors on the 31 GB card).sha256sumon the resident file through the cache: on a poisoned run the hash no longer matches the filename.posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)) and hash again: it matches — proving RAM-only corruption, disk intact.Regression Information
Unknown — 26.05.37020.3 is the only driver this hardware has run in production here, so there is no known-good/known-bad version pair. Not claimed as a regression.
Diagnostics & Logs
System Logs / dmesg Output:
Storm signature (three storms on 0000:44:00.0 and one on 0000:48:00.0 within ~15 minutes, kernel 7.0.0-27-generic):
The single-op candidate trigger produced the same signature (17:10, same day):
Recovery-path escalation when an FLR was issued on a wedged card (04:08–04:10) — note the TLB invalidation fence timeout, which seems directly relevant to the corruption mechanism (stale device-side translations surviving into reset):
Full journal excerpts for any window available on request (persistent journal covers all incidents).
API Call Logs: Not captured during the incidents (production runs). We can capture a unitrace/CLIntercept log of a provoking run on request.
strace Logs: Not applicable — the corruption manifests in unrelated reader processes; no syscall-level anomaly in the GPU process beyond the DEVICE_LOST return.
Backtrace: No UMD crash in the training process (it receives
UR_RESULT_ERROR_DEVICE_LOSTcleanly). One related breadcrumb: on the wedged card, the next UMD touch after FLR aborted atioctl_helper_xe.cpp:1572; recovery required xe driver unbind/bind.Reproducer Information
Source Code / Reproducer:
Candidate single-op storm trigger (PyTorch 2.13.0+xpu; observed once producing the
VM worker error: -12storm above; not re-run to confirm determinism because each storm risks corrupting resident pages of unrelated services on this production host):Note: this size sits just past a range where the same softmax kernel silently returns wrong results (intel/torch-xpu-ops#4772) — the same kernel family escalates from wrong-math to device-kill as size grows.
Corruption detection/remediation recipe (usable by anyone):
A post-purge mismatch would indicate true disk corruption — never observed across many incidents; it was page-cache damage every time.
Command Line / Application Details:
Additional Information
oneAPI Version: Not using a standalone oneAPI install; Level-Zero is exercised through the PyTorch 2.13.0+xpu wheel (SYCL_COMPILER_VERSION=20260000, oneDNN v3.12.0,
Intel(R) oneAPI Unified Runtime over Level-Zero V2).Additional Notes:
TLB invalidation fence timeoutduring the recovery path above is exactly the kind of failure that could leave stale device-side translations pointing at recycled host pages. [GSD-13038] Regression in 26.09+: dual-GPU workloads eagerly pin a host-RAM mirror of every large device USM allocation on the peer device, exhausting system memory #953 (host-RAM pinning of USM mirrors) is a plausible route by which file-backed pages end up within the device's reach; [GSD-12973] Arc Pro B70 4-GPU: SYCL/Level Zero peer memcpy corrupts data and causes DEVICE_LOST #942 shows BMG DMA paths corrupting data in another context.