Skip to content

[GSD-13120] Incorrect process memory usage #964

Description

@Steve-Tech

Pre-submission Checklist

  • I am using the latest GPU driver version (releases)
  • I have searched for similar issues and found none

GPU Hardware

Intel Arc A770

DRI Devices Information

stephen@Stephens-PC:~$ ls -ls /dev/dri/*
0 crw-rw----+ 1 root video  226,   0 Jul 27 10:38 /dev/dri/card0
0 crw-rw----+ 1 root render 226, 128 Jul 27 10:38 /dev/dri/renderD128

/dev/dri/by-path:
total 0
0 lrwxrwxrwx+ 1 root root  8 Jul 27 10:36 pci-0000:03:00.0-card -> ../card0
0 lrwxrwxrwx+ 1 root root 13 Jul 27 10:36 pci-0000:03:00.0-render -> ../renderD128
stephen@Stephens-PC:~$ ls -la /dev/dri/by-path/
total 0
drwxr-xr-x  2 root root  80 Jul 27 10:36 .
drwxr-xr-x  3 root root 100 Jul 27 10:36 ..
lrwxrwxrwx+ 1 root root   8 Jul 27 10:36 pci-0000:03:00.0-card -> ../card0
lrwxrwxrwx+ 1 root root  13 Jul 27 10:36 pci-0000:03:00.0-render -> ../renderD128

GPU Detailed Information (lspci output)

stephen@Stephens-PC:~$ sudo lspci -vvv -k -s 0000:03:00.0
03:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A770] (rev 08) (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1020
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupts: unknown pin routed to IRQ 179, MSI(X) routed to IRQ 179
        IOMMU group: 16
        Region 0: Memory at dd000000 (64-bit, non-prefetchable) [size=16M]
        Region 2: Memory at f800000000 (64-bit, prefetchable) [size=16G]
        Expansion ROM at de000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
                CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
                      EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
                      1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
                      DDROCCAP- OCEn- DDRWrtVrefEn+ DDR3LEn+
                CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
                      SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
                      PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
                      DDR3MaxFreq=2932MHz LPDDR3En-
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [d0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [420 v1] Physical Resizable BAR
                BAR 2: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
        Capabilities: [400 v1] Latency Tolerance Reporting
                Max snoop latency: 1048576ns
                Max no snoop latency: 1048576ns
        Kernel driver in use: xe
        Kernel modules: i915, xe

Driver Version

26.27.39122.11-0

Installed GPU Driver Packages

stephen@Stephens-PC:~$ sudo dpkg --list | grep -iE "igc|gmm|opencl|level-zero|fc|level_zero|ocloc|libze"
ii  clinfo                                               3.0.25.02.14-1                            amd64        Query OpenCL system information
ii  intel-igc-core-2                                     2.38.2                                    amd64        Intel(R) Graphics Compiler for OpenCL(TM)
ii  intel-igc-opencl-2                                   2.38.2                                    amd64        Intel(R) Graphics Compiler for OpenCL(TM)
ii  intel-ocloc                                          26.27.39122.11-0                          amd64        Tool for managing Intel Compute GPU device binary format
ii  intel-opencl-icd                                     26.27.39122.11-0                          amd64        Intel graphics compute runtime for OpenCL
ii  intltool-debian                                      0.35.0+20060710.6                         all          Help i18n of RFC822 compliant config files
ii  libcbor0.10:amd64                                    0.10.2-2.1+b1                             amd64        library for parsing and generating CBOR (RFC 7049)
ii  libclblast-dev:amd64                                 1.7.0-1                                   amd64        Tuned OpenCL BLAS library (development files)
ii  libclblast1:amd64                                    1.7.0-1                                   amd64        Tuned OpenCL BLAS library
ii  libdebconfclient0:amd64                              0.283                                     amd64        Debian Configuration Management System (C-implementation library)
ii  libemail-date-format-perl                            1.008-1                                   all          Module to generate RFC-2822-valid date strings
ii  libfile-fcntllock-perl                               0.22-4+b4                                 amd64        Perl module for file locking with fcntl(2)
ii  libigc2                                              2.36.3-2~25.10                            amd64        Core libraries for Intel(R) Graphics Compiler for OpenCL(TM)
ii  libigdfcl2                                           2.38.3-3~26.04                            amd64        OpenCL library for Intel(R) Graphics Compiler for OpenCL(TM)
ii  libigdgmm12:amd64                                    22.10.0                                   amd64        Intel Graphics Memory Management Library -- shared library
ii  libkf5configcore5:amd64                              5.116.0-2+b1                              amd64        configuration settings framework for Qt
ii  libkf6configcore6:amd64                              6.26.0-1                                  amd64        configuration settings framework for Qt
ii  libkf6newstuffcore6:amd64                            6.26.0-1                                  amd64        Support for downloading application assets from the network
ii  libsigc++-2.0-0v5:amd64                              2.12.1-4+b1                               amd64        type-safe Signal Framework for C++ - runtime
ii  liburiparser1:amd64                                  1.0.1+dfsg-1                              amd64        URI parsing library compliant with RFC 3986
ii  libze-dev:amd64                                      1.32.0-1~26.04~ppa1                       amd64        oneAPI Level Zero -- development files
ii  libze-intel-gpu-raytracing                           1.2.4-1~25.10~ppa3                        amd64        Level Zero Ray Tracing Support library
ii  libze-intel-gpu1                                     26.27.39122.11-0                          amd64        Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
ii  libze1:amd64                                         1.32.0-1~26.04~ppa1                       amd64        oneAPI Level Zero -- share libraries
ii  linux-headers-7.1.5steves+                           7.1.5-00026-gcd82fcea38d9-228             amd64        Linux kernel headers for 7.1.5steves+ on amd64
rc  linux-image-6.17.9+                                  6.17.9-00022-gd1c231fc69d2-215            amd64        Linux kernel, version 6.17.9+
rc  linux-image-6.9.2+                                   6.9.2-00034-g134fc5e78263-123             amd64        Linux kernel, version 6.9.2+
ii  linux-image-7.1.5steves+                             7.1.5-00026-gcd82fcea38d9-228             amd64        Linux kernel, version 7.1.5steves+
ii  node-punycode                                        2.3.1+~2.1.4-1                            all          Nodejs robust Punycode converter fully RFC compliant
ii  node-ws                                              8.21.1+~cs14.19.1-1                       all          RFC-6455 WebSocket implementation module for Node.js
ii  ocl-icd-libopencl1:amd64                             2.3.4-1+b1                                amd64        Generic OpenCL ICD Loader
ii  ocl-icd-opencl-dev:amd64                             2.3.4-1+b1                                amd64        OpenCL development files
ii  opencl-c-headers                                     3.0~2025.07.22-2                          all          OpenCL (Open Computing Language) C header files
ii  opencl-clhpp-headers                                 3.0~2025.07.22-1                          all          C++ headers for OpenCL development
ii  python3-idna                                         3.18-1                                    all          Python IDNA2008 (RFC 5891) handling (Python 3)
ii  python3-rfc3339-validator                            0.1.4-2                                   all          RFC 3339 validator (Python library)
ii  python3-rfc3986-validator                            0.1.1-2                                   all          RFC 3986 validator (Python library)
ii  python3-rfc3987                                      1.3.8-3                                   all          Parsing and validation of URIs (RFC 3986) and IRIs (RFC 3987)
ii  python3-uritemplate                                  4.2.0-4                                   all          implementation of RFC6570 - Python 3.x
ii  smistrip                                             0.4.8+dfsg2-18                            all          extract MIB from text files like RFC

Driver Installation Details

Added deb [arch=amd64v3] https://ppa.launchpadcontent.net/kobuk-team/intel-graphics-staging/ubuntu/ ./devel main to /etc/apt/sources.list.d/kobuk-team-ubuntu-intel-graphics-trixie.list

I am aware this is an Ubuntu repository, and I am running Debian, which is probably quite unsupported. I also installed the debian packages from the releases page, as the repo doesn't work for me at the moment since I don't have libc6 >= 2.43 (only 2.42-17).

Linux Distribution

Other (please specify below)

Other Linux Distribution

Debian 14

Kernel Version & Boot Parameters

stephen@Stephens-PC:~$ uname -r
7.1.5steves+
stephen@Stephens-PC:~$ cat /proc/cmdline
root=PARTUUID=c3456c59-931c-428e-ad1a-ea051c20c80f resume=PARTUUID=752e7e07-bb23-498c-9073-8ff2f049c315 ro rootfstype=btrfs rootflags=ssd,subvol=@ quiet splash amd_pstate=guided i915.force_probe=!56a0 xe.force_probe=56a0 drm.edid_firmware=DP-2:edid/Samsung_LC24RG50_VRR.bin,DP-3:edid/MSI_MP2412_VRR.bin systemd.gpt_auto=no systemd.machine_id=609236adbfca4a1eaf13fde52e9bb104
stephen@Stephens-PC:~$ lsmod | grep -E 'i915|xe'
xe                   3928064  380
drm_gpusvm_helper      57344  1 xe
drm_gpuvm              53248  1 xe
drm_suballoc_helper    20480  1 xe
drm_exec               12288  2 drm_gpuvm,xe
gpu_sched              65536  1 xe
drm_ttm_helper         16384  1 xe
i915                 4538368  0
intel_gtt              24576  1 i915
drm_buddy              12288  2 xe,i915
i2c_algo_bit           20480  2 xe,i915
drm_display_helper    270336  6 xe,i915
cec                    77824  3 drm_display_helper,xe,i915
ttm                   135168  3 drm_ttm_helper,xe,i915
drm_client_lib         16384  2 xe,i915
video                  77824  3 asus_wmi,xe,i915
drm_kms_helper        233472  5 drm_display_helper,drm_ttm_helper,drm_client_lib,xe,i915
drm                   794624  70 gpu_sched,drm_kms_helper,drm_exec,drm_gpuvm,drm_suballoc_helper,drm_display_helper,drm_buddy,drm_gpusvm_helper,drm_ttm_helper,drm_client_lib,xe,i915,ttm
configfs               57344  3 rdma_cm,xe

Actual Behavior

The process 'GPU Memory Usage' values from xpu-smi seem to be much more than the card's total memory usage.

E.g. Firefox is reported to be using 7692 MiB, when the card's total is only 3642MiB.

+----------------------------------------------+----------------------------+------------------------+
|            Intel XPU-SMI v2.0    Driver: 96E1802BAF7F7118043A89B    Level Zero: 1.32.0             |
+----------------------------------------------+----------------------------+------------------------+
| GPU  Name                    Persistence-M   | Bus-Id            Disp.A   |   Volatile Uncorr. ECC |
| Fan              Temp  Pwr:Usage/Cap         | Memory-Usage               |   GPU-Util  Compute M. |
+----------------------------------------------+----------------------------+------------------------+
|   0  Intel(R) Arc(TM) A770   Off             | 0000:03:00.0      Off      |               Disabled |
|      Graphics                                |                            |                        |
| N/A               N/A  45W / 190W            | 3642MiB / 16288MiB         |      N/A       Default |
+----------------------------------------------+----------------------------+------------------------+

+-------+----------+--------+----------------------------------------------------+--------------------+
| Processes:                                                                                          |
|   GPU        PID    Type    Process Name                                           GPU Memory Usage |
+-------+----------+--------+----------------------------------------------------+--------------------+
|     0       2345     C      /usr/bin/ksecretd                                                23 MiB |
|     0       2675     C      /usr/bin/Xwayland                                                29 MiB |
|     0       2800     C      /usr/bin/ksmserver                                               13 MiB |
|     0       2802     C      /usr/bin/kded6                                                   23 MiB |
|     0       2857     C      /usr/bin/plasmashell                                           1402 MiB |
|     0       2898     C      /usr/bin/krdpserver                                              23 MiB |
|     0       2901     C      /usr/bin/kaccess                                                 13 MiB |
|     0       3206     C      /usr/bin/kdeconnectd                                             23 MiB |
|     0       3362     C      /usr/lib/x86_64-linux-gnu/libexec/DiscoverNotifier               23 MiB |
|     0       3377     C      /usr/bin/kup-daemon                                              23 MiB |
|     0       3539     C      /usr/lib/x86_64-linux-gnu/libexec/xdg-desktop-p...               23 MiB |
|     0       3545     C      openrgb                                                          23 MiB |
|     0       3643     C      /app/lib/firefox/firefox                                       7692 MiB |
|     0       3785     C      /app/discord/Discord                                            408 MiB |
|     0       3885     C      /app/lib/firefox/firefox-bin -contentproc -pare...               45 MiB |
|     0       4089     C      /usr/bin/plasma-browser-integration-host                         23 MiB |
|     0      30005     C      /usr/bin/kwalletd6                                               23 MiB |
|     0      49966     C      /app/chromium/chrome --type=gpu-process --ozone...             1119 MiB |
|     0      61859     C      /usr/lib/x86_64-linux-gnu/libexec/baloorunner                    23 MiB |
|     0     176823     C      /usr/bin/dolphin                                                 23 MiB |
|     0     186527     C      /app/share/zotero/zotero-bin                                    729 MiB |
|     0     196808     C      /usr/bin/konsole                                                 23 MiB |
|     0     265680     C      qalculate-qt                                                     23 MiB |
|     0     266813     C      /usr/share/code/code --type=gpu-process --ozone...             1152 MiB |
+-------+----------+--------+----------------------------------------------------+--------------------+

While nvtop reports Firefox as using 1977MiB which seems much more reasonable.

Expected Behavior

I believe this is because xpu-smi adds all the values from the process fdinfo memory usage together, instead of only adding values with a unique drm-client-id.

stephen@Stephens-PC:~$ grep -E '^drm-client-id|^drm-total-vram0' /proc/3643/fdinfo/*
/proc/3643/fdinfo/170:drm-client-id:    294
/proc/3643/fdinfo/170:drm-total-vram0:  0
/proc/3643/fdinfo/176:drm-client-id:    296
/proc/3643/fdinfo/176:drm-total-vram0:  0
/proc/3643/fdinfo/179:drm-client-id:    305
/proc/3643/fdinfo/179:drm-total-vram0:  0
/proc/3643/fdinfo/184:drm-client-id:    297
/proc/3643/fdinfo/184:drm-total-vram0:  37036 KiB
/proc/3643/fdinfo/21:drm-client-id:     47
/proc/3643/fdinfo/21:drm-total-vram0:   1950572 KiB
/proc/3643/fdinfo/297:drm-client-id:    307
/proc/3643/fdinfo/297:drm-total-vram0:  0
/proc/3643/fdinfo/318:drm-client-id:    308
/proc/3643/fdinfo/318:drm-total-vram0:  37036 KiB
/proc/3643/fdinfo/50:drm-client-id:     47
/proc/3643/fdinfo/50:drm-total-vram0:   1950572 KiB
/proc/3643/fdinfo/51:drm-client-id:     47
/proc/3643/fdinfo/51:drm-total-vram0:   1950572 KiB
/proc/3643/fdinfo/52:drm-client-id:     47
/proc/3643/fdinfo/52:drm-total-vram0:   1950572 KiB
/proc/3643/fdinfo/53:drm-client-id:     51
/proc/3643/fdinfo/53:drm-total-vram0:   0
/proc/3643/fdinfo/54:drm-client-id:     51
/proc/3643/fdinfo/54:drm-total-vram0:   0
/proc/3643/fdinfo/55:drm-client-id:     51
/proc/3643/fdinfo/55:drm-total-vram0:   0

E.g. (1950572×4+37036+37036)/1024 ≈ 7692MiB matches xpu-smi, while the actual memory usage is (1950572+37036+37036)/1024 ≈ 1977MiB (matches nvtop).

I was also told by @savery42 in intel/xpumanager#145:

Great find! This is indeed an issue with the level-zero (Sysman) runtime implementation (no deduplication in getMemoryStatsUsedByProcess which is called in a opened GPU fd loop). The userspace API is zesDeviceProccessGetState.

Reproduction Rate

Frequently reproduces ~ 75% - 99%
Always reproduces - 100%

Steps to Reproduce

This probably isn't very useful, but my general usage of Firefox seems to reliably replicate this. There's likely a better way to trigger this.

@savery42 has also made a reproducer, linked below.

Is this a regression?

  • Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

No response

First Known Failing Driver Version

No response

API Call Logs

No response

strace Logs

No response

System Logs / dmesg Output

No response

Backtrace (if crash or hang occurred)

No response

Source Code / Reproducer

See process_memstress.cpp at the end of intel/xpumanager#145 (comment)

Command Line / Application Details

No response

oneAPI Version (if applicable)

No response

Screenshots / Video

No response

Additional Notes

First reported here: intel/xpumanager#145

Metadata

Metadata

Assignees

No one assigned

    Labels

    Component: Level Zero SysmanLevel Zero System Management API for hardware monitoringOS: LinuxIssue specific to Linux distributions (Ubuntu, Fedora, RHEL, etc.)Type: BugGeneral bug report, unexpected behavior or crash

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions