Add ESXi support to platform in-band collectors by spiral6 · Pull Request #282 · amd/node-scraper

spiral6 · 2026-09-10T23:39:28Z

Enable the platform in-band collectors on ESXi via esxcli/smbiosDump, plus pcie and dmesg:

os, bios, dimm, kernel, storage, device_enumeration — esxcli/smbiosDump branches. kernel reuses the uname -a path (ESXi reports the release in the same field); dimm shares a new _parse_dmi_sizes helper across dmidecode (Linux) and smbiosDump (ESXi); device_enumeration counts GPU PF/VF by device ID, adding devid_ep/devid_ep_vf to SystemInfo.
pcie — ESXi has no lspci device filter, so GPU/VF BDFs are resolved from esxcli hardware pci list by matching the SKU's PF/VF device IDs, and extended config space is sliced from lspci -e. (Uses the SystemInfo.devid_ep fields above.)
dmesg collector — ESXi has no dmesg ring buffer; read /var/log/vmkernel.log (+ vmkernel.[.gz] rotations) instead of dmesg.
dmesg analyzer — format-aware for ESXi: ISO8601-Z timestamps, per-flavor severity from the driver-internal message body (ESXi -ALERT/-INFO tokens are unreliable), ESXi mxGPU RAS ERROR_REGEX entries, and unknown-error detection.

Validated on ESXi 9.1 and Linux; Linux/Windows paths unchanged.
pre-commit (black/ruff/mypy) green; full unit suite passes.

Enable os, bios, dimm, kernel, storage, and device_enumeration collectors on ESXi via esxcli/smbiosDump. kernel reuses the existing `uname -a` path (ESXi reports the release in the same field). dimm shares a _parse_dmi_sizes helper across dmidecode (Linux) and smbiosDump (ESXi). device_enumeration counts GPU PF/VF by device ID, adding devid_ep/devid_ep_vf to SystemInfo. Validated on ESXi 9.1.0 and Linux; Linux/Windows paths unchanged.

ESXi busybox lspci lacks per-device (-s) and bus-path (-PP) options, so dump all extended config space once via `lspci -e`, split by BDF, and select the GPU/VF BDFs resolved from `esxcli hardware pci list` by SKU device ID (system_info.devid_ep/_vf). Extract a shared _cfg_space_from_hex parser (used by the Linux per-BDF path too). Upstream-bridge traversal is skipped on ESXi (GPU + VF only). Depends on the SystemInfo devid_ep/devid_ep_vf fields.

ESXi has no dmesg ring buffer; read the kernel log from /var/log/vmkernel.log (and vmkernel.<n>[.gz] rotations) instead of `dmesg`. Validated on ESXi: reads vmkernel.log; Linux path unchanged.

Make the dmesg analyzer format-aware so it handles ESXi vmkernel.log as well as Linux dmesg: - Extract ESXi ISO8601 dot-ms/Z timestamps (e.g. 2026-08-20T09:35:58.380Z) via ESXI_TIMESTAMP_PATTERN, set on __init__ so event grouping and date-range filtering both use it; Linux keeps the base comma-form pattern. - filter_dmesg is now an instance method and reuses the base timestamp extractor, so a single code path honors whichever pattern is active. - Add ESXi mxGPU (gim/amdgpuv) RAS ERROR_REGEX entries (Block-capitalized correctable/uncorrectable, ECC Fatal Error, Whole GPU reset); these are inert on Linux logs. - Unknown-error detection keys off the driver-internal severity in the message body ("gim/amdgpuv error/warn") on ESXi, where the vmkernel -ALERT/-INFO tokens are unreliable; Linux keeps the "kern :err:" form. Validated on real ESXi vmkernel.log (7.2 MB) and Linux dmesg (no regression), plus synthetic RAS lines confirming per-OS phrasing is discriminated correctly.

- filter_dmesg: keep it a classmethod (public API used as DmesgAnalyzer.filter_dmesg(content, ...) in tests). Recognize both Linux comma-form and ESXi dot-ms/Z timestamps via a combined pattern instead of the instance TIMESTAMP_PATTERN; normalize the trailing Z so fromisoformat accepts it on Python < 3.11. - Collector: restore the exact Linux "No /var/log/dmesg files found (including rotations)." wording for the no-rotations event and add an ESXi-specific vmkernel.log variant, rather than a generic reword. Restores test_dmesg_filter and test_collect_rotations_no_files; full dmesg collector+analyzer suite (57 tests) green.

Pre-commit black (line-length 100) flagged formatting in the ESXi branches. Expand the branch-selection ternaries to explicit if/else (pcie devid resolve, storage percent, dmesg cmd/log-label selection) and let black normalize the two long single-line constants (dmesg CMD_LOGS_ESXI, analyzer ESXI_TIMESTAMP_PATTERN). No behavior change; black --check and ruff clean, dmesg+storage suites green.

Hoisting the percent computation reused the name "percent", which the Linux branch later binds to a str from split() before re.sub()/float() — mypy flagged the float-vs-str conflict. Rename the ESXi-branch value to usage_percent.

alexandraBara · 2026-09-14T14:20:31Z

+    devid_ep: Optional[int] = None
+    devid_ep_vf: Optional[int] = None


where do these get populated? These seem to be used later on in the plugin but they will always be None. Notice we have these in amd-smi as well but they are being passed in as analyzer_args where the user gets to populate them. That is something that we can do for the plugins that use them as well.

This is for tackling the guest VM recognition (that's why it's using VF); needs both dev and vendor ID for PCIe devices to identify properly.

I'll take another look; it's possible these aren't getting populated by Node Scraper and are by our internal repo.

Good call — you're right that these were dead upstream: nothing populated them, so the ESXi GPU/VF resolution in device_enumeration and pcie was always inert.

Moved them to per-collector args as you suggested (053b734). Dropped the now-unused devid_ep / devid_ep_vf fields from SystemInfo.

alexandraBara · 2026-09-14T14:23:48Z

+    CMD_CPU_COUNT_ESXI = "esxcli hardware cpu global get | awk '/CPU Packages:/ {print $NF}'"
+    CMD_PCI_COUNT_ESXI = "esxcli hardware pci list | grep -E '^ *Device ID: 0x{device_id}' | wc -l"


this needs to also match: format(0x744c, "x") → 744c. That misses Device ID: 0x744C and Device ID: 0x0000744c.

the awk isnt a very safe cmd here, if it returns something more than an int we will fail on this later on: " device_enum.cpu_count = int(cpu_count_res.stdout)" <-- line 151 ?

Fixed and covered in 545661e

alexandraBara · 2026-09-14T14:25:46Z

+            elif current_bdf and stripped.lower().startswith("device id:"):
+                devid = stripped.split(":", 1)[1].strip().lower().removeprefix("0x")


I believe this will miss padded 0000744c vs 744c

Fixed and covered in 545661e

Per review feedback on the ESXi device-ID handling: - pcie: compare the esxcli "Device ID" to the expected PF/VF id by integer value instead of an exact lowercase-string match, so uppercase ("0x744C") and zero-padded ("0x0000744c") ids are matched. - device_enumeration: make the PCI-count grep case-insensitive and zero-pad tolerant ("0x0*"), with a trailing [^0-9a-f]/$ guard so a shorter id does not match a longer one (744c vs 744cd). - device_enumeration: parse the CPU/GPU/VF counts defensively (guard non-zero exit and non-numeric stdout) instead of int()-ing command output directly, so an unexpected esxcli/awk result warns rather than raising. Validated on ESXi 9.1 (8 GPUs matched; counts parsed) and Linux (no regression); padded/uppercase ids confirmed against busybox grep and the int compare.

The expected GPU PF/VF PCI device IDs were SystemInfo fields that nothing populated upstream, so the ESXi GPU/VF resolution in device_enumeration and pcie was always inert. Move them to per-collector args (DeviceEnumerationCollectorArgs / PcieCollectorArgs), user-populated, matching how amd-smi takes them via args; read from args instead of SystemInfo and drop the unused SystemInfo fields. Validated on ESXi (8 GPU PF BDFs / gpu_count 8 via args) and Linux (no regression).

github-actions Bot added plugins-inband framework labels Sep 10, 2026

spiral6 added 6 commits September 11, 2026 03:58

Add ESXi support to DmesgCollector

5bd26fe

ESXi has no dmesg ring buffer; read the kernel log from /var/log/vmkernel.log (and vmkernel.<n>[.gz] rotations) instead of `dmesg`. Validated on ESXi: reads vmkernel.log; Linux path unchanged.

Fix mypy name collision in storage ESXi branch

6e9d3d2

Hoisting the percent computation reused the name "percent", which the Linux branch later binds to a str from split() before re.sub()/float() — mypy flagged the float-vs-str conflict. Rename the ESXi-branch value to usage_percent.

spiral6 marked this pull request as ready for review September 11, 2026 18:27

spiral6 requested review from alexandraBara and graepaul September 11, 2026 18:28

alexandraBara requested changes Sep 14, 2026

View reviewed changes

spiral6 added 2 commits September 14, 2026 18:26

github-actions Bot removed the framework label Sep 15, 2026

Navigation Menu

Uh oh!

Add ESXi support to platform in-band collectors - #282

Add ESXi support to platform in-band collectors#282
spiral6 wants to merge 9 commits into
developmentfrom
esxi-platform-collectors

spiral6 commented Sep 10, 2026 •

edited

Loading

Uh oh!

alexandraBara Sep 14, 2026

Uh oh!

spiral6 Sep 14, 2026

Uh oh!

spiral6 Sep 14, 2026

Uh oh!

spiral6 Sep 15, 2026

Uh oh!

alexandraBara Sep 14, 2026

Uh oh!

alexandraBara Sep 14, 2026

Uh oh!

spiral6 Sep 14, 2026

Uh oh!

alexandraBara Sep 14, 2026

Uh oh!

spiral6 Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		devid_ep: Optional[int] = None
		devid_ep_vf: Optional[int] = None

		CMD_CPU_COUNT_ESXI = "esxcli hardware cpu global get \| awk '/CPU Packages:/ {print $NF}'"
		CMD_PCI_COUNT_ESXI = "esxcli hardware pci list \| grep -E '^ *Device ID: 0x{device_id}' \| wc -l"

		elif current_bdf and stripped.lower().startswith("device id:"):
		devid = stripped.split(":", 1)[1].strip().lower().removeprefix("0x")

Uh oh!

Conversation

spiral6 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

spiral6 commented Sep 10, 2026 •

edited

Loading