ESP32-S3 CSI crash: SPI flash cache race in wDev_ProcessFiq during promiscuous mode

[proffesor-for-testing]

Summary

ESP32-S3 CSI nodes crash with LoadProhibited in the WiFi driver's interrupt handler when promiscuous mode captures frames at high rates. The crash is inside Espressif's closed-source binary blob and cannot be fixed at the application level without reducing the WiFi hardware interrupt rate.

Crash signature

Guru Meditation Error: Core 0 panic'ed (LoadProhibited)
EXCVADDR: 0x00000004

Decoded backtrace (xtensa-esp32s3-elf-addr2line):

_xt_lowint1                    ← WiFi hardware interrupt
wDev_ProcessFiq                ← WiFi driver FIQ (closed-source blob)
spi_flash_restore_cache        ← SPI flash cache restore
cache_ll_l1_resume_icache      ← L1 ICache resume ← NULL deref here

Root cause

The WiFi MAC hardware generates a Level 1 interrupt for every frame captured by the promiscuous filter. wDev_ProcessFiq handles these interrupts and at some point calls spi_flash_restore_cache, which calls cache_ll_l1_resume_icache. When the interrupt rate is high enough (>50 Hz), this function encounters a NULL pointer — likely because the SPI flash cache state is in an inconsistent state from a concurrent flash operation (display QSPI, NVS write, etc.).

The crash is not in application code. It's inside the ESP-IDF WiFi binary blob (libpp.a).

Controlled experiments

All tests on ESP32-S3 (QFN56 rev v0.2, 8MB PSRAM, MAC 80:b5:4e:c1:be:b8), ESP-IDF v5.4, WiFi SSID Spiridonovi1 ch2.

#	Build	Display	Promiscuous filter	Effective CSI rate	Crash point	Result
1	Our build	ON	MGMT+DATA	~500 Hz	~2400 cb (~70s)	Crash
2	Our build	OFF	MGMT+DATA	~500 Hz	~5300 cb (~90s)	Crash (slower)
3	Our build	OFF	MGMT-only	~10 Hz	2700+ cb (4.7 min)	Stable
4	Our build	ON	MGMT-only	~10 Hz	2400+ cb (4 min+)	Stable
5	Our build + 50Hz callback gate	ON	MGMT+DATA	~50 Hz	~1300 cb	Crash
6	Ruv's v0.6.1-esp32 release	OFF	MGMT+DATA	~100 Hz	~8200 cb	Crash (19x in 2 min)

Key findings:

• Display OFF doubles time-to-crash but doesn't prevent it (test 2 vs 1, test 6)
• Callback-level rate limiting does NOT help because the WiFi HW interrupt fires for every captured frame regardless of callback execution (test 5)
• MGMT-only filter is the only fix that works — it reduces the hardware interrupt rate itself (test 3, 4)
• v0.6.1-esp32 release crashes with the same bug (test 6)

What doesn't work

Callback rate limiting

A 50 Hz early gate in wifi_csi_callback that returns immediately for excess frames does not help. The crash occurs in wDev_ProcessFiq which runs before the callback is invoked. Reducing callback execution time has no effect on interrupt rate.

SPIRAM XIP (CONFIG_SPIRAM_FETCH_INSTRUCTIONS + CONFIG_SPIRAM_RODATA)

In theory this eliminates the SPI flash cache race entirely (instructions served from PSRAM, cache never suspended). In practice, manual sdkconfig edits with CONFIG_SPIRAM_MODE_QUAD=y produced an IllegalInstruction crash-loop from boot — likely because this board's PSRAM is Octal, not Quad. Needs proper idf.py menuconfig validation with the correct PSRAM mode for this hardware.

Additional IRAM options

CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y and CONFIG_ESP_WIFI_SLP_IRAM_OPT=y were tested as part of the SPIRAM build but couldn't be isolated due to the PSRAM misconfiguration.

What works

MGMT-only promiscuous filter

wifi_promiscuous_filter_t filt = {
    .filter_mask = WIFI_PROMIS_FILTER_MASK_MGMT,  // was MGMT | DATA
};

Reduces WiFi hardware interrupt rate from ~100-500 Hz to ~10 Hz (beacon/probe frames only). Tested stable for 4+ minutes with display ON, zero crashes.

Trade-off: CSI data rate drops from ~100-500 frames/sec to ~10 frames/sec. However, 10 Hz is sufficient for presence detection, breathing rate (10-30 BPM), and heart rate detection. Edge processing adaptive calibration completes successfully at this rate.

Proper fix path (not yet tested)

SPIRAM XIP is the correct platform-level fix. When CONFIG_SPIRAM_FETCH_INSTRUCTIONS=y + CONFIG_SPIRAM_RODATA=y, the SPI flash cache is never suspended during flash operations, eliminating the race entirely. This requires:

1. Determine correct PSRAM mode for this board (Quad vs Octal) — check Waveshare ESP32-S3 datasheet
2. Configure via idf.py menuconfig (not manual sdkconfig edits) to get all dependencies right
3. Test with full MGMT+DATA promiscuous at 500 Hz

Also affected: node_id clobber

Separately from the crash, the g_nvs_config.node_id clobber (#390) is confirmed on our hardware. Ruv's v0.6.1 late capture at csi_collector_init() works on some boots but not all — we proved wifi_init_sta() corrupts the struct before the capture runs. Our early capture (csi_collector_set_node_id() called before wifi_init_sta()) is the reliable fix. See PR #393 comments.

Hardware

• Board: Waveshare ESP32-S3 AMOLED 1.8" (SH8601 368x448 QSPI display)
• Chip: ESP32-S3 (QFN56) rev v0.2, 8MB PSRAM (AP_3v3), 16MB flash (Boya)
• ESP-IDF: v5.4
• WiFi: Spiridonovi1 ch2, WPA2-PSK

Refs

• fix(firmware): defensive node_id capture prevents runtime clobber (#390) #393 (node_id fix, merged without our improvements)
• ruvnet/RuView v0.6.1-esp32 release (crash confirmed)

[proffesor-for-testing]

Update: SPIRAM Octal XIP tested — changes crash type but doesn't fix it

PSRAM detection (confirmed working)

octal_psram: vendor id    : 0x0d (AP)
octal_psram: dev id       : 0x02 (generation 3)
octal_psram: density      : 0x03 (64 Mbit)
esp_psram: Found 8MB PSRAM device
esp_psram: Speed: 80MHz

Board is ESP32-S3 N16R8 (16MB flash Quad + 8MB PSRAM Octal). Previous attempt failed because we used Quad mode — Octal is correct.

sdkconfig.defaults additions

CONFIG_SPIRAM=y
CONFIG_SPIRAM_MODE_OCT=y
CONFIG_SPIRAM_SPEED_80M=y
CONFIG_SPIRAM_FETCH_INSTRUCTIONS=y
CONFIG_SPIRAM_RODATA=y
CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y

Result: different crash, not fixed

	Before XIP	With XIP
Crash type	LoadProhibited (NULL deref in cache resume)	Cache disabled but cached memory region accessed
EXCCAUSE	0x1c	0x07
Crash point	~1500-8200 cb depending on display	~1500 cb
Crashes in 2.5 min	19	16

XIP eliminated the original NULL pointer crash (SPIRAM is serving instructions correctly) but revealed a new one: something is still disabling the cache, and the WiFi ISR then tries to access cached memory (PSRAM or flash-mapped) while it's off.

ESP-IDF docs claim "cache won't be disabled during SPI1 Flash operations" with FETCH_INSTRUCTIONS + RODATA, but the crash proves otherwise. Possible causes:

• The WiFi blob (libpp.a) internally disables cache for its own flash operations
• NVS writes trigger cache disable that isn't covered by the XIP mechanism
• OTA HTTP server or other subsystem doing flash I/O

Current status

MGMT-only promiscuous filter remains the only working fix. All other approaches crash:

Approach	Result
MGMT+DATA (default)	Crash at ~2400 cb
Display disabled	Crash at ~5300 cb
50 Hz callback rate limiter	Crash at ~1300 cb
SPIRAM Octal XIP	Crash at ~1500 cb (different type)
MGMT-only filter	Stable indefinitely

Next steps

1. Deploy MGMT-only filter to all 3 ESP32 nodes (working solution)
2. Investigate what's disabling the cache during XIP — may need ESP-IDF debug builds or Espressif support
3. 10 Hz from MGMT beacons is sufficient for presence/breathing/heartrate but limits resolution for fall detection and fine-grained motion

[proffesor-for-testing]

Final solution: MGMT-only promiscuous filter at ~10 Hz

After exhaustive testing, MGMT-only is the only stable configuration for CSI on this hardware. All attempts to increase the rate beyond ~10 Hz trigger the SPI flash cache crash.

Full test matrix (updated)

#	Build	Display	Filter	Rate control	Effective rate	Result
1	MGMT+DATA	ON	MGMT+DATA	None	~500 Hz	Crash at ~2400 cb
2	MGMT+DATA	OFF	MGMT+DATA	None	~500 Hz	Crash at ~5300 cb
3	MGMT-only	OFF	MGMT-only	None	~10 Hz	Stable 4+ min
4	MGMT-only	ON	MGMT-only	None	~10 Hz	Stable 4+ min
5	MGMT+DATA + gate	ON	MGMT+DATA	50 Hz callback gate	~50 Hz	Crash at ~1300 cb
6	Ruv v0.6.1	OFF	MGMT+DATA	None	~100 Hz	Crash 19x in 2 min
7	SPIRAM Octal XIP	ON	MGMT+DATA	None	~500 Hz	Crash (different type)
8	Null-data injection	ON	MGMT-only	10 Hz TX timer	~20 Hz	Crash (TX adds interrupt pressure)
9	Probe request injection	ON	MGMT-only	10 Hz broadcast probe	~10 Hz + channel spam	Crash (channel toggling)
10	MGMT-only (final)	ON	MGMT-only	None	~10 Hz	Stable — deployed

Why higher rates fail

The crash is in wDev_ProcessFiq (ESP-IDF closed-source WiFi blob). Every WiFi frame captured by the promiscuous filter — and every frame we transmit — generates a hardware interrupt. At rates above ~10-20 Hz, the interrupt handler races with SPI flash cache operations.

Callback-level rate limiting doesn't help (test 5) because the hardware interrupt fires before our callback. Frame injection doesn't help (tests 8, 9) because TX generates additional interrupts. SPIRAM XIP changes the crash type (test 7) but doesn't prevent it — something beyond SPI1 flash operations still disables the cache.

Why ~10 Hz is acceptable

The edge processing pipeline (edge_processing.c:718) has sample_rate = 20.0f, so 10 Hz input is below design spec. However:

• Nyquist minimum for heart rate (2.0 Hz) = 4 Hz — met
• Nyquist minimum for breathing (0.5 Hz) = 1 Hz — met
• Cogs on seeds (health-monitor, breathing-sync, cardiac-arrhythmia) all use sample_rate = 10.0 — matched exactly
• Edge processing produces valid vitals at 10 Hz: br=25-36 BPM, hr=76-99 BPM, presence=YES
• Adaptive calibration completes successfully at 1200 frames

The sample_rate constant in edge_processing.c should be updated from 20.0 to 10.0 to match the actual input rate. This affects bandpass filter coefficients and BPM estimation accuracy.

Deployed firmware

All 3 ESP32-S3 nodes flashed with MGMT-only firmware and verified stable:

ESP32	node_id	Target Seed	Frames delivered	Status
#1 (c1:be:b8)	3	Seed C (192.168.1.95)	310K+	Stable, vitals active
#2 (c1:b5:68)	2	Seed B (192.168.1.117)	1.04M+	Stable
#3 (c1:c4:f0)	1	Seed A (192.168.1.105)	1.44M+	Stable

Code changes (on branch fix/esp32-node-id-clobber in local clone)

1. csi_collector.c: MGMT-only promiscuous filter with documentation (RuView#396)
2. csi_collector.c: Early node_id capture before wifi_init_sta() (ESP32-S3: g_nvs_config.node_id clobbered to 1 between main.c:140 and csi_collector_init + LoadProhibited panic loop #390)
3. csi_collector.c: Defensive filter_mac copy in callback
4. csi_collector.c: 50 Hz early rate gate (defense-in-depth, doesn't fix crash alone)
5. csi_collector.c: Null-data injection timer (disabled — adds interrupt pressure)
6. provision.py: write_flash esptool v5 compat fix
7. sdkconfig.defaults: CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y

Path to higher CSI rates (future work)

The only viable path to >10 Hz requires fixing the SPI flash cache race at the platform level:

1. Espressif support: File a bug with Espressif about wDev_ProcessFiq crashing in cache_ll_l1_resume_icache during promiscuous mode on ESP32-S3 with QSPI display
2. ESP-IDF upgrade: Test with ESP-IDF v5.5+ when available — Espressif may have fixed the cache handling
3. SPIRAM XIP with proper config: Our Octal PSRAM XIP test changed the crash type (from LoadProhibited to "Cache disabled but cached memory region accessed"), suggesting the XIP mechanism is partially working. A properly tuned XIP config — possibly with CONFIG_SPI_FLASH_AUTO_SUSPEND=n and additional cache tuning — might eliminate both crash types
4. Board without QSPI display: The Waveshare ESP32-S3 AMOLED shares the SPI bus between flash and display. A board with a parallel (non-SPI) display or no display would reduce SPI contention