Flaky test: test_snapshot_pruning_removes_outdated_records fails with assert 3 == 2

[vdusek]

Description

test_snapshot_pruning_removes_outdated_records in tests/unit/_autoscaling/test_snapshotter.py is consistently failing across multiple CI runs on Ubuntu (Python 3.13 and 3.14), even when running in isolation (run_alone marker with numprocesses=1).

Failure

FAILED tests/unit/_autoscaling/test_snapshotter.py::test_snapshot_pruning_removes_outdated_records - assert 3 == 2

The test expects that after emitting 4 events with timestamps [now-5h, now-3h, now-2h, now] and a _SNAPSHOT_HISTORY of 2 hours, only 2 snapshots should remain (the ones within the 2-hour window). However, 3 snapshots are retained instead of 2.

Root cause hypothesis

The snapshot pruning logic likely has an off-by-one or boundary condition issue when determining which records fall outside the history window. The now-2h timestamp sits exactly at the boundary of the 2-hour history window, and depending on timing/precision of datetime.now() between the test setup and the pruning execution, it may or may not be pruned.

Observed in

• CI run on ubuntu-latest with Python 3.13 and 3.14 (PR test: Mark flaky tests with run_alone to reduce CI failures #1733)
• CI run on ubuntu-latest with Python 3.13 (commit 9d3da48)
• Also seen as test_get_cpu_sample failure (same file, similar pattern) on Windows with Python 3.13 (commit df1347a)

Steps to reproduce

Run the test repeatedly:

pytest tests/unit/_autoscaling/test_snapshotter.py::test_snapshot_pruning_removes_outdated_records -x --count=50

[Pijukatel]

I think that the problem is probably in some race condition in sorting/pruning snapshots. Where both are modifying the list in place and most likely are running at the same time.