Description
Python 3.14 unit tests intermittently stall after all tests pass (reaches 99-100% completion), hanging until the 5-minute step timeout kills the process. Other Python versions (3.9–3.13) consistently complete in ~1.5 minutes.
Observed Behavior
- All tests pass successfully
- The last output before the stall is typically a deprecation warning or the final test result
- The process hangs during pytest session teardown/finalization
- The stall is intermittent — sometimes Python 3.14 passes, sometimes it hangs
Builds Exhibiting This Issue
- 1651220 — canceled at 30-min timeout
- 1651218 — canceled at 30-min timeout
- 1654753 — failed at 5-min timeout (after timeout was reduced)
Mitigations Already Applied (PR #934)
- Replaced deprecated
datetime.utcnow() with datetime.now(timezone.utc) — did not resolve the stall
- Removed
2>&1 | tee pipe (replaced with --log-file) — did not resolve the stall
- Added
timeoutInMinutes: 5 on the pytest step to fail fast
Suspected Root Causes
Without agent-level access, we cannot definitively determine the cause. Likely candidates:
- pytest-azurepipelines plugin — may hang during post-test result upload on 3.14
- Python 3.14 interpreter shutdown — thread/GC finalization deadlock in pre-release runtime
- Dependencies compiled from source (no pre-built 3.14 wheels) — buggy atexit/cleanup handler
- Hosted agent resource contention — network timeout to ADO during result publishing
Suggested Next Steps
- Try running with
-p no:azurepipelines to rule out plugin issues
- Add
continueOnError: true for Python 3.14 matrix entry to unblock PRs
- Consider removing Python 3.14 from the CI matrix until official release and ecosystem wheel availability
- Revisit once Python 3.14 reaches stable release
Description
Python 3.14 unit tests intermittently stall after all tests pass (reaches 99-100% completion), hanging until the 5-minute step timeout kills the process. Other Python versions (3.9–3.13) consistently complete in ~1.5 minutes.
Observed Behavior
Builds Exhibiting This Issue
Mitigations Already Applied (PR #934)
datetime.utcnow()withdatetime.now(timezone.utc)— did not resolve the stall2>&1 | teepipe (replaced with--log-file) — did not resolve the stalltimeoutInMinutes: 5on the pytest step to fail fastSuspected Root Causes
Without agent-level access, we cannot definitively determine the cause. Likely candidates:
Suggested Next Steps
-p no:azurepipelinesto rule out plugin issuescontinueOnError: truefor Python 3.14 matrix entry to unblock PRs