Skip to content

[Bug] Python 3.14 unit tests intermittently stall during pytest shutdown #935

Description

@RyAuld

Description

Python 3.14 unit tests intermittently stall after all tests pass (reaches 99-100% completion), hanging until the 5-minute step timeout kills the process. Other Python versions (3.9–3.13) consistently complete in ~1.5 minutes.

Observed Behavior

  • All tests pass successfully
  • The last output before the stall is typically a deprecation warning or the final test result
  • The process hangs during pytest session teardown/finalization
  • The stall is intermittent — sometimes Python 3.14 passes, sometimes it hangs

Builds Exhibiting This Issue

  • 1651220 — canceled at 30-min timeout
  • 1651218 — canceled at 30-min timeout
  • 1654753 — failed at 5-min timeout (after timeout was reduced)

Mitigations Already Applied (PR #934)

  • Replaced deprecated datetime.utcnow() with datetime.now(timezone.utc) — did not resolve the stall
  • Removed 2>&1 | tee pipe (replaced with --log-file) — did not resolve the stall
  • Added timeoutInMinutes: 5 on the pytest step to fail fast

Suspected Root Causes

Without agent-level access, we cannot definitively determine the cause. Likely candidates:

  1. pytest-azurepipelines plugin — may hang during post-test result upload on 3.14
  2. Python 3.14 interpreter shutdown — thread/GC finalization deadlock in pre-release runtime
  3. Dependencies compiled from source (no pre-built 3.14 wheels) — buggy atexit/cleanup handler
  4. Hosted agent resource contention — network timeout to ADO during result publishing

Suggested Next Steps

  • Try running with -p no:azurepipelines to rule out plugin issues
  • Add continueOnError: true for Python 3.14 matrix entry to unblock PRs
  • Consider removing Python 3.14 from the CI matrix until official release and ecosystem wheel availability
  • Revisit once Python 3.14 reaches stable release

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions