Repository navigation
Diagnose slow worker shutdown in warning test - #2098
Draft
VegetarianOrc wants to merge 1 commit into
Draft
VegetarianOrc wants to merge 1 commit into
VegetarianOrc wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Keep the original worker lifecycle in
test_warns_when_workers_too_low, but log the task queue and pending asyncio task stacks at 5, 25, and 55 seconds when shutdown is slow. The warning test has intermittently timed out at 60 seconds in CI without identifying which operation remained pending. This draft is for collecting CI evidence, not for changing worker behavior or landing a fix.Local tracing of the unmodified test saw one activity poll remain pending for 53.947 seconds while the workflow poll finished in 0.002 seconds; the first worker finished in 58.3 seconds. Core relies on server-driven poll completion when the server advertises that capability. The new diagnostics can identify the pending Python/bridge task and correlate the task queue with Core/server logs, but cannot alone determine the
ShutdownWorkerRPC result.Validation
poe test -s -k 'test_warns_when_workers_too_low or test_can_run_resource_based_worker or test_can_run_composite_tuner_worker' tests/worker/test_worker.py: 3 passed.uv run pyright tests/worker/test_worker.py, Ruff import/format checks, andgit diff --check: passed.No broader test suite or full lint was run for this diagnostic-only change.