Repository navigation
fix(httpserver): release req_status.lock before yielding streamed tokens - #1739
Merged
Merged
Conversation
_wait_to_token_package yielded tokens inside `async with req_status.lock`. When a streaming client read slowly (or a proxy stopped pulling), the generator stayed suspended at the yield while holding the lock. handle_loop takes the same lock for every request, so one stalled client blocked token delivery for all requests: out_tokens_queue stopped draining, detokenization never set can_released_mark, and requests were never recycled. Swap the pending list out under the lock and yield outside it. handle_loop either extends the old list before the swap or the new list after it (and sets the event), so no token is lost or duplicated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
HttpServerManager._wait_to_token_packageyielded each token insideasync with req_status.lock. Withstream=True, a client that reads slowly (or a proxy that stops pulling) leaves the async generator suspended at theyieldwhile it still holds the lock.handle_looptakes the same lock for every request in turn (req_status.out_token_info_list.extend(...)), so a single stalled client blocks token delivery for all requests:out_tokens_queuestops draining, so detokenization never setscan_released_markcan_release()staysFalse(ref_count stuck at 2), sorecycle_resource_loopnever frees the request: a request/memory leakCancellation or an abandoned generator has the same effect: the lock is only released when the generator is eventually closed or garbage-collected.
Fix
Under the lock, swap out the pending list and clear the event, then iterate and
yieldoutside the lock:handle_loopeither extends the old list before the swap (so its tokens are intoken_batch) or the new list after it (and sets the event, so the next iteration picks them up). No token is lost or duplicated. The empty-listcontinueand the trailingclear()are no longer needed. The rest of the change is re-indentation;git diff -wshows the real change.Tests
test_wait_to_token_package_releases_lock_while_consumer_is_suspended: suspends the consumer after the first token, asserts the lock is free, has a producer append under the lock, and checks token ordera, b, c. It fails on currentmain(producer times out waiting for the lock) and passes with this change.unit_tests/server/httpserver,unit_tests/server/core/objs/test_stop_sequence_output.py,unit_tests/server/test_pd_local_selection.py: 552 passed..pre-commit-config.yaml.🤖 Generated with Claude Code