Skip to content

fix(httpserver): release req_status.lock before yielding streamed tokens - #1739

Merged
sufubao merged 1 commit into
ModelTC:mainfrom
sufubao:fix/httpserver-lock-across-yield
Nov 17, 2026
Merged

sufubao merged 1 commit into
ModelTC:mainfrom
sufubao:fix/httpserver-lock-across-yield

Conversation

@sufubao

@sufubao sufubao commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Problem

HttpServerManager._wait_to_token_package yielded each token inside async with req_status.lock. With stream=True, a client that reads slowly (or a proxy that stops pulling) leaves the async generator suspended at the yield while it still holds the lock.

handle_loop takes the same lock for every request in turn (req_status.out_token_info_list.extend(...)), so a single stalled client blocks token delivery for all requests:

  • tokens are no longer moved from shared memory into the HTTP-side buffers
  • out_tokens_queue stops draining, so detokenization never sets can_released_mark
  • can_release() stays False (ref_count stuck at 2), so recycle_resource_loop never frees the request: a request/memory leak

Cancellation or an abandoned generator has the same effect: the lock is only released when the generator is eventually closed or garbage-collected.

Fix

Under the lock, swap out the pending list and clear the event, then iterate and yield outside the lock:

async with req_status.lock:
    event.clear()
    token_batch = req_status.out_token_info_list
    req_status.out_token_info_list = []

for ... in token_batch:
    ...
    yield ...

handle_loop either extends the old list before the swap (so its tokens are in token_batch) or the new list after it (and sets the event, so the next iteration picks them up). No token is lost or duplicated. The empty-list continue and the trailing clear() are no longer needed. The rest of the change is re-indentation; git diff -w shows the real change.

Tests

  • New regression test test_wait_to_token_package_releases_lock_while_consumer_is_suspended: suspends the consumer after the first token, asserts the lock is free, has a producer append under the lock, and checks token order a, b, c. It fails on current main (producer times out waiting for the lock) and passes with this change.
  • unit_tests/server/httpserver, unit_tests/server/core/objs/test_stop_sequence_output.py, unit_tests/server/test_pd_local_selection.py: 552 passed.
  • black / flake8 at the versions pinned in .pre-commit-config.yaml.
  • Not run: end-to-end serving with a real model (no weights on the dev machine).

🤖 Generated with Claude Code

_wait_to_token_package yielded tokens inside `async with req_status.lock`.
When a streaming client read slowly (or a proxy stopped pulling), the
generator stayed suspended at the yield while holding the lock. handle_loop
takes the same lock for every request, so one stalled client blocked token
delivery for all requests: out_tokens_queue stopped draining, detokenization
never set can_released_mark, and requests were never recycled.

Swap the pending list out under the lock and yield outside it. handle_loop
either extends the old list before the swap or the new list after it (and
sets the event), so no token is lost or duplicated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sufubao
sufubao merged commit 0d57783 into ModelTC:main Nov 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant