Skip to content

fix(ENG-RELEASE-WINDOWS): the api-server gate can report its own failure again - #1159

Merged
localai-bot merged 2 commits into
mainfrom
row/ENG-RELEASE-WINDOWS-test-thread-raii
Aug 18, 2026
Merged

fix(ENG-RELEASE-WINDOWS): the api-server gate can report its own failure again#1159
localai-bot merged 2 commits into
mainfrom
row/ENG-RELEASE-WINDOWS-test-thread-raii

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Issue: #584 (stays OPEN — see the last section). Row ENG-RELEASE-WINDOWS.
Spec: .agents/specs/windows-test-thread-raii.md.

Both Windows lanes have printed nothing but the doctest version banner since
test_openai_api_server.exe started running: no Status: line, no
assertions: line, exit -1073740791 (0xC0000409). That is not extra
information about the crash — it is the crash's own reporting failure, and
until it is closed nobody can read what fails.

0xC0000409 is the status __fastfail raises for every fail-fast code, so
abort() — and therefore std::terminate() — reaches it, and __fastfail
bypasses SEH by design, so doctest's Windows handler never runs and its buffered
stdout is discarded unflushed.

tests/vllm/entrypoints/openai/test_api_server.cpp supplies that
std::terminate() at 17 places. Every socket case runs the server on a
background thread and then asserts against it, and a bare std::thread held
across those assertions ends the process by two separate paths:

  • ~thread on a joinable thread calls std::terminate ([thread.thread.destr]).
    A failing REQUIRE, or a bare json::parse on an unexpected body, unwinds
    past the thread object.
  • An exception escaping a thread's initial function is std::terminate too
    ([except.handle]/9), so a throw inside serve() does the same from the other
    side. Nothing in the file caught that.

So a named assertion failure arrives in CI as an opaque exit code. This
change closes both paths for that file.

What changed

Two types at the top of the file. ScopedThread joins in its destructor on
every path, runs the body inside a catch-all, and rethrows the escaped exception
from join(), which is a synchronisation point — at a place where doctest can
translate and name it. Its destructor never rethrows, because throwing while
unwinding is the failure it exists to prevent.

ScopedServerThread is that for the shape which dominates the file: it serves an
ApiServer and owns the stop() as well as the join, so a case that throws
before reaching its stop line still ends.

Converted: 14 server threads, the 6-client vector in the concurrency case, and
the two console-handler threads in the _WIN32-only teardown case — 17 in all,
which is every std::thread in the file. Every one of them holds an assertion
that can throw before its join.

Two details are load-bearing rather than incidental:

  • The stop action waits for the accept loop first.
    httplib::Server::stop() is a no-op while is_running_ is false
    (third_party/httplib/httplib.h:11460) and listen_internal raises that flag
    only once it is inside the loop (:12027). A joiner that ignored this would
    turn a 0.79 s fast-fail into a 180-minute CI timeout, which is a worse
    instrument, not a better one.
  • The explicit h.server.stop() lines are removed, so the scoped type owns
    the stop exclusively. A second stop() is not a no-op: it sees is_running_
    still true while the accept loop unwinds and svr_sock_ already exchanged to
    INVALID_SOCKET, which trips assert(svr_sock_ != INVALID_SOCKET) at
    httplib.h:11462 on every build that is not NDEBUG — which is this suite's
    own Linux build.

The 6 client threads also moved below the two vectors their bodies write into,
so the joining destructor now runs before those vectors are destroyed. The old
order was safe only because a joinable std::thread ended the process instead
of unwinding.

Evidence

Host Linux, GCC, cmake -S . -B build-584 -DVLLM_CPP_BUILD_TESTS=ON — the CI
build-test-cpu configuration, so NDEBUG is NOT defined and httplib's
assert is live. One build directory for every arm, target
test_openai_api_server, -j 4, zero compiler warnings. Base
affc2a7fdfaa1a75c6c2b8bacd2e79b2990446f7.

Case count, stated and non-zero rather than "it passed". Identical on both
arms, so the conversion added no case and removed none:

arm run
before test cases: 62 | 62 passed | 0 failed | 0 skipped, assertions: 733 | 733 passed | 0 failed, Status: SUCCESS!
after test cases: 62 | 62 passed | 0 failed | 0 skipped, assertions: 733 | 733 passed | 0 failed, Status: SUCCESS!

Red-first mutation. One assertion in the socket-smoke case is made to fail
while the server thread is still joinable: CHECK(res->status == 200) on
/health becomes REQUIRE(res->status == 999). Each arm compiled with rc 0 and
git diff --stat showed exactly one changed line, so neither reading is a build
failure or an unapplied edit wearing a pass.

arm exit what the run printed
before 134 (SIGABRT) terminate called without an active exception, then test case CRASHED: SIGABRT - Abort (abnormal termination) signal
after 1 the named failure and nothing else: FATAL ERROR: REQUIRE( res->status == 999 ) is NOT correct! values: REQUIRE( 200 == 999 )

terminate called without an active exception names the mechanism exactly: it
is ~thread on a joinable thread, not an escaped exception.

Why a decade of green Linux runs is not evidence against this. The before
arm still printed its assertion, because SIGABRT is catchable and doctest's
POSIX handler reports it and flushes. __fastfail is neither, so the same
std::terminate prints nothing on Windows. The defect is platform-independent;
only its reportability is not.

Records: scripts/agent-preflight.sh rc 0, check-agent-record.py
agent record OK: ENGINE=157 MODEL=377 QUANT=82 KERNEL=51 BACKEND=85,
check-issue-index-append-only.py OK: issue index append-only,
check-pr-size.py OK: every explicit path class is within its review budget.,
check-commit-style.py and check-commit-trailers.py OK over an explicit
two-commit range, and agent-ready.py All gates green.

What this does and does not establish

Establishes: an assertion failure anywhere in that file is now reported by name.

Does not establish that #584's fast-fail was a joinable-thread terminate. A
/GS cookie failure and a CRT invalid-parameter call raise the identical status
and nothing in the job log distinguishes them, because no output survives to
say. No second cause was found by inspection of the :1294-:1325 window
either: the httplib assert in stop() is compiled out by the lane's
Release config, the worker pool is 12 threads and not an exhaustion story, and
AsyncLLM's output handler already catches everything and is joined by
shutdown().

#584 therefore stays open. This is the reporting repair it needs first, and
the next Windows run is the measurement.

.agents/issue-index.md is not appended to: #584 already has a row there
(ENG-RELEASE-WINDOWS), and the index forbids a second row for the same issue.
The three places that must agree are that row, the spec, and this body.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]

mudler added 2 commits August 17, 2026 22:53
…report

#584 kills test_openai_api_server.exe with 0xC0000409 and prints nothing but
the doctest version banner, so nothing about the failure can be read off either
Windows job. One half of that is provable by inspection rather than inferred:
the file holds joinable std::thread objects across assertions that throw, and
~thread on a joinable thread is std::terminate, which MSVC raises as the same
status a /GS failure would. A named assertion failure therefore arrives as an
opaque fail-fast with no reporter output.

This spec lands before the conversion so the commit order shows it did. It is
deliberately scoped as a REPORTING repair: it does not claim the cure, and #584
stays open when it lands.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
…ure again

Every socket case in test_api_server.cpp held a bare joinable std::thread across
assertions that throw. A failing REQUIRE, or a bare json::parse on an unexpected
body, unwound past the thread object, and ~thread on a joinable thread is
std::terminate. Nothing caught an exception escaping serve() either, which is
std::terminate from the other direction.

On MSVC that reaches abort(), which is __fastfail, which raises 0xC0000409 and
bypasses SEH. doctest's Windows handler never runs and its buffered stdout is
discarded, so a NAMED assertion failure arrives as an opaque exit code with no
Status: and no assertions: line -- which is all either Windows lane has printed
since #584 opened.

ScopedThread joins on every path, runs the body in a catch-all, and rethrows the
escaped exception from join() where doctest can name it. ScopedServerThread adds
the stop() for the shape that dominates the file, and waits for the accept loop
before stopping, because httplib's stop() is a no-op until is_running_ is up and
a joiner that ignored that would trade a 0.79 s fast-fail for a 180-minute CI
timeout. It owns the stop exclusively, so the explicit h.server.stop() lines go:
a second stop() is not a no-op, it trips assert(svr_sock_ != INVALID_SOCKET) on
any build that is not NDEBUG. 17 sites converted, which is every std::thread in
the file. The 6 client threads also moved below the vectors their bodies write
into, so the join now happens before those vectors are destroyed.

Measured, same build dir, CI's own no-NDEBUG configuration. Case count unchanged
and non-zero: 62 cases / 733 assertions / SUCCESS on both arms. Red-first, with
one assertion mutated to fail while the thread is joinable, each arm compiling
rc 0 and showing one changed line: BEFORE exits 134 with "terminate called
without an active exception" and "test case CRASHED: SIGABRT"; AFTER exits 1
with the named failure and no abort at all.

This is a REPORTING repair, not the cure. A /GS cookie failure and a CRT
invalid-parameter call raise the identical status and nothing in the job log
distinguishes them, so #584 stays open and the next Windows run is what
measures it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
@localai-bot
localai-bot merged commit 8daf58e into main Aug 18, 2026
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants