feat(vllm): register single-shot so the call pays as it runs

[rickstaa]

vllm registered as persistent while its own README apologised for it:

This app is single-shot by nature but currently registers as persistent. It will switch to single-shot once #5 lands.

The app is one request in, one response out, with no state between calls. It stayed persistent only because a single-shot call could not keep paying, so metering it required a session the gateway held open by hand.

That capability is on the pinned SDK branch now. call_runner sets needs_ongoing_funding when the price unit is metered and starts a funding loop around the request (live_runner.py:862,888-906); for a streamed response the returned stream owns that loop, so an SSE generation keeps paying while tokens flow. Orchestrator side, ProxyLiveRunnerSingleShot reserves a session around the call and defers ReleaseSession until the response finishes.

So _forward drops from three SDK calls to two, and now reads exactly like hello-world:

cursor = await runner_selector(discovery_url=args.discovery, app=APP_ID)   # Livepeer: 1
runner = cursor.candidates[0]
result = await call_runner(                                                # Livepeer: 2
    runner=runner, runner_url=runner.url.rstrip("/") + runner_path,
    payload=payload, signer_url=signer_url,
)

This fills the empty cell in the axis table. Every single-shot example paid once and every metered one was persistent, so single-shot paired with metered pricing — one call, unknown duration, paid per second, client managing no session — had no example.

Verified end to end

Ran the real stack on an RTX 3090, offchain.

• Buffered: client.py --prompt "In one sentence, what is Livepeer?" returned a normal completion.
• Streaming: client.py --stream printed a haiku token by token, so SSE survives the mode change.
• Session lifecycle, from the orchestrator log, one pair per call, which is the single-shot contract:

live runner session reserved  session_id=session_g5tymgov app=vllm/qwen2.5-0.5b-instruct
live runner session released  session_id=session_g5tymgov duration=109.490733ms

One behaviour this exposes

A single-shot call holds its capacity slot for the whole call, so with capacity: 1 a second concurrent request is refused. Testing that, the orchestrator's 503 escaped the gateway as aiohttp's default 500 Server got itself in trouble — useless to an OpenAI client. It is now caught and returned as JSON with the upstream status:

req1 http=200
req2 http=503  {"error": {"message": "HTTP 503 ... no capacity available for runner", "type": "livepeer_error"}}

That failure predates this PR, but under single-shot a busy runner is the normal thing a concurrent client meets, so it should not surface as an opaque 500.

Supersedes #9, which only touched the README and runners.json and never converted gateway.py. Closes #4, closes #5.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR switches the vllm example to single-shot runner mode now that single-shot metered payment can be maintained across the duration of a call, and updates the local OpenAI-compatible gateway to use the simpler single-shot discovery → call flow while preserving streaming (SSE).

Changes:

• Update vllm runner registration to mode: "single-shot" (static runners.json).
• Simplify vllm/gateway.py from reserve/call/release to runner_selector → call_runner, including explicit handling for upstream 503 capacity errors.
• Refresh repo and example documentation to reflect single-shot + metered pricing behavior and the updated call flow.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
vllm/runners.json	Switch vLLM runner mode from persistent to single-shot.
vllm/README.md	Update example docs to reflect single-shot mode and metered-per-call behavior.
vllm/gateway.py	Drop session management; use discovery + call_runner for buffered + SSE requests; add Livepeer HTTP error mapping.
README.md	Update the top-level matrix and runner-mode guidance to list vllm as single-shot.

Suppressed comments (1)

vllm/gateway.py:123

• _forward_or_error only handles LivepeerHTTPError. If runner discovery fails or no candidates are available (errors that are not HTTP status codes), the gateway will still return aiohttp's default 500 instead of a readable OpenAI-style JSON error.

        except LivepeerHTTPError as exc:
            return web.json_response(
                {"error": {"message": str(exc), "type": "livepeer_error"}},
                status=exc.status_code,
            )

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (3)

vllm/gateway.py:42

• _forward_or_error currently only catches LivepeerHTTPError. runner_selector / call_runner can also raise other LivepeerGatewayError subclasses (e.g. no orchestrator/runner available, as handled in other examples), which would again surface as aiohttp's default 500. Importing the base error here allows returning a JSON error consistently.

from livepeer_gateway.errors import LivepeerHTTPError
from livepeer_gateway.live_runner import call_runner
from livepeer_gateway.selection import runner_selector

vllm/gateway.py:131

• Errors like NoOrchestratorAvailableError / NoRunnerAvailableError (which are LivepeerGatewayError subclasses) will currently fall through and become aiohttp's default 500 HTML response. Returning a JSON error for these cases keeps the gateway consistently OpenAI-client-friendly (even when there isn't a usable upstream).

        try:
            return await _forward(request)
        except LivepeerHTTPError as exc:
            return web.json_response(
                {"error": {"message": str(exc), "type": "livepeer_error"}},
                status=exc.status_code,
            )

vllm/gateway.py:83

• cursor.candidates[0] will raise IndexError when discovery returns zero candidates, which would surface as an opaque 500 to OpenAI clients. Also runner_url is built from request.path, which drops any query string; using request.rel_url preserves path + query for full reverse-proxy behavior.

        runner = cursor.candidates[0]
        runner_url = runner.url.rstrip("/") + runner_path

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

vllm/gateway.py:132

• _forward_or_error only catches LivepeerHTTPError. Discovery/selection failures (e.g., no orchestrator/runner available) and other gateway-level errors will still surface as aiohttp’s default 500, which undermines the goal of returning a readable JSON error to OpenAI clients. Consider also catching LivepeerGatewayError (or the specific selection errors) and mapping them to an appropriate upstream-facing status (e.g., 502/503) with the same JSON error envelope.

    async def _forward_or_error(request: web.Request) -> web.StreamResponse:
        # A single-shot call holds a capacity slot for its duration, so a busy runner
        # answers 503. Hand that back as JSON an OpenAI client can read.
        try:
            return await _forward(request)
        except LivepeerHTTPError as exc:
            return web.json_response(
                {"error": {"message": str(exc), "type": "livepeer_error"}},
                status=exc.status_code,
            )

vllm/gateway.py:83

• runner_path = request.path drops query parameters, so any request like /v1/...?... will be forwarded without its query string. Use request.path_qs (or request.rel_url) so the upstream sees the same URL the client sent.

        # GET /v1/models carries no body; everything else posts JSON.
        payload = await request.json() if request.can_read_body else {}
        runner_path = request.path  # e.g. /v1/chat/completions
        cursor = await runner_selector(  # Livepeer: 1
            discovery_url=args.discovery,  # omit if the signer does discovery itself
            app=APP_ID,
        )
        runner = cursor.candidates[0]
        runner_url = runner.url.rstrip("/") + runner_path