Skip to content

[Brook S2] Generated the Stream service bindings with nexgen, and added the writer client. - #372

Open
moetemp wants to merge 4 commits into
moe/brook/review/s1-stream-protosfrom
moe/brook/review/s2-stream-bindings
Open

moetemp wants to merge 4 commits into
moe/brook/review/s1-stream-protosfrom
moe/brook/review/s2-stream-bindings

Conversation

@moetemp

@moetemp moetemp commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

What was changed

This PR vendors the Stream service WIT and generates its Python bindings with nexgen, into temporalio/common/_streams/system/. It also adds the writer client in writer_system/, written by hand.

  • The WIT is A1's file byte for byte. nexgen generates the client target and the Workflow target. The stream handle's methods are publish, read, close and describe. A read item is the StreamItem union of StreamItemData, StreamItemSuperseded and StreamItemClose.
  • The writer has no WIT. writer_system/ keeps the shape of nexgen's client output by hand and raises the service's errors, so a later switch to generated code changes no caller. Only Activities and clients make writer calls. S4 implements create_nexus_client for both clients.
  • read-records has @nexus.poll. The stream handle's iter_read takes the next request from next_read_request in temporalio.nexus.system._support. A read with no wait ends at an empty page that didn't move.
  • The WIT's @nexus.source lines fill request fields from where the call runs, such as the Stream id, the owner and the Producer. This PR adds those sources to temporalio.nexus.system._support. The Workflow sources raise until Workflow code can write in S7. client_producer raises, since a generated call doesn't say which client makes it.
  • Proto enums are generated IntEnum classes, as on nexgen main (temporalio/nexgen#262), with A1's names. Five of them have @nexus.fallback for values they don't know. StreamHandle.owner is an Execution model. The service errors are generated classes, which S3 exports with the public names.
  • The client extension has only the operations that no resource method binds, as in temporalio/nexgen#344.
  • The bindings name temporalio.common.StreamDefinition, so this PR adds the base class.
  • poe gen-protos runs the generator, and check-protos installs the pinned nexgen, so drift fails CI.
  • The Workflow Service bindings move to nexgen 0.2.11, the change of temporalio/sdk-python#2001. Its dispatch passes a System Nexus call that it didn't generate to start_system_nexus_operation on the Workflow outbound interceptor. S7 serves the Stream calls of Workflow code through that hook.
  • Fork-only pin. nexgen is built from moetemp/nexgen at f9578ddb, the fork chain below on top of temporalio/nexgen#344, #325 and #345. Upstream installs a nexgen release by --version once the chain merges.

Depends on S1 (#371), A1 (temporalio/api#895), temporalio/nexgen#344, #325 and #345, and moetemp/nexgen#101, moetemp/nexgen#102, moetemp/nexgen#78, moetemp/nexgen#80, moetemp/nexgen#82, moetemp/nexgen#84, moetemp/nexgen#85, moetemp/nexgen#87, moetemp/nexgen#89, moetemp/nexgen#92, moetemp/nexgen#94, moetemp/nexgen#95, moetemp/nexgen#103 and moetemp/nexgen#99. No changelog fragment, since S9 has the Streams entry.
Part of the Brook review series.

Why?

The Streams API calls the Stream service through generated bindings, as the SDK calls the Workflow service System Nexus operations. They're generated, so they get their own PR, before the hand-written Stream code. The writer client is here too, since it has the same shape.

Checklist

  1. How was this tested:
    • gen_stream_service.py --install-nexgen and poe gen-protos-docker leave no diff. poe lint, poe gen-docs and the binding tests pass.
    • Fork CI passes, check-protos included, except the arm jobs, since the fork has no arm runners.

@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 25, 2026
@moetemp
moetemp force-pushed the moe/brook/review/s2-stream-bindings branch 3 times, most recently from 3925df2 to f2a5ec4 Compare October 26, 2026 08:44
@moetemp
moetemp force-pushed the moe/brook/review/s1-stream-protos branch from 235fde0 to 3b5246f Compare October 26, 2026 08:44
@moetemp
moetemp force-pushed the moe/brook/review/s2-stream-bindings branch from f2a5ec4 to ba649e0 Compare October 27, 2026 09:07
@moetemp
moetemp force-pushed the moe/brook/review/s1-stream-protos branch from 3b5246f to 92725de Compare October 27, 2026 09:07
@moetemp
moetemp force-pushed the moe/brook/review/s2-stream-bindings branch from ba649e0 to e1864c4 Compare October 27, 2026 09:17
No nexgen release has the long-poll, client and client-only features, so
the generator installs nexgen at a fork revision. `poe gen-protos` runs
it, so `check-protos` checks the bindings for drift.
@moetemp
moetemp force-pushed the moe/brook/review/s1-stream-protos branch from 92725de to 7afb312 Compare October 28, 2026 01:33
@moetemp
moetemp force-pushed the moe/brook/review/s2-stream-bindings branch from e1864c4 to 50537f7 Compare October 28, 2026 01:33
The writer has no WIT, so its client keeps the shape of nexgen's output by hand and raises the
service's errors. The bindings need `StreamDefinition` and the Stream helpers of the support
module, so this layer adds them. The Workflow sources fail until Workflow code can write. The
Workflow Service bindings move to nexgen 0.2.11, whose dispatch gives an unknown System Nexus
call to the generic hook.
@moetemp moetemp changed the title [Brook S2] Generated the Stream service and writer bindings with nexgen. [Brook S2] Generated the Stream service bindings with nexgen, and added the writer client. Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant