feat: harden control-plane protocol for reliable board communication#3491
Merged
Conversation
An idle GET /events stream now emits an SSE comment (": ping") every
15s, so clients can tell a quiet session from a hung transport (e.g.
across a paused VM) and reconnect instead of blocking forever on a read
that will never fail. SSE comments carry no id and are invisible to
EventSource clients, so the sequenced stream and its replay are
untouched.
Event writes and heartbeats come from different goroutines, so frame
writes are serialized with a mutex.
The snapshot now carries the session's total cost (TotalCost, which includes sub-session and item-level costs such as compaction). External consumers like the board no longer need to fetch every message and sum per-message costs — which undercounted — to display what a session has spent.
…ble code
GET /snapshot of a session the server does not know now 404s with
{"code":"unknown_session"} in the body. Clients that own their session
ids (the board) can finally tell the two 404s apart: with the code, the
server is current but the session is not created/attached yet — keep
waiting; without it, the route itself is missing (a binary that predates
/snapshot) — relaunch to pick up the current binary. Previously both
looked identical and the board could only guess.
The new server tests cover heartbeats, the aggregate snapshot cost, and
this error code.
The event fan-out drops events for a slow subscriber rather than block the bus — fine for content deltas, but dropping a stream_started, stream_stopped, user_message, error, paused or session_title skews an SSE consumer's turn accounting for good: the event never reaches the control plane's replay buffer, so reconnecting cannot recover it (the board's stuck-running cards). On overflow of a turn-boundary event, evict the oldest pending message — almost always a delta the next one supersedes — and retry once. Other events keep the previous drop behavior, and a wedged subscriber still never blocks the fan-out.
The built-in board's control-plane client arms an idle watchdog once
the server proves it sends heartbeats (": ping" SSE comments): three
missed heartbeats means the transport is hung — not that the session is
quiet — so the stream is aborted and the card's watcher reconnects and
resyncs, instead of showing a stale status forever. Servers that
predate heartbeats never arm the watchdog, so idle streams to older
agents keep working.
aheritier
added
area/api
aheritier
approved these changes
Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
External observers —
docker agent boardand other dashboards — communicate with the agent over the--listencontrol-plane via SSE and snapshot polling. On degraded transports such as future docker-sandbox tunnels, a silent TCP connection is indistinguishable from a live-but-quiet one, and a handful of event types were silently dropped under backpressure, leaving boards stuck showing a running session that had already finished. This set of changes makes the protocol robust enough to survive those conditions.The server now emits a
: pingSSE comment every 15 seconds on idle/eventsstreams so clients can distinguish a quiet session from a hung transport; writes are mutex-serialized with real event frames so a heartbeat never interleaves with a delta.GET /snapshotnow includes the session's aggregate cost viasess.TotalCost(), fixing undercount that happened when clients tried to sum per-message fields themselves. A 404 response for an unknown session now carries{"code":"unknown_session"}so a client can tell "session not yet created, keep polling" from "route missing or old binary, relaunch" — previously both were opaque 404s. On the fan-out side, subscriber overflow no longer dropsstream_started,stream_stopped,user_message,error,paused, orsession_titleevents; instead the oldest pending delta is evicted. Those turn-boundary events never reached the SSE replay buffer, so dropping them caused boards to show perpetually-running cards. The existing non-blockingStreamStoppedpath (deadlock fix #3070) is unchanged. Finally, the built-in board's SSE client arms an idle watchdog after receiving its first heartbeat ping: three missed heartbeats trigger a reconnect and full resync. Agents that predate heartbeat support never arm the watchdog, so older binaries keep working without change.All five changes are backward compatible for existing clients. New tests cover the heartbeat ticker (
pkg/server/server_heartbeat_test.go), the fan-out eviction logic (pkg/app/fanout_test.go), and the board client watchdog (pkg/board/client_test.go); all pass under-race.