Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -434,4 +434,6 @@ tasks:
env:
KUBECONFIG: '{{.KUBECONFIG_PATH}}'
cmds:
- go test -tags e2e -v -count=1 -timeout 10m ./...
# -p 1: the shared-host example measures a CLUSTER-WIDE API-server watch gauge, which the
# backend suite's own real watches corrupt if their packages run concurrently.
- go test -tags e2e -v -count=1 -p 1 -timeout 10m ./...
9 changes: 9 additions & 0 deletions docs/adopting.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,3 +149,12 @@ boundary; a per-user backend keeps Kubernetes RBAC as the direct boundary by con

Next: [saving.md](saving.md) for the host-owned write path and [operations.md](operations.md) for
stream monitoring and limits.

## Tested shared host and middleware

Use the [compiled shared ConfigMap host](../gateway/kube/examples/sharedstream/README.md) to compose
participant identity resolution, service-account sharing, per-subscriber authorization, session
expiry and `WriteTimeout`. Its [middleware capability recipe](../gateway/kube/examples/sharedstream/README.md#middleware-capability-test)
checks the actual wrapped server writer in a host test. `CheckHTTPStreaming` must run before
streaming: it emits no response, but clears the existing write deadline. A positive timeout requires
flush and deadline support; unsupported transports abort and can cause browser reconnects.
17 changes: 16 additions & 1 deletion docs/auth.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ opts.Clients = func(context.Context, string, gateway.Principal) (gateway.Backend
```

The [`SubjectAccessReviewAuthorizer`](../gateway/kube/authz.go) adapter delegates the decision to
Kubernetes. `subjectOf` maps the principal to the Kubernetes username and groups. The adapter checks
Kubernetes. `subjectOf` supplies the API-server-resolved username, groups, UID and extras. The adapter checks
both `list` and `watch`. An incomplete review is refused, and an explicit `Denied` wins over
`Allowed`.

Expand All @@ -116,3 +116,18 @@ The host owns writes, CSRF protection, audit and write authorization. Before a m
`gateway.ValidateMergePatch` with the effective projection and current object, and include the
captured UID and resourceVersion preconditions. Project any resource returned to the browser.
See [saving](saving.md) for the complete flow.

## Tested shared-host composition

The [shared ConfigMap host](../gateway/kube/examples/sharedstream/README.md) demonstrates a local
SelfSubjectReview helper using participant credentials, service-account SARs and data access,
fixed scope, session/token expiry and bounded HTTP delivery. Identity resolution is an example,
not a public library authentication API. It does not re-resolve identity on every timed check.

`ReauthorizationTimeout` applies to periodic callbacks after acquiring the delivery gate. Opening
and cycle authorization need their own host callback deadlines. Do not put a short callback deadline
on the entire healthy stream. Write bounds limit in-flight HTTP I/O, not all gate waiting, backend
operations or callback work; declare and measure the total revocation budget under the intended load.
For 200 allowed participants, opening can issue 400 SARs plus 200 SSRs. Timers can align, recovery
adds checks, and client-side throttling consumes callback budgets. Neither the example's rate settings
nor Voter's reported rehearsal results are production defaults or supported-version evidence.
56 changes: 56 additions & 0 deletions docs/facts/shared-host-rehearsal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Observed: the shared-stream host under 200 independent identities

> **Recorded by hand from a manual run** — unlike [observed-v1.36.2+k3s1.md](observed-v1.36.2+k3s1.md),
> this file is not generated by `task cluster-facts`. It is a WITNESS to one run of one scenario on
> one disposable cluster. It is not a capacity guarantee, a latency distribution, or a statement of
> supported Kubernetes versions.

## What was run

| | |
|---|---|
| Commit | `ed5c4d0` (branch `feat/shared-stream-transport`, unreleased) |
| Scenario | `TestSharedHostRealAPI` in [`gateway/kube/examples/sharedstream`](../../gateway/kube/examples/sharedstream/) |
| Command | from `gateway/kube`: `KRM_SHARED_SUBSCRIBERS=200 go test -tags e2e -count=1 -p 1 -run TestSharedHostRealAPI ./examples/sharedstream/` |
| Cluster | k3s `v1.36.2+k3s1` via `task cluster-up`, Go `1.27.1` |
| Host config | `WriteTimeout=5s`, `ReauthorizationInterval=30s`, `ReauthorizationTimeout=5s` |
| Fixture client | `QPS=100`, `Burst=400` — declared fixture capacity, not a library default |

## What it did

| Observation | Result |
|---|---|
| 200 independent identities reached synced | 109.2 ms |
| API-server ConfigMap WATCH requests | baseline 26 → **27 while all 200 were streaming** |
| One RBAC grant withdrawn mid-stream | affected stream closed in 27.26 s; **199 peers retained** |
| After all disconnects | API watch count and host gauges returned to baseline |

The **+1** is the claim worth keeping: 200 authenticated subscribers cost the API server exactly one
additional ConfigMap watch, measured from `apiserver_longrunning_requests` on the API server itself
rather than from the library's own counters.

Revocation at 27.26 s sits inside Proposal 0006's 60-second reference budget, with a 30-second
recheck interval. One sample does not establish a bound: it does not include a stalled callback, a
blocked peer under write pressure, or an aligned timer cohort.

## What this run does NOT show

- **No proxy and no browsers.** Delivery went to an in-process `httptest` server over loopback. There
was no Traefik, no TLS termination hop, and no browser renderer or reconnect behavior.
- **No resource limits.** This ran in the devcontainer, not under a constrained CPU/memory budget, so
the numbers say nothing about behavior near saturation.
- **One run, not a distribution.** No percentiles, no sustained soak, no peak memory or CPU sample.
- **Service-account tokens, not logins.** The 200 identities are namespaced service accounts with
scoped RBAC. No OIDC provider or session system was exercised.
- **Not a version-support statement.** This is the repository's supported cluster version only.

## Reproducing it

`task test-cluster` runs this scenario at 2 subscribers as part of the manual gate. Set
`KRM_SHARED_SUBSCRIBERS` (2..200) for the burst profile.

The scenario needs **exclusive use of the cluster**: `apiserver_longrunning_requests` is cluster-wide
and carries no namespace label, so any other ConfigMap watcher is counted in the same number. This is
why `task test-cluster` passes `-p 1`. Running the packages concurrently corrupts the reading — the
backend e2e suite's own real watches inflate it — and the failure looks like a watch leak that is not
there.
46 changes: 45 additions & 1 deletion docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,10 @@ object name, UID, principal, patch contents, or an error message.

| Signal | Meaning | First response |
|---|---|---|
| `http_transport_rejected` | bounded HTTP serving lacks required writer capabilities; no logical stream opened | test the mounted middleware and provide flush/deadline support; aborted requests may reconnect |
| `consumer_resync` rising | upstream continuity was lost or a shared subscriber fell behind | correlate with API-server errors, shared overflows, and deployments |
| `shared_overflow` | one subscriber exceeded `SharedOptions.QueueDepth` | increase only after checking browser stalls and event rate; resnapshot is intentional |
| `terminal_error` | authorization, upstream, or protocol failure ended a stream | alert by low-cardinality error code; browsers must not retry terminal errors |
| `terminal_error` | logical-stream failure, observed before attempting its terminal frame; delivery can fail | alert by low-cardinality error code; browsers must not retry terminal errors |
| `event_suppressed` ratio | `krm-spec/v1` is removing expected churn | a sharp drop may mean callers selected `krm-full/v1` or a projection changed |
| stream count / snapshot duration | connection pressure or oversized scopes | narrow namespaces/selectors; avoid accidental all-namespaces watches |
| unorderable `resourceVersion` terminal errors | an unsupported or aggregated API does not meet strict ordering | use `OrderingLenient` only after accepting the reduced monotonicity guarantee |
Expand All @@ -27,6 +28,7 @@ object name, UID, principal, patch contents, or an error message.

| Control | Default | Use |
|---|---:|---|
| `gateway.Options.WriteTimeout` | 0 (no deadline) | set a positive per-operation write-plus-flush budget for bounded HTTP delivery |
| `gateway.Options.HeartbeatInterval` | 20 seconds | set below the shortest proxy idle timeout |
| `gateway.SharedOptions.QueueDepth` | 256 live events | tune after measuring; it bounds memory per slow subscriber |
| `ScopePolicy.AllowLabelSelector` | false | enable only for an endpoint that deliberately supports caller narrowing |
Expand All @@ -36,3 +38,45 @@ object name, UID, principal, patch contents, or an error message.
Snapshot object and byte limits remain a host-level scope policy concern. The gateway refuses to guess a
safe universal cap: object size, useful namespace size, and recovery behavior are product-specific.
Measure snapshot size and duration per allowed scope before opening broad all-namespaces endpoints.

## Counting lifetimes

`stream_opened`/`stream_closed` count `StreamProjection` entry/return, including authorization
failure, but exclude HTTP identity/scope refusals before entry. `shared_subscription_opened` and
`shared_subscription_closed` count active attachments, including warm-cache joins. Overflow, scope
death or leave ends an attachment once; repeated `Stop` does not count again. Cleanup may finish
later. A resnapshot can replace an attachment on one logical stream without another API watch.

Callbacks are synchronous and may run concurrently, including under shared locks. Return promptly;
do not panic or reenter the gateway. Update gauge counters synchronously rather than through a lossy
exporter queue. Opens precede matching closes, with no global order across independent lifetimes.
Ignore unknown kinds. Scope values are not safe metric labels. See the tested
[counter mapping](../gateway/kube/examples/sharedstream/metrics.go).

These gauges do not count HTTP requests still resolving identity or upstream watch handles. Keep
host metrics for those needs. Measure physical API-server WATCH requests independently with the
metrics available on the tested Kubernetes version; the shared-host fixture uses
`apiserver_longrunning_requests` filtered to the resource and `verb="WATCH"`.

## Bounded HTTP delivery

Set `WriteTimeout` explicitly; zero preserves no library-installed deadline and negative values
are rejected. Each header, event or heartbeat write and flush shares one deadline. Successful
operations clear it, so quiet streams can outlive many timeout periods. A failed operation cancels
its stream and cannot be retried as a terminal frame. Flush success does not acknowledge browser
receipt. Generic `Stream` sinks remain responsible for bounded I/O.

The transport owns write deadlines while serving and attempts to clear them at exit. It cannot
retrieve and restore a host's previous deadline. A whole-response `http.Server.WriteTimeout` is
not a per-operation substitute for a long-lived stream. Earlier request deadlines constrain I/O;
request cancellation may still wait for the current operation's bound. Backend opening/cleanup,
authorization gate waiting, encoding and host callbacks are outside that I/O budget.

With a positive bound, unsupported writers abort before streaming and emit `http_transport_rejected`.
No unbounded diagnostic frame is attempted; browser reconnects are possible. Test the actual mounted
middleware using the [capability-check recipe](../gateway/kube/examples/sharedstream/README.md#middleware-capability-test).
HTTP/1.1 sockets and TLS HTTP/2 (including two streams on one connection) are tested, along with
transparent `Unwrap` and opaque wrappers. Other middleware/proxy combinations need host tests.

For authorization bursts and the limits of the 200-subscriber profile, see the
[shared-host capacity guide](../gateway/kube/examples/sharedstream/README.md#verification-and-capacity).
5 changes: 5 additions & 0 deletions docs/proposals/0006-stream-and-save-implementation-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ Follow the standing [design rules](../../CONTRIBUTING.md#design-rules) and
[release policy](../releasing.md). [Proposal 0005](0005-kubernetes-stream-and-save-semantics.md)
explains the unresolved tradeoffs; this document owns work order and acceptance criteria.

[Proposal 0007](0007-shared-stream-host-integration.md) proposes bounded HTTP delivery, lifecycle
observations and tested shared-host composition. That work can support the authorization bounds
and continuation measurements below, but adds no dependency or acceptance gate to this plan.
Hosts may demonstrate the existing requirements with their own bounded sinks and instrumentation.

## Baseline and order

Use the [adoption guide](../adopting.md), [saving guide](../saving.md) and
Expand Down
Loading