Add V3 streaming event ingestion endpoint

[ejsmith]

Summary

Adds a V3 event ingestion endpoint that makes sending one event trivial and streaming many events efficient. V3 is a transport over the existing event pipeline, so V2 and V3 process events identically.

• POST /api/v3/events and POST /api/v3/projects/{projectId}/events (Minimal APIs, client token).
• The body can be one JSON event, a JSON array of events, or newline-delimited/whitespace-separated events. Event boundaries come from the JSON structure, so pretty-printed events work. Bodies are read incrementally from the request PipeReader and never buffered whole.
• Every event property is optional. The smallest event is {"message":"..."}; type defaults to error or log like V2. Oversized fields are truncated by the pipeline instead of rejecting the event.
• Errors can be sent as exception_type + raw stack_trace (stored as a V2 simple error) or as a structured error (the V2 error model), which groups exactly like V2.
• Each microbatch runs through EventPipeline.RunAsync, the same call EventPostsJob makes. Stacking, source maps, bot throttling, sessions, 404s, notifications, statistics, and usage accounting are shared with V2, with no object storage write, queue round trip, or second deserialization.
• One bad event does not fail the request: invalid, oversized, or malformed events are reported with their zero-based index and skipped (malformed NDJSON resumes at the next line that begins with {). Oversized events are skipped by a byte scanner without being retained, and reading stays linear for long tokens.
• An optional event id makes resending safe within the idempotency window (1 day by default). Ids are claimed as pending, marked stored after the pipeline, and released when an event is blocked, invalid, or fails. A resend that arrives while the original is still processing gets a retryable event_in_progress instead of a possibly false duplicate.
• Responses: 200 with counts (persisted, discarded, duplicate, blocked, invalid, failed) and per-event errors; 402 before reading when the organization is suspended or out of events; 429/503 with Retry-After; 404 when V3 is disabled; problem details include partial_result and status-specific retry guidance.
• gzip/Brotli, compressed/decompressed/per-event limits, per-instance stream and processing concurrency limits, request timeout, and project/organization allowlists. Disabled by default.
• Isolated V3 OpenAPI document at /docs/v3/openapi.json, HTTP examples, architecture and client protocol documents, BenchmarkDotNet suites, and a V2/V3 load harness.

Design change from the earlier revision

The earlier revision implemented a second ingestion pipeline (server-side stack parsing and fingerprinting, a stack route cache, distributed quota reservations, a batch writer, a stack usage pipeline, and a side-effect worker). Review found it grouped errors differently from V2, which would have duplicated every stack for a project moving to V3, skipped V2 behavior such as source maps, and changed shared V2 code. That implementation was removed and the shared V2 files now match main.

This reverses the issue's decision to keep V3 off PipelineBase. Pipeline performance work now belongs in the shared pipeline so V2 benefits too. Follow-ups:

• Server-side stack trace parsing in the shared pipeline (the SimpleErrorPlugin TODO), verified against real SDK output so V2 and V3 produce identical signatures. The prototype parser is preserved in 17bb0a30d.
• Detect discarded stacks before applying the event limit.
• Create-only bulk writes with per-item results.
• Overlap reading with processing if measurements show a benefit.

Compatibility

V3 is a new, opt-in surface. V2 routes, payloads, the queued handoff, the 202 response, and the shared pipeline, usage, stack, and notification code are unchanged from main. Shared changes are additive: OverageMiddleware skips /api/v3 (the V3 endpoint performs its own submission, suspension, and event-limit checks), EventPostRequestBodyStream gains optional rejection messages, and the OpenAPI setup now produces separate v2 and v3 documents with an unchanged v2 snapshot.

Verification

• dotnet build Exceptionless.slnx — 0 warnings, 0 errors.
• Full backend suite — 3,230 passed, 0 failed, 3 skipped.
• V3 tests: stream reader (36, including fragmented input, oversized single tokens, and resynchronization), endpoint integration (28, including V2/V3 same-stack checks for structured and raw-trace errors, duplicates, in-progress resends, and partial results), concurrency limiter, request body middleware, and V2/V3 OpenAPI snapshots.
• BenchmarkDotNet dry run — all 13 benchmarks executed. The load harness was not run against a deployed environment.

Part of #2368

[niemyjski]

Follow-up feedback and thermo-nuclear audit complete.

Feedback inventory and classification:

• 0 review threads, 0 inline review comments, and 0 submitted reviews from human, Codex, or Copilot reviewers. There were therefore no thread replies or resolutions to perform.
• The sole conversation comment is the GitHub Actions coverage report (78% overall). It is informational and already passing, not an actionable review finding.

Thermo-nuclear full-diff result:

• One structural blocker was found: UsageServiceTests.cs had grown from 658 to 1,183 lines because the V3 quota and settlement tests were appended to the existing broad fixture.
• RCA: the new tests shared fixture state, so they accumulated in the original file instead of being separated by behavior.
• Fix: commit 6d92911 splits the V3 quota and settlement cases into a focused partial test file without changing production code or test identities. The files are now 985 and 210 lines; no branch-added C# file crosses from below 1,000 lines to above it. The complete diff was also rechecked for spaghetti growth, boundary/type leaks, thin abstractions, duplicated helpers, sequential orchestration, and non-atomic updates; no other proven actionable defect remains.

Verification:

• git diff --check: passed.
• dotnet build --no-restore --disable-build-servers -m:1: passed with 0 warnings and 0 errors.
• Test identity comparison: no method-name drift; all 29 UsageServiceTests cases are discovered after the split.
• Local focused integration execution was blocked before test bodies ran by another worktree sharing the test-* Elasticsearch aliases (invalid_alias_name_exception). The isolated authoritative GitHub run is green: test-api 11m1s, test-e2e 5m7s, test-client 1m14s, docker-build 3m13s, website/version/CLA green.

No concrete external blocker remains. PR draft/admin state was intentionally left unchanged.

[ejsmith]

Review: V3 streaming ingestion

Verdict: request changes. NDJSON over a normal URL with a bearer token, with the server parsing raw stack traces, is the right shape. But this PR is a second ingestion pipeline, not just a streaming endpoint. It has its own materializer, stack assignment, quota leases, stack-count pipeline, notification path and side-effect worker: ~18.3k lines across 143 files. The fork already disagrees with V2 on grouping, so a project that switches to V3 would get every stack duplicated.

Line numbers are at 17bb0a30d.

Against the goals

Easy to send a single event: not yet

• Content-Type: application/json is rejected with 415; only application/x-ndjson is accepted.
• A pretty-printed object fails. The reader splits on newlines, so line 1 is just {, which throws JsonReaderException and returns 400 for the whole request. That's what Postman, JSON.stringify(x, null, 2) or a hand-written curl body produce. Confirmed with a probe test against EventIngestionV3StreamReader.
• id and type are both required. V2 infers type (error vs log) and doesn't require an id.
• Field limits reject the event instead of truncating it: a message over 2,000 chars, a 101-char tag, 51 tags, or a malformed reference_id drops the whole event (EventIngestionV3Processor.Validate). EventMaterializer.cs:127-133 already truncates the V2 way, but that code can't run because validation rejects first. Losing an error because its message is long is the wrong trade for an error tracker.

Streaming a huge set: works, but brittle

• One bad line kills the rest of the stream. A syntax error or a wrong-typed routing field ("id": 1, confirmed) throws out of the reader and returns 400. The response then says "Retry the complete request", which fails the same way every time. Lines are already framed, so the server could count that line as invalid (with its index) and continue.
• Errors identify events only by client id (max 100), so an event with a missing or bad id can't be matched back to what the client sent.
• Exceeding 10k events returns 413 after earlier microbatches were already committed.
• Retry-After is never sent, even though the protocol doc tells clients to honor it.
• Over-quota events return 200 with blocked: n, so clients get no signal to back off.

Fast: fewer hops than V2, but chatty
For a warm 100-event microbatch (cached route, events carrying dates), each step runs one after another:

• ~11–14 Redis calls, plus a per-organization distributed lock.
• 2 Elasticsearch calls: a multi-get duplicate pre-check, then the bulk create.

On top of that:

• The side-effect worker re-saves every event that has request or environment data.
• A Redis key is kept per stored event for 7 days.
• Reading stops while each microbatch processes.
• The per-organization reservation lock serializes every microbatch for one organization across all replicas. It also doesn't provide the guarantee it claims, because commit and release don't take it (UsageService.cs:715-722).
• OverageMiddleware is bypassed, so an exhausted organization still pays for fingerprinting, routing, the duplicate check and the lock on every event before the events are blocked.

Easy for clients: the protocol doc (Profiles 0–3, the conformance kit) is genuinely good. But there's no way to send a structured error, so the existing .NET/JS SDKs would lose fidelity and get different grouping by switching.

Correctness blockers

1. V3 groups errors differently from V2. Six real .NET errors were compared: the 6.2.0 SDK's ToErrorModel against ex.ToString() sent through V3. Every signature hash differed.
   • V3 strips method parameters, which V2 keeps (StackTraceParser.cs:232,337-341 vs MethodExtensions.GetSignature).
   • Async methods are named differently.
   • Source maps are skipped; 15_SourceMapPlugin runs before the signature in V2.
   • Manual stacking uses a new hash scheme.
   • The result is duplicated stacks, with fixed/ignored/discarded status, history and notification state lost.
   • The compatibility test can't catch this because it builds the "V2" error from V3's own parser output.
2. The parser misgroups real traces (StackTraceParser.cs:184,200-210,343-361):
   • Any line containing @ becomes a frame, so Error: user bob@example.com not found makes one stack per email address and stores part of the address in the stack.
   • A message line starting with "At …" is treated as a frame.
   • Location-only frames (anonymous JS, Node internals) put line:col and the bundle hash into the fingerprint, so there's a new stack every deploy.
   • Python uses the outermost frame, so unrelated Django errors share a stack.
   • The Go fallback hashes pointer values, giving a new stack per occurrence.
   • About 200 nested ---> inner exceptions exceed the serializer's max depth.
3. Usage undercount (UsageService.cs:826-857). A late settlement into a bucket the saver never visited for that organization or project is added and then expires. Billing is undercounted and the monthly limit can be exceeded.
4. Stored but never counted (EventBatchWriter.cs:259-311). Regression handling runs between the bulk add and enqueueing the side-effect work item. A lock timeout or crash there leaves events persisted with no stack counts or notifications.
5. Stale discard routes (StackRouteResolver.cs:114). Route versions are last-writer-wins on currentVersion + 1, not Elasticsearch order. A discard/reopen race can silently discard an open stack's events, or keep storing a discarded one's, for up to the 1-hour cache.

"V2 unchanged" isn't true

• Usage saver: it now saves one bucket later, but GetEventsLeftAsync still sums only the current and previous buckets, which leaves a limit gap at every 5-minute boundary. It also now caps by the month's remaining events, which changes how V2 trims batches.
• Stack saves: StackRepository enables OriginalsEnabled, adding an extra get before every stack save, plus Redis lock and cache work on every save. A lock timeout can now fail V2 saves after Elasticsearch has already succeeded.
• Notifications: V2 notifications and webhooks now go through duplicate-detection markers.
• Redis keys: 7-day :v3:processed keys are written for every organization and project in every bucket, even for V2-only traffic.
• Benchmark instrumentation in the public contract: a track-processing header appears in the V2 OpenAPI, plus an exposed CORS header and an EventPostService tracking path.

Suggested direction

Split it so the first PR is only the transport:

1. One endpoint, forgiving input. Accept application/json and NDJSON. Frame by JSON token depth instead of newlines, so all of these work: a single object, a pretty-printed object, a top-level array (streamed one element at a time, not buffered) and NDJSON. Keep the per-record byte cap. For NDJSON, resync at the next line after a bad one.
2. Minimal event = {"message":"…"}. Make id optional (recommended for retries), infer type the way V2 does, and truncate oversized fields the way V2 does.
3. Feed the existing pipeline. Materialize each microbatch to PersistentEvent and call EventPipeline.RunAsync(events, organization, project) inline, exactly as EventPostsJob does. Source maps, bot throttling, identical stacking, sessions/404s, notifications and stats then come for free, with no fork.
4. Optimize the shared pipeline in follow-ups that also help V2: check for discarded stacks before quota, and use create-only bulk with per-item results (201 = persisted and billable, 409 = duplicate).
5. Server-side raw stack parsing as its own PR, with golden fixtures from real SDK output so V2 and V3 must produce identical hashes. Accept a structured error from day one so existing SDKs can move without regrouping.
6. Client signals:
   • Report errors with each event's index and id.
   • Make retry guidance specific to the status code.
   • Send Retry-After on 429 and 503.
   • Return 402 for an exhausted organization instead of 200 with blocked.

Separately, the branch conflicts with main (370 commits behind), and CI hasn't run on the current head.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.AppHost	23%	23%	128	❌
Exceptionless.Core	77%	69%	10940	✔
Exceptionless.Insulation	51%	43%	370	➖
Exceptionless.Web	86%	71%	9494	✔
Summary	80% (28560 / 35663)	69% (14108 / 20481)	20932	✔

[ejsmith]

Closing in favor of the revised plan in #2368 (comment). Research into comparable products showed that acknowledging after a durable enqueue (not after full processing) and evolving /api/v2/events with NDJSON is the better design for client stability, server scalability, and compatibility. The feature/v3-event-ingestion branch is kept for reference; its stream reader, concurrency limiter, and load harness will be carried into the focused PRs listed in the plan.