Skip to content

feat: apply the shared event delivery rules - #230

Open
Zaimwa9 wants to merge 2 commits into
mainfrom
feat/events-shared-retry-rule
Open

Zaimwa9 wants to merge 2 commits into
mainfrom
feat/events-shared-retry-rule

Conversation

@Zaimwa9

@Zaimwa9 Zaimwa9 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Closes Flagsmith/flagsmith-private#316. Follow-up to #226.

Applies the shared event delivery rules (flagsmith-private#318). The Go SDK already implements them.

Changes

  • Retry: 408, 429, 502, 503, 504, network errors and timeouts are retried. There are 3 attempts in total, with full-jitter backoff starting at 1s, doubling each time, capped at 10s. Every other status drops the batch, 500 included. feat: experimentation support #226 retried any 5xx once.
  • Re-queue: a batch that still fails goes back to the head of the buffer and waits for the next timed flush or flushEvents(). It is dropped if it fails during close().
  • Buffer bound: maxBufferItems, dropping the oldest events first. feat: experimentation support #226 used max(maxBufferItems, 1000). A batch is never larger than maxBufferItems.
  • 401/403: the timer stops, the buffer is dropped, later events are dropped, and one error is logged. Sending resumes only once the client is re-created.
  • Dedupe: an exposure stays deduplicated until its event is delivered (2xx, partial 202 included) or dropped.
  • rejected[]: entries on a 2xx are logged, never resent, and counted as dropped.
  • Dropped counter: FlagsmithClient.getDroppedEventCount(), monotonic.
  • close() bound: one batch's worst case, which is now 3 attempts at the client's timeouts plus the 1s and 2s backoff ceilings.
  • Docs: a README section, plus Javadoc on flushEvents(), close() and withEventsMaxBufferItems(), including the requirement that short-lived processes call close().

Behaviour change

With a very small maxBufferItems, a burst of events can now drop events even when the events API is healthy, because the buffer is capped at maxBufferItems while two batches are in flight. This is what the shared rule specifies, and Go behaves the same. The default is 1,000.

Testing

  • mvn verify and mvn verify -P test-okhttp4 both pass (502 tests).
  • Each new rule was broken on purpose (10 mutants in total), and the tests catch every one.

Event delivery now follows the shared rules (flagsmith-private#316, #318):
- Retry 408, 429, 502, 503, 504, network errors and timeouts: 3 attempts
  in total, full-jitter backoff from 1s doubling to a 10s cap. Every other
  status, 500 included, drops the batch.
- A batch that still fails is put back at the head of the buffer and waits
  for the next timed flush or flushEvents(). On close() it is dropped.
- The buffer is bounded by maxBufferItems, dropping the oldest events,
  instead of max(maxBufferItems, 1000). A batch never exceeds it.
- A 401 or 403 stops the processor: the timer stops, the buffer is dropped,
  later events are dropped, and one error is logged.
- An exposure stays deduplicated until its event is delivered or dropped.
- rejected[] entries on a 2xx are logged and counted as dropped.
- FlagsmithClient.getDroppedEventCount() returns the monotonic drop count.
- The close() bound covers the three attempts and their backoffs.
- README and Javadoc cover flushEvents() and close() for short-lived
  processes.
@Zaimwa9
Zaimwa9 requested a review from a team as a code owner September 30, 2026 09:02
@Zaimwa9
Zaimwa9 requested review from khvn26 and removed request for a team September 30, 2026 09:02
@Zaimwa9

Zaimwa9 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

@themis-blindfold review

@themis-blindfold

Copy link
Copy Markdown

⚖️ Themis review: ✅ Ship it

The event queue now follows the stated delivery rules: bounded buffering, selective retries with jitter, requeueing, authentication shutdown, and monotonic drop accounting. CI is still running.

Area Score
🎯 Correctness 5/5
🧪 Test coverage 5/5
📐 Code quality 4/5
🚀 Product impact 4/5
📝 Walkthrough
  • Event delivery - retries only the documented transient statuses and network failures, then requeues or drops according to lifecycle state.
  • Buffering and deduplication - caps pending events at the configured limit while retaining exposures until delivery or a terminal drop.
  • Client API and docs - exposes dropped-event accounting and documents the shutdown requirement for short-lived processes.
  • Coverage - exercises retry classifications, requeue overflow, unauthorised shutdown, close semantics, jitter bounds, and drop accounting.
🧪 How to verify
  1. Run mvn -Dtest=EventProcessorTest,FlagsmithClientTest test.
  2. Run mvn verify.
  3. Run mvn verify -P test-okhttp4.
  4. Exercise a retryable 503 followed by success, then three consecutive 503s followed by flushEvents() and close().
    Automate: Retain the event-processor matrix in both OkHttp compatibility builds.

Product take: A solid reliability improvement for experimentation telemetry, especially for intermittent events API failures and short-lived clients. The bounded buffer makes its delivery trade-off explicit and observable.

🧭 Assumptions & unverified claims
  • Focused Maven tests could not be run here because Maven is unavailable; CI was still running.

The event queue now has a clear exit strategy when the network gets dramatic. · reviewed at b087b19

@Zaimwa9

Zaimwa9 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

@themis-blindfold review

@themis-blindfold

Copy link
Copy Markdown

⚖️ Themis review: ✅ Ship it

The shared event-delivery rules are implemented consistently across batching, retries, shutdown, counters, tests, and user-facing guidance. The CI test matrix is still in progress; Maven is not installed in this environment, so I could not rerun the suite locally.

Area Score
🎯 Correctness 5/5
🧪 Test coverage 5/5
📐 Code quality 4/5
🚀 Product impact 4/5
📝 Walkthrough
  • Event delivery - narrows retry eligibility, adds full-jitter retry backoff, and requeues exhausted transient failures.
  • Buffer lifecycle - bounds queued events to the configured batch size, tracks drops, and preserves exposure deduplication until settlement.
  • Client lifecycle - makes close flush delivery bounded and documents the requirement for short-lived processes.
  • Observability - exposes a monotonic dropped-event counter and accounts for API rejections and stopped delivery.
🧪 How to verify
  1. Run mvn verify.
  2. Run mvn verify -P test-okhttp4.
  3. Exercise 408, 429, 502, 503, 504, connection-error, 400, 401, and 403 event responses; confirm the retry, drop, and stop rules.
  4. Saturate two in-flight batches with a small event buffer and confirm oldest queued events are counted as dropped.
  5. Call close() with a transiently failing batch and confirm it completes within the configured timeout bound and counts the batch as dropped.
    Automate: keep the event-processor matrix covering retry, requeue, overflow, unauthorised, and close paths in the regular test suite.

Product take: Solid reliability and operability improvement for experimentation telemetry, especially for transient outages and short-lived workloads. The new drop counter makes the intentional bounded-buffer trade-off observable.

🧭 Assumptions & unverified claims
  • The CI test matrix had not completed at review time.
  • Maven is unavailable in this environment, so local test execution was not possible.

The events now fail with receipts instead of disappearing into the night · reviewed at 5896682

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant