Skip to content

Add canonical dedicated database usage samples - #32

Merged
abnegate merged 3 commits into
mainfrom
feat/dedicated-database-samples
Aug 27, 2026
Merged

Add canonical dedicated database usage samples#32
abnegate merged 3 commits into
mainfrom
feat/dedicated-database-samples

Conversation

@abnegate

@abnegate abnegate commented Aug 27, 2026

Copy link
Copy Markdown
Member

Summary

  • add an immutable canonical sample contract for dedicated database usage
  • store samples in a separate ClickHouse MergeTree without changing existing event, gauge, daily, or telemetry APIs
  • expose exact bounded ingestion-ID snapshots with duplicate, conflict, gap, discontinuity, and truncation evidence

This is dependency support for Track B Phase 1 only. It does not enable any Cloud reader, billing mutation, release, or producer deployment.

Canonical contract

A sample carries environment, region, project internal ID, database internal ID, member, generation, sequence, metric, interval start/end, value, and event version. Identity and payload digests use documented length-prefixed SHA-256 encoding so producers and consumers can reproduce them across languages.

Each logical row receives an adapter-owned random ingestion ID before transport. getSampleWatermark() performs one bounded ClickHouse snapshot query for an exact stream/range and captures exact (ingest ID, canonical ID, payload hash) entry fingerprints visible in that snapshot. findSamples() admits only that exact membership set. Later inserts cannot cross the boundary, while a repeated HTTP request retains its IDs and cannot change the canonical result.

Conflict selection uses one row-consistent tuple. Multiple full payload tuples for one canonical identity produce conflict evidence and no representative sample, so independent column aggregates can neither fabricate an observation nor throw while constructing a mixed interval. Any watermark/result truncation, conflict, gap, or discontinuity makes isComplete() false.

This contract does not make a producer HTTP request or spool durable. Downstream producers must retain the identical payload until acknowledgment and publish authoritative member/generation/sequence boundaries. Consumers must query exact interval boundaries and refuse billable mutation unless SampleResult::isComplete() is true.

Verification

Seen red before implementation:

  • initial SampleTest: 3 failures/errors because Utopia\\Usage\\Sample did not exist
  • exact-watermark/conflict regressions: schema failure, two representative-row failures, and a watermark API error against real ClickHouse

Exact head edb11155408b229b5a63092f0fc2f1fd61030657:

  • docker compose exec -T usage vendor/bin/phpunit --configuration phpunit.xml tests: 293 tests, 1,427 assertions
  • composer lint: pass
  • composer check: pass at PHPStan maximum level
  • focused real ClickHouse sample suite: 12 tests, 69 assertions

Regression coverage includes identical logical duplicates, crash retry, post-watermark insert exclusion without sleeps, conflicting values/intervals, row-consistent conflict refusal, compact sequence gaps, interval discontinuity, result and watermark truncation, range binding, stable transport retry IDs, adversarial reused-ID conflict, async-insert acknowledgment forcing, and exact schema columns.

Compatibility

The branch merges cleanly with current PR #30 and the earlier combined tree passed lint and PHPStan. The combined existing-schema database suite exposed a separate #30 upgrade-path blocker: 41 Database adapter tests reject the newly introduced category attribute because setup does not backfill existing collections. This PR has no code dependency on #30 and deliberately does not modify it.

@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown

Greptile Summary

The PR adds an immutable ClickHouse-backed canonical sample contract for dedicated database usage without changing existing event, gauge, daily-rollup, or telemetry semantics.

  • Adds canonical sample, range, watermark, result, and compact-gap value objects.
  • Adds durable sample ingestion and bounded ingestion-ID snapshot reads to the ClickHouse adapter.
  • Detects duplicate observations, conflicts, sequence gaps, interval discontinuities, and truncation.
  • Adds public Usage forwarding APIs, documentation, schema setup, and focused ClickHouse regression coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
src/Usage/Adapter/ClickHouse.php Adds the sample ledger schema, durable batched writes, exact bounded watermark membership, row-consistent conflict handling, and completeness evidence.
src/Usage/Sample.php Defines the immutable canonical sample and reproducible length-prefixed identity and payload digests.
src/Usage/SampleRange.php Defines validated stream, sequence, and interval boundaries for exact sample reads.
src/Usage/SampleWatermark.php Encapsulates validated ingestion-ID membership and enforces exact range matching.
src/Usage/SampleResult.php Exposes canonical samples and fail-closed completeness evidence.
tests/Usage/Adapter/ClickHouseSampleTest.php Covers schema, duplicates, conflicts, exact watermark isolation, gaps, discontinuities, truncation, and async-insert acknowledgment.

Reviews (3): Last reviewed commit: "(fix): Bind watermarks to exact sample e..." | Re-trigger Greptile

Comment thread src/Usage/Adapter/ClickHouse.php Outdated
Comment thread src/Usage/Adapter/ClickHouse.php Outdated
@abnegate
abnegate merged commit c8ff0f2 into main Aug 27, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant