perf(clickhouse): split mid-hour windows so grouped event reads route

[lohanidamodar]

The problem

#34 reshaped the three event projections onto (tenant, metric, toStartOfHour(time,'UTC'), dims) and added bucketAlignedFilters() to re-express time predicates onto that hourly key. The rewrite only applies when both bounds land exactly on an hour (if ($bound % 3_600_000 !== 0) return null;).

Console windows never do — the production request that motivated this was 900,000 ms off the hour. So the rewrite always declined, the projections were never used, and breakdown queries fell back to a full base-table scan.

The split

A mid-hour window is now read as up to three branches that partition it:

• an interior on whole hours, predicated on toStartOfHour(time,'UTC') — the only part expressible on the projection key, and where nearly all the data is
• up to two partial edge hours, predicated on raw time against the base table, bounded at 2/windowHours of the scan

They are UNION ALLed and re-summed in an outer query. ORDER BY / LIMIT / OFFSET move to that outer query, since no single branch holds a whole group. Each branch compiles independently and numbers its bindings from param0, so each set is renamed apart (b0_, b1_, …) before the branches are merged.

The read stays on its single unsplit query when there is nothing to gain:

shape	behaviour
both bounds hour-aligned	#34's existing rewrite, unchanged
no whole hour between the edges	unsplit, raw time
sub-hour interval (15m)	unsplit, base table
aggregate('max')	unsplit — the projections store sum(value)
gauges (TYPE_GAUGE)	unsplit, raw time
dims or filters no projection covers	unsplit — three base-table branches would be strictly worse

Slate

Added method, status, clientType, clientName, deviceName, osName, sdk — the dimensions the console breaks down on. Each is single-dim: a shared (method, status) projection keys the cross product, measuring 19.2M rows on a 100M-row fixture against 1.15M + 2.55M for the two on their own. ip, hostname, city and resourceId stay off as too high-cardinality (resourceId alone keys 3.1M).

Measured — 100M-row fixture, 26-day window 15 min off the hour, whale tenant

Rows and marks read are cache-independent; ch_ms is from system.query_log with mark, uncompressed and query-condition caches dropped before each run.

shape	rows before → after	marks before → after	ch_ms	projection
groupBy(clientName)	14,999,552 → 49,152 (305x)	1831 → 6	481 → 21	p_by_clientName
groupBy(deviceName)	14,999,552 → 40,960 (366x)	1831 → 5	447 → 19	p_by_deviceName
groupBy(clientType)	14,999,552 → 40,960 (366x)	1831 → 5	454 → 19	p_by_clientType
groupBy(service)	14,999,552 → 40,960 (366x)	1831 → 5	502 → 18	p_by_service
groupBy(country)	14,999,552 → 49,152 (305x)	1831 → 6	480 → 18	p_by_country
chart 1h / 1d / 1w / 1M	14,999,552 → 40,960 (366x)	1831 → 5	~490 → ~17	any single-dim
chart 1d + groupBy(clientName)	14,999,552 → 49,152 (305x)	1831 → 6	639 → 19	p_by_clientName
one-sided edge (aligned start)	14,991,360 → 32,768	1830 → 4	508 → 14	p_by_clientName
24h window	581,632 → 49,152	71 → 6	33 → 18	p_by_clientName
groupBy(resourceType, resourceId)	14,999,552 → 14,999,552	1831 → 1831	—	none — not split
15m interval	581,632 → 581,632	71 → 71	—	none — unchanged
window inside one hour	16,384 → 16,384	2 → 2	—	none — unchanged

Projection footprint on that fixture: 1,042,500 rows / 1.79 MiB across six projections against a 100M-row / 1.18 GiB base table.

path is the exception — measure it before trusting it

The synthetic fixtures generate path as one of 200 templated routes. Production path holds resource ids, not route templates (/v1/databases/682c…/collections/682c…/documents), so its cardinality tracks a customer's resource count. The 229x improvement the 200-path fixture shows for p_by_path does not generalise.

Re-measured on a fixture with realistic path cardinality — 20M rows, one tenant, 2 metrics, 26 days, 30,000 distinct paths:

dim	rows before → after	marks	bytes read	ch_ms
path (30k values)	9,998,900 → 7,849,533 — 1.27x	1221 → 959	1.12 GB → 0.91 GB	538 → 485
country (100 values)	9,998,900 → 167,284 — 59.8x	1221 → 22	270 MB → 4.98 MB	166 → 24
clientType (4 values)	9,998,900 → 61,300 — 163x	1221 → 10	270 MB → 1.68 MB	170 → 20

And the storage cost on the same fixture:

projection	rows	on disk	vs base (20M rows, ~613 MiB of base columns)
p_by_path	15,606,668	716.91 MiB	78% of the rows, larger than the base columns
p_by_country	187,200	267.22 KiB	0.9% of rows
p_by_clientType	7,488	24.10 KiB	0.04% of rows

Global distinct keys (ignoring the per-part duplication above): p_by_path is 9.70M against 20M base rows — 48.5% of the table.

So p_by_path costs more disk than the columns it summarises, is written on every insert, and returns ~1.27x on the read. This PR keeps it as briefed, but the number that should decide its future is a production uniqExact((tenant, metric, toStartOfHour(time), path)) — nobody has that yet. The bounded-enum dims (clientType, deviceName, clientName, osName, method, status, sdk, country, service) are where the wins are, and they are the ones this PR adds.

Correctness

Results are byte-identical to the unsplit query. Proven by running the same find() calls against origin/main and this branch on the 100M-row fixture and diffing the serialised Metric documents:

• 34 shapes — every dim breakdown, charts at 1h/1d/1w/1M, chart+dim, top-N with an outer LIMIT, orderBy value/time/dim with offset, between / exclusive / open-ended bounds, sub-second edges, filters on and off the projection, aggregate('max'), 15m — all identical.
• 15,000 path groups identical on the realistic-cardinality fixture.

One caveat, pre-existing and unchanged: ORDER BY value DESC LIMIT n has no tiebreaker, so when groups tie at the cut-off which ones come back is arbitrary. On the realistic fixture 195 paths tie at the top value, and origin/main disagrees with itself across two runs on that shape. This PR does not change that.

TYPE_EVENT + grouped remains the gate — it is never relaxed to "aggregation hint present". TYPE_GAUGE reads (VerifyBilledUsage, concurrency rollups) never reach the split and keep their raw time predicate, asserted by testGaugeReadIsNeverSplit. A 5-minute-aligned delta window keeps its exact results whether or not it is wide enough to peel an hour out of, asserted by testFiveMinuteAlignedDeltaWindowKeepsItsResults.

Tests

New ClickHouseWindowSplitTest (33 tests): parity against a optimize_use_projections = 0 base-table scan plus a system.query_log.projections routing assertion for all ten slate dims and every chart interval; top-N limit applied after recombination; one-sided edges; per-branch binding renaming; unsplittable-window fallback; 15m on the base table; uncovered dims and filters not split; max not split; gauges not split; 5-minute delta windows.

ClickHouseDimRoutingTest::testMidHourWindowKeepsRawPredicateAndTotals asserted that a mid-hour window does not route — the premise this PR inverts. It is now testMidHourWindowSplitsSoTheInteriorRoutes and asserts the projection fires with unchanged totals. Two dimensionless assertions were relaxed from naming p_by_path to "some projection fired", since with ten entries the optimizer's pick among equivalent candidates is arbitrary.

Suite: 339 tests, and the set of failing test IDs is identical to an unmodified origin/main checkout in the same environment (41 DatabaseTest with no MariaDB locally, 2 ClickHouseSchemaTest index-DDL rendering on ClickHouse 26.9 vs the 25.11 CI image, 1 ClickHouseSampleTest). phpstan and pint clean.

Notes

• ADD PROJECTION does not backfill existing parts. The seven new projections cover newly written parts and parts as they merge; historical coverage needs MATERIALIZE PROJECTION, same as fix(clickhouse): key event projections tenant-first on an hourly bucket #34.
• getTimeSeriesFromTable() also calls bucketAlignedFilters() and is deliberately left unchanged — the timing-out console queries go through find().

[greptile-apps]

Greptile Summary

The PR accelerates mid-hour grouped ClickHouse event reads by splitting them into projection-routable whole-hour interiors and raw-table edge windows, then recombining their aggregates.

• Adds window parsing, partitioning, branch binding isolation, and outer ordering/pagination.
• Expands the single-dimension event projection slate for common console breakdowns.
• Adds parity and projection-routing coverage for split, fallback, boundary, aggregation, and metric-type scenarios.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete correctness or security defect established in the changed paths.

The split branches form disjoint half-open windows, retain millisecond precision, independently rename bindings, re-sum additive event aggregates before pagination, and fall back for unsupported dimensions, intervals, gauges, and aggregate modes.

Important Files Changed

Filename	Overview
src/Usage/Adapter/ClickHouse.php	Adds projection-aware window splitting and outer recombination while retaining unsplit fallbacks for unsupported query shapes.
tests/Usage/Adapter/ClickHouseDimRoutingTest.php	Updates routing expectations for mid-hour windows and extends dimension projection coverage.
tests/Usage/Adapter/ClickHouseWindowSplitTest.php	Adds comprehensive result-parity, routing, edge-boundary, binding, fallback, and pagination tests for split reads.

_{Reviews (1): Last reviewed commit: "perf(clickhouse): split mid-hour windows..." | Re-trigger Greptile}