fix(clickhouse): cancel reads the client has abandoned

[lohanidamodar]

Problem

In nyc3 prod on 2026-08-30, /v1/usage/events breakdown requests for one large
tenant returned 500 after exactly ~30.0s, ~495/hour:

ClickHouse query failed: Operation timed out
  Utopia\Usage\Adapter\ClickHouse::query()
  <- findAggregatedFromTable() <- findFromTable()

30.0s is utopia-php/client's DEFAULT_TIMEOUT, which is the socket timeout
on both bundled adapters.

The amplifier is what this PR fixes: the read path sent no max_execution_time
and no query_id. setNextQueryId() existed but was only ever called from
benchmarks and routing tests, and no SETTINGS clause was emitted on reads. So
when the socket died at 30s, the ClickHouse query kept running to completion,
and each user reload stacked another one. Measured blast radius during the burst
(Prometheus): 23.0M rows/s, 1.33 GB/s, 0.91 normalised CPU — 90% of all cores —
~14.6B rows read in 35 minutes, with no comparable excursion in the prior 14 days.

Change

Reads only:

1. Execution cap. Reads send max_execution_time plus
   timeout_overflow_mode = throw. The cap defaults to 25s — deliberately
   below the 30s client socket timeout, so ClickHouse aborts the query itself
   and returns a clean TIMEOUT_EXCEEDED instead of the socket dying with the
   query still live. Configurable via the new readTimeout constructor argument;
   null disables it. throw is pinned because a profile defaulting to break
   would hand back a silently truncated result set, and these are usage totals.
2. query_id per read, reusing the existing setNextQueryId(). Its contract
   is single-use-then-cleared, and benchmarks/routing tests pin an id to correlate
   against system.query_log — so a caller-pinned id is honoured and only an
   unpinned read gets a generated one.
3. Cancellation. When a read fails, the adapter issues
   KILL QUERY WHERE query_id = {queryId:String} ASYNC. ASYNC so it returns
   without waiting for the query to stop; every failure from it is swallowed so a
   missing KILL privilege can't mask the real error.

Deliberately not applied to the write/ingest path — starving inserts is worse
than a slow read. purge() and all DDL keep going through the plain query().

Degradation: a server that refuses the settings (readonly = 1 user → Code: 164,
or an older build → Code: 115) retries the read once without them and stops
trying for the life of that adapter instance.

Verification

Run against a real ClickHouse 26.9.1 through the actual adapter.

(a) The cap aborts the read server-side (readTimeout: 2):

adapter failed after 2.01s
exception: ClickHouse query failed with HTTP 408: Code: 159. DB::Exception:
  Timeout exceeded: elapsed 2000.284 ms, maximum: 2000.000 ms. (TIMEOUT_EXCEEDED)
  [Query: SELECT count() AS c FROM numbers_mt(100000000000) WHERE number % 7 = ...]
probe queries still running on the server: 0

(b) KILL QUERY reaps a query the client abandoned. Control first — cap and
query_id disabled, client socket timeout 3s (today's behaviour):

adapter failed after 3.00s: ClickHouse query failed: Operation timed out after 3002 ms
system.processes 2s after the client gave up:
query_id                                elapsed_s  read_rows
e802687e-6389-44ca-bf4a-1d81798d49e2    5          54.93 billion

Same 3s socket death with the fix:

query_id: raju-evidence-bfe1499a
system.processes for that query_id: 0
probe queries still running: 0
query_log verdict for that query_id:
type                       exception_code  exception
ExceptionWhileProcessing   394             Code: 394. DB::Exception: Query was cancelled. (QUERY_WAS_CANCELLED)

(c) Settings and query_id actually reach ClickHouse on the public read path,
and the write path stays uncapped (system.query_log after find() +
addBatch()):

query_kind  query_id                          auto_generated_id  max_execution_time
Select      8162dcc9eefea2e827149a88f8d3619a  1                  25
Insert      bc1d0d3f-9cb4-439f-8589-a2b3d0e32a77  0

(timeout_overflow_mode does not appear in Settings because throw is the
server default and query_log only records settings that differ from it. That it
is honoured at all is shown by ?max_execution_time=1&timeout_overflow_mode=break
returning HTTP 200 where throw returns HTTP 408, and by
timeout_overflow_mode=nonsense failing with
Code: 145 ... Must be one of ['break', 'throw'].)

(d) readonly = 1 user degrades instead of breaking:

raw probe as user ro: 1 (plain select works)
find() as readonly user returned 4 row(s)
second find() (settings now skipped for the instance) returned 4 row(s)
adapter request count: 3   (capped attempt + fallback + second read)

Tests

• ClickHouseReadTimeoutTest (integration): the cap and a generated 32-hex
  query_id land in system.query_log; a caller-pinned id survives; readTimeout: null sends no cap; the insert path is not capped; a slow read is aborted by
  the cap; a read abandoned by a 1s client socket leaves nothing in
  system.processes; a non-positive readTimeout is rejected.
• ClickHouseReadCancellationTest (hermetic, scripted PSR-18 transport): the KILL
  is issued for the right query_id; a failing KILL does not mask the read error;
  an uncapped read is never cancelled; a Code: 164 rejection falls back to an
  uncapped read and stops re-trying the settings.

Review follow-ups

• query_id ownership (b5f08f8). The generated id was staged in the
  adapter-wide nextQueryId slot and read back by query(), so it stayed
  visible across random_bytes() and the multipart build — both hookable I/O
  under Swoole. A concurrent read could adopt it and be cancelled in this read's
  place. The id is now taken from the slot in one read-and-clear step
  (consumeNextQueryId()), passed to query() explicitly, and the KILL gets
  its own id so it never consumes one pinned for another read.
• No KILL when the connection never opened (b5f08f8). A dead host would
  otherwise cost a second socket timeout on top of the first. Both bundled
  transports classify that as ConnectionException/DnsException, distinct from
  the TimeoutException this PR targets, so cancellation is skipped there.
  A KILL is still sent after a server-returned error: it costs one cheap
  metadata request and covers a gateway timing out in front of ClickHouse while
  the query is still live.

[greptile-apps]

Greptile Summary

The PR adds server-side execution limits and query IDs to ClickHouse reads, then attempts asynchronous server-side cancellation when a transport abandons a query.

• Defaults reads to a configurable 25-second execution cap while leaving writes and DDL uncapped.
• Falls back to uncapped reads when ClickHouse rejects per-query settings.
• Keeps read and cleanup query IDs separately owned so concurrent cancellation cannot target another read.
• Skips the cleanup request for connection and DNS failures where no server-side query could have started.
• Adds integration and scripted-transport coverage for timeout, cancellation, fallback, and query-ID behavior.

Confidence Score: 5/5

The PR appears safe to merge because the previously reported query-ID ownership and unnecessary outage-cleanup issues are resolved.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/Usage/Adapter/ClickHouse.php	Adds capped read execution, explicit per-read query-ID ownership, settings-rejection fallback, and best-effort cancellation; both previously reported issues are resolved at the current head.
tests/Usage/Adapter/ClickHouseReadCancellationTest.php	Covers abandoned-read cancellation, swallowed cleanup failures, settings fallback, unreachable servers, and preservation of another read’s staged ID.
tests/Usage/Adapter/ClickHouseReadTimeoutTest.php	Verifies execution-cap propagation, generated and pinned query IDs, uncapped writes, server-side timeout behavior, and client-abandonment cleanup.
tests/Usage/Adapter/ScriptedClient.php	Provides deterministic transport outcomes and request capture for cancellation-path tests.
README.md	Documents read-timeout configuration, socket-timeout ordering, readonly-user behavior, and the uncapped write path.
CHANGELOG.md	Records the new ClickHouse read cap, query IDs, cancellation behavior, and degradation strategy.

_{Reviews (3): Last reviewed commit: "docs(readme): document the ClickHouse re..." | Re-trigger Greptile}

[lohanidamodar]

Closing in favour of handling this server-side. cancel_http_readonly_queries_on_client_close=1 cancels the query at client disconnect, which is the actual failure mode here — an abandoned read continuing to completion and burning cluster CPU. Verified on 26.9.1: baseline still running 8s in (5s after the client died); with the setting, cancelled at exactly the disconnect with Code 394 QUERY_WAS_CANCELLED. That needs no application change and no deploy, so the app-side cap and KILL machinery here are not worth the surface area.