[do not merge] A/B runs for ClickHouse issue #114640 (Q10/Q11 on 192-core machines) - #1360
Closed
alexey-milovidov wants to merge 15 commits into
Closed
[do not merge] A/B runs for ClickHouse issue #114640 (Q10/Q11 on 192-core machines)#1360alexey-milovidov wants to merge 15 commits into
alexey-milovidov wants to merge 15 commits into
Conversation
…48xl Three copies of the clickhouse system with pinned binaries, to attribute the Q10/Q11 hot-time shift seen in daily mainline runs on 192-core machines (ClickHouse/ClickHouse#114640): - clickhouse-i114640-a: v26.5.6.113 (last minor before the suspect window) - clickhouse-i114640-pre: master @ 5719dd022fc (before ThreadPool LIFO #100177) - clickhouse-i114640-post: master @ e78a4dac750 (the ThreadPool LIFO merge) The concurrent-QPS phase is skipped (BENCH_CONCURRENT_DURATION=0); the query protocol is unchanged. Not intended to be merged.
alexey-milovidov
had a problem deploying
to
benchmark-approval
August 13, 2026 16:46 — with 
GitHub Actions
Error
GitHub Actions
Error
…st, clickhouse-i114640-pre (c7a.metal-48xl)
Contributor
|
Results for Logs:
|
… point Round 1 result: 26.5.6.113, master@5719dd022fc (pre-LIFO #100177) and master@e78a4dac750 (post-LIFO) are ALL fast on Q11 (~0.095s) on c7a.metal-48xl, while the same-day daily run of latest stable is slow (0.237s) and comparable on the other 42 queries. So the ThreadPool LIFO change is exonerated, and the cause lies in master 06-13..06-23 or a 26.6 release-branch backport. Round 2 pins: - clickhouse-i114640-v2661: official 26.6.1.1193 (expected slow) - clickhouse-i114640-m0618: master @ 0ea5338b83a (2026-06-18) - clickhouse-i114640-m0622: master @ 7ef85344d14 (2026-06-22)
alexey-milovidov
had a problem deploying
to
benchmark-approval
August 13, 2026 17:35 — with
GitHub Actions
Error
GitHub Actions
Error
…0-m0622, clickhouse-i114640-v2661 (c7a.metal-48xl)
Contributor
|
Results for Logs:
|
…to 06-23..07-18 Round 2 showed official 26.6.1.1193 and masters 06-18/06-22 are ALL fast on Q11 - because the daily 'clickhouse' system installs via 'curl https://clickhouse.com/ | sh', which downloads builds.clickhouse.com/master/<arch>/clickhouse: the LATEST MASTER build, not the latest stable release. The version attribution in issue #114640 was therefore wrong; the true regression window is master 2026-06-23..2026-07-18. Round 3 pins: - clickhouse-i114640-daily: the exact daily install (control, expected slow) - clickhouse-i114640-m0629: master @ 68024785a70 - clickhouse-i114640-m0711: master @ ab8535b3ab0 (26.7 branch point) - clickhouse-i114640-m0718: master @ 9d197f142d0
alexey-milovidov
had a problem deploying
to
benchmark-approval
August 13, 2026 18:03 — with
GitHub Actions
Error
GitHub Actions
Error
…0-m0629, clickhouse-i114640-m0711, clickhouse-i114640-m0718 (c7a.metal-48xl)
Contributor
|
Results for Logs:
|
…indow) Round 3: Q10/Q11/Q24/Q26 flip between master 06-29 (fast) and 07-11 (slow); Q23 flips separately between 07-11 and 07-18; the daily-CDN control reproduces the dashboard slow mode (so it is code, not the artifact form). Round 4 pins: - x1: c3f8ef333f5 (07-02, just before #108928) - x2: 597ffae623d (07-05, with #108928, without #109389) - x3: 6a27522419b (07-07, with both #108928 and #109389)
alexey-milovidov
had a problem deploying
to
benchmark-approval
August 13, 2026 18:31 — with
GitHub Actions
Error
GitHub Actions
Error
…2, clickhouse-i114640-x3 (c7a.metal-48xl)
Contributor
|
Results for Logs:
|
…directly Round 4: X1 (07-02, pre-#108928) is already slow -> the uniqExact copy-on-write PRs are exonerated; window is master 06-29..07-02. The affected queries (Q10/Q11/Q24/Q26: full scan + selective non-PK filter) are exactly the query-condition-cache beneficiaries, while the untouched ones (Q35 no WHERE, Q36/Q37 PK-pruned) are not. Round 5 pins: - y1: ddbfc52e731 (06-30 23:42, just before qcc-for-topn) - y2: 61b644e0380 (07-01 02:50, just after qcc-for-topn) - y3: m0711 slow build + use_query_condition_cache=0 - y4: y1 fast build + use_query_condition_cache=0 (does fast mode need QCC?)
alexey-milovidov
had a problem deploying
to
benchmark-approval
August 13, 2026 19:03 — with
GitHub Actions
Error
GitHub Actions
Error
…2, clickhouse-i114640-y3, clickhouse-i114640-y4 (c7a.metal-48xl)
Contributor
|
Results for Logs:
|
Round 5 results split the regression in two: - Q24/Q26: caused by qcc-for-topn #104478 (flip at y1->y2) and fully recovered by use_query_condition_cache=0 on the slow build - the QCC integration is the regression for the TopN family. - Q10/Q11: NOT QCC (QCC-off leaves m0711 slow: 0.218/0.344), flip in 07-01 02:50 .. 07-02 10:05. Prime suspect #107490: memory accounting feeds the two-level conversion bytes threshold. z1 = c79b0d9227e (07-01 07:31), z2 = 32129f76fca (07-01 07:39): the span contains only #107490 and a CI-only fuzz PR.
alexey-milovidov
had a problem deploying
to
benchmark-approval
August 13, 2026 19:34 — with
GitHub Actions
Error
GitHub Actions
Error
Runs in parallel with round 6 (the #107490 straddle): - w1: slow m0711 + group_by_two_level_threshold_bytes=0 - w2: slow m0711 + max_bytes_ratio_before_external_group_by=0 - w3: fast y1 + group_by_two_level_threshold=1 (force two-level) If w1 recovers and w3 degrades, the mechanism is bytes-threshold-driven two-level conversion of tiny per-thread hash tables at 192 threads, driven by the #107490 accounting change.
alexey-milovidov
had a problem deploying
to
benchmark-approval
August 13, 2026 19:36 — with
GitHub Actions
Error
GitHub Actions
Error
…2, clickhouse-i114640-w3, clickhouse-i114640-z1, clickhouse-i114640-z2 (c7a.metal-48xl)
Contributor
|
Results for Logs:
|
This was referenced Aug 13, 2026
alexey-milovidov
requested a deployment
to
benchmark-approval
August 13, 2026 20:15 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
Contributor
|
Results for Logs:
|
Member
Author
|
Investigation complete — closing. Summary of what these 20 runs established:
A suggestion for ClickBench from this exercise: have the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Investigation vehicle for ClickHouse/ClickHouse#114640 — the Q10/Q11 (
COUNT(DISTINCT UserID)) hot-time shift that daily mainline runs show on 192-physical-core machines (c7a.metal-48xl,c8g.metal-48xl) but not onc6a.metalor smaller.Analysis of the daily results narrowed the window to 26.5 → 26.6 (last fast run 2026-06-24 on 26.5, first slow run 2026-07-19 on 26.6), with the prime suspect being the
ThreadPoolLIFO wake-up change (ClickHouse/ClickHouse#100177, merged 2026-06-13, first released in 26.6.1). This PR adds three pinned systems to A/B exactly that merge on the affected hardware:clickhouse-i114640-a— v26.5.6.113 anchor (expected fast)clickhouse-i114640-pre— master @5719dd022fc(immediately before the LIFO merge)clickhouse-i114640-post— master @e78a4dac750(the LIFO merge itself)Runs are dispatched on
c7a.metal-48xlvia the manual workflow with this PR number, so results land here. Not intended to be merged — the branch will be closed after the investigation.