Skip to content

[feature](query) support max_visible_partition() syntax - #66187

Open
re20052 wants to merge 1 commit into
apache:masterfrom
re20052:feature-max-visable-partition
Open

[feature](query) support max_visible_partition() syntax#66187
re20052 wants to merge 1 commit into
apache:masterfrom
re20052:feature-max-visable-partition

Conversation

@re20052

@re20052 re20052 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:
Support a new max_visible_partition() syntax after the table name so a query can be automatically pruned to the latest partition that has visible data, without hardcoding the partition name.

The common workaround where pt = (select max(pt) from t) requires an extra sub-query, adding overhead in the Plan / Schedule / Execute phases. This syntax lets the engine resolve the target partition during Nereids binding and reuse the existing partition-pruning path, so the change is confined to FE and does not touch the execution layer.

  • RANGE table: prune to the partition with the greatest range upper bound that has visible data (physical comparison on the upper endpoint via PartitionKey.compareTo, type-aware for numbers/dates, not by partition name).
  • UNPARTITIONED table: return the single partition if it has data, otherwise empty.
  • LIST table: no natural "latest" partition, return empty result.
  • Only OLAP table and materialized view are supported; other table types (View / external tables) are rejected explicitly at bind time.
  • When no visible partition is found, an empty query (LogicalEmptyRelation) is built instead of throwing or falling back to a full scan.

Release note

Support max_visible_partition() syntax to query the latest visible partition.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

    Regression suite: regression-test/suites/nereids_syntax_p0/max_visible_partition.groovy, covering RANGE pruning, materialized view, View rejection, and UNPARTITIONED / LIST empty-result cases.
    Manual test on cluster doris_ad_ttam_my3 (table ad_stats_test.new_uid_sketch_creative_hourly): sub-query form 594ms / 4 fragments vs max_visible_partition() 27ms / 2 fragments.

  • Behavior changed:

    • No.
    • Yes. Adds a new optional max_visible_partition() clause after the table name. MAX_VISIBLE_PARTITION is a non-reserved keyword, so existing SQL is unaffected.

  • Does this need documentation?

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@re20052
[feature](query) support max_visible_partition() syntax
1de88be 
### What problem does this PR solve?

Problem Summary:
Support a new `max_visible_partition()` syntax after the table name so a query
can be automatically pruned to the latest partition that has visible data,
without hardcoding the partition name.

- RANGE table: prune to the partition with the greatest range upper bound that
  has visible data.
- UNPARTITIONED table: return the single partition if it has data, otherwise empty.
- LIST table: no natural "latest" partition, return empty result.
- Only OLAP table and materialized view are supported; other table types are
  rejected explicitly.

### Release note

Support `max_visible_partition()` syntax to query the latest visible partition.
@re20052

re20052 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

/review

@re20052

re20052 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Jul 28, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29463 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1de88be1a05b3e31669f0a4ee630eb3f4c0d0cc2, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17590	4115	4082	4082
q2	2065	328	198	198
q3	10239	1424	843	843
q4	4678	467	339	339
q5	7495	868	598	598
q6	178	169	136	136
q7	739	811	600	600
q8	9357	1648	1557	1557
q9	5571	4318	4343	4318
q10	6729	1713	1455	1455
q11	502	357	331	331
q12	734	582	444	444
q13	18135	3332	2765	2765
q14	269	254	234	234
q15	q16	786	771	706	706
q17	934	988	1024	988
q18	7054	5593	5490	5490
q19	1214	1271	1130	1130
q20	815	670	603	603
q21	5749	2655	2352	2352
q22	425	359	294	294
Total cold run time: 101258 ms
Total hot run time: 29463 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4364	4296	4323	4296
q2	299	320	211	211
q3	4515	4939	4374	4374
q4	2056	2142	1350	1350
q5	4348	4248	4251	4248
q6	224	176	126	126
q7	1709	1587	1852	1587
q8	2700	2188	2181	2181
q9	7892	8047	7738	7738
q10	4668	4641	4204	4204
q11	567	409	379	379
q12	770	778	554	554
q13	3255	3597	2918	2918
q14	296	301	284	284
q15	q16	723	735	628	628
q17	1335	1332	1330	1330
q18	7990	7470	7094	7094
q19	1173	1105	1093	1093
q20	2212	2187	1915	1915
q21	5172	4520	4351	4351
q22	501	434	407	407
Total cold run time: 56769 ms
Total hot run time: 51268 ms

@hello-stephen

hello-stephen commented Jul 28, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 176586 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1de88be1a05b3e31669f0a4ee630eb3f4c0d0cc2, data reload: false

query5	4357	626	474	474
query6	459	214	217	214
query7	4844	576	345	345
query8	327	182	164	164
query9	8793	4000	4001	4000
query10	484	388	306	306
query11	5926	2311	2159	2159
query12	159	101	103	101
query13	1273	619	425	425
query14	6245	5204	4863	4863
query14_1	4224	4205	4221	4205
query15	214	205	178	178
query16	971	453	448	448
query17	954	719	583	583
query18	2461	483	354	354
query19	210	189	158	158
query20	112	109	112	109
query21	241	157	134	134
query22	13570	13509	13337	13337
query23	17297	16422	16123	16123
query23_1	16203	16264	16272	16264
query24	7459	1743	1278	1278
query24_1	1300	1281	1280	1280
query25	557	450	379	379
query26	1372	347	228	228
query27	2565	591	384	384
query28	4443	1978	1975	1975
query29	1080	614	491	491
query30	352	264	228	228
query31	1136	1088	969	969
query32	101	64	63	63
query33	522	330	252	252
query34	1186	1121	632	632
query35	766	788	669	669
query36	1042	1024	855	855
query37	157	113	103	103
query38	1908	1675	1656	1656
query39	872	885	829	829
query39_1	823	833	834	833
query40	245	163	143	143
query41	64	62	62	62
query42	91	89	87	87
query43	320	325	299	299
query44	1405	758	758	758
query45	195	181	171	171
query46	1046	1158	717	717
query47	2092	2067	1994	1994
query48	397	408	273	273
query49	563	403	310	310
query50	1070	432	343	343
query51	10539	10675	10805	10675
query52	87	85	69	69
query53	265	260	197	197
query54	285	237	208	208
query55	72	69	64	64
query56	301	304	284	284
query57	1297	1255	1205	1205
query58	277	265	231	231
query59	1568	1642	1385	1385
query60	322	272	248	248
query61	147	149	147	147
query62	539	496	426	426
query63	248	198	198	198
query64	2826	1038	824	824
query65	4707	4635	4621	4621
query66	1813	541	378	378
query67	29261	29089	29068	29068
query68	3130	1490	1010	1010
query69	426	315	261	261
query70	867	830	784	784
query71	366	327	327	327
query72	3036	2663	2335	2335
query73	835	761	417	417
query74	5078	4935	4683	4683
query75	2521	2484	2136	2136
query76	2303	1164	760	760
query77	349	376	276	276
query78	11860	11852	11276	11276
query79	1419	1124	746	746
query80	1187	562	463	463
query81	526	335	295	295
query82	581	149	118	118
query83	366	320	292	292
query84	329	155	131	131
query85	972	595	507	507
query86	380	233	222	222
query87	1826	1807	1745	1745
query88	3643	2746	2731	2731
query89	428	382	331	331
query90	1807	195	198	195
query91	197	189	154	154
query92	61	57	56	56
query93	1625	1546	927	927
query94	656	332	314	314
query95	802	618	488	488
query96	1118	771	334	334
query97	2613	2606	2473	2473
query98	210	214	198	198
query99	1098	1102	966	966
Total cold run time: 261881 ms
Total hot run time: 176586 ms

@hello-stephen

hello-stephen commented Jul 28, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1de88be1a05b3e31669f0a4ee630eb3f4c0d0cc2, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.09	0.05
query3	0.26	0.14	0.13
query4	1.61	0.12	0.13
query5	0.24	0.22	0.22
query6	1.28	1.07	1.05
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.38	0.32	0.31
query10	0.55	0.55	0.55
query11	0.19	0.13	0.13
query12	0.17	0.14	0.14
query13	0.47	0.46	0.47
query14	0.99	1.00	0.99
query15	0.64	0.60	0.60
query16	0.31	0.34	0.32
query17	1.12	1.12	1.10
query18	0.22	0.20	0.20
query19	2.10	1.98	1.97
query20	0.02	0.01	0.02
query21	15.43	0.19	0.14
query22	4.95	0.05	0.06
query23	16.15	0.30	0.12
query24	2.99	0.40	0.30
query25	0.12	0.05	0.04
query26	0.73	0.21	0.14
query27	0.05	0.05	0.04
query28	3.52	0.91	0.52
query29	12.49	4.02	3.27
query30	0.26	0.15	0.16
query31	2.77	0.59	0.31
query32	3.22	0.59	0.50
query33	3.25	3.24	3.15
query34	15.61	4.15	3.51
query35	3.52	3.50	3.48
query36	0.56	0.43	0.42
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.21	0.17	0.16
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.93 s
Total hot run time: 24.92 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner
@starocean999 starocean999 Awaiting requested review from starocean999 starocean999 is a code owner
@924060929 924060929 Awaiting requested review from 924060929 924060929 is a code owner
@englefly englefly Awaiting requested review from englefly englefly is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@re20052 @hello-stephen