-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Clean up dynamic filter configs #27637
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clean up dynamic filter configs #27637
Conversation
146341a to
dfe2b8a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR cleans up dynamic filter configuration by removing the deprecated enable-large-dynamic-filters configuration property and dynamic-filtering.small.* settings, while renaming dynamic-filtering.large.* properties to dynamic-filtering.*. Large dynamic filters have been the default behavior for a while, making the small/large distinction unnecessary.
Key changes:
- Removed
enable-large-dynamic-filtersconfiguration property and corresponding session property - Removed all
dynamic-filtering.small.*configuration properties (now marked as defunct) - Renamed
dynamic-filtering.large.*todynamic-filtering.*with proper legacy config support - Simplified code by removing conditional logic based on filter size mode
- Updated tests to use new configuration property names with appropriate limits
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
core/trino-main/src/main/java/io/trino/execution/DynamicFilterConfig.java |
Removed small/large filter config fields, renamed large to default, added defunct and legacy config annotations for backward compatibility |
core/trino-main/src/main/java/io/trino/SystemSessionProperties.java |
Removed ENABLE_LARGE_DYNAMIC_FILTERS constant, session property definition, and accessor method |
core/trino-main/src/main/java/io/trino/sql/planner/LocalExecutionPlanner.java |
Removed small filter fields, simplified methods to no longer check session for filter size mode |
core/trino-main/src/main/java/io/trino/server/DynamicFilterService.java |
Removed small filter size limit field and conditional size limit selection logic |
core/trino-main/src/test/java/io/trino/execution/TestDynamicFilterConfig.java |
Updated test to validate only the new configuration properties without small/large distinction |
core/trino-main/src/test/java/io/trino/server/TestDynamicFilterService.java |
Removed session property setup and config method calls for small filters |
testing/trino-tests/src/test/java/io/trino/execution/TestCoordinatorDynamicFiltering.java |
Updated config properties from small to default names and added explicit range row limits |
testing/trino-testing/src/main/java/io/trino/testing/BaseDynamicPartitionPruningTest.java |
Updated config property names from large-partitioned to partitioned, removed session property setup |
testing/trino-testing/src/main/java/io/trino/testing/AbstractDistributedEngineOnlyQueries.java |
Removed session builder setup for large dynamic filters in test |
testing/trino-faulttolerant-tests/src/test/java/io/trino/faulttolerant/TestFaultTolerantExecutionDynamicFiltering.java |
Updated config properties from small to default names |
plugin/trino-memory/src/test/java/io/trino/plugin/memory/TestMemoryConnectorTest.java |
Consolidated small/large configs into single set, removed withLargeDynamicFilters helper method, updated test expectations to reflect large filters always enabled |
docs/src/main/sphinx/admin/dynamic-filtering.md |
Updated documentation to remove references to small/large filter distinction and enable-large-dynamic-filters property |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
dfe2b8a to
ce8ea0c
Compare
Large dynamic filters have been the default for a while The configs around small dynamic filter limits can be removed now
Since distinction between small and large configs has been removed, these configs are renamed to drop the reference to large
ce8ea0c to
937fece
Compare
sopel39
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that makes a lot of sense, thanks
| return smallPartitionedMaxDistinctValuesPerDriver; | ||
| } | ||
|
|
||
| @Config("dynamic-filtering.small-partitioned.max-distinct-values-per-driver") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't there be a deprecation period during which config still works but using it triggers a warning?
cc @kokosing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not when we're outright removing some functionality.
The "small" configs became irrelevant when large DFs were enabled by default more than an year back.
Description
Remove
enable-large-dynamic-filtersanddynamic-filtering.small.*configuration properties since large dynamic filters have been the default for a while.Rename
dynamic-filtering.large.*configs to droplargesince the small/large distinction is no longer relevant.Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: