Skip to content

fix: skip HashJoinExec nodes carrying dynamic filters in JoinSelection - #26123

Open
siddubakka wants to merge 4 commits into
apache:mainfrom
siddubakka:fix-join-selection-dynamic-filter
Open

siddubakka wants to merge 4 commits into
apache:mainfrom
siddubakka:fix-join-selection-dynamic-filter

Conversation

@siddubakka

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

JoinSelection is not safe to run on plans that already carry dynamic filters. This happens during re-optimization passes (e.g. downstream pipelines that wrap an already-optimized plan into a writer sink and re-run physical optimization).

Once FilterPushdown has wired up a dynamic filter between the build side and probe side of a HashJoinExec, swapping the inputs via swap_inputs panics with:

Internal error: Cannot swap HashJoinExec inputs after dynamic filters have been constructed.

The join's build side is already committed at that stage, and swapping inputs would invalidate the dynamic filter expressions that reference probe-side columns. JoinSelection should detect this and leave the HashJoinExec unchanged instead of failing the plan.

What changes are included in this PR?

  • In JoinSelection::statistical_join_selection_subrule, check if !hash_join.dynamic_expressions_produced().is_empty() and return None (leaving the plan unchanged).
  • In can_swap_hash_join, guard against swapping when dynamic expressions are produced.
  • In hash_join_swap_subrule, guard against swapping unbounded left inputs when dynamic expressions are produced.
  • Added tests in datafusion/core/tests/physical_optimizer/join_selection.rs verifying that JoinSelection skips HashJoinExec carrying dynamic filters in both CollectLeft and Partitioned modes.

What is the testing strategy for this PR?

Added test_join_selection_skips_hash_join_with_dynamic_filter in datafusion/core/tests/physical_optimizer/join_selection.rs verifying that JoinSelection leaves the plan unchanged for both CollectLeft and Partitioned modes without error.

Are there any user-facing changes?

No API changes. Fixes an internal error when re-optimizing plans that contain dynamic filters.

apache#26106)

## Which issue does this PR close?

- Closes apache#26106.

## Rationale for this change

`JoinSelection` is not safe to run on plans that already carry dynamic
filters, which occurs during re-optimization passes (e.g. downstream
pipelines that wrap an already-optimized plan into a writer sink and
re-run physical optimization).

Once `FilterPushdown` has wired up a dynamic filter between the build
side and probe side of a `HashJoinExec`, swapping the inputs via
`swap_inputs` panics with:
`Internal error: Cannot swap HashJoinExec inputs after dynamic filter has been constructed`

The join's build side is already committed at that stage, and swapping
inputs would invalidate the dynamic filter expressions that reference
probe-side columns. `JoinSelection` should detect this and leave the
`HashJoinExec` unchanged instead of failing the query.

## What changes are included in this PR?

- In `JoinSelection::statistical_join_selection_subrule`, check if
  `!hash_join.dynamic_expressions_produced().is_empty()` and return
  `None` (leaving the plan unchanged).
- In `can_swap_hash_join`, guard against swapping when dynamic expressions
  are produced.
- In `hash_join_swap_subrule`, guard against swapping unbounded left inputs
  when dynamic expressions are produced.
- Added tests in `join_selection.rs` verifying that `JoinSelection` skips
  `HashJoinExec` carrying dynamic filters in both `CollectLeft` and
  `Partitioned` modes.

## What is the testing strategy for this PR?

Added `test_join_selection_skips_hash_join_with_dynamic_filter` in
`datafusion/core/tests/physical_optimizer/join_selection.rs` verifying
that `JoinSelection` leaves the plan unchanged for both `CollectLeft`
and `Partitioned` modes without error.

## Are there any user-facing changes?

No API changes. Fixes an internal error when re-optimizing plans that
contain dynamic filters.
@zhuqi-lucas

Copy link
Copy Markdown
Contributor

@siddubakka The CI has some fails, need to make it pass first.

@siddubakka

Copy link
Copy Markdown
Author

@zhuqi-lucas fixed the fmt and test failure, CI should be good now

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.83333% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.74%. Comparing base (3ed377a) to head (94976be).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...atafusion/physical-optimizer/src/join_selection.rs 70.83% 3 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26123      +/-   ##
==========================================
- Coverage   82.74%   82.74%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      449767   449771       +4     
  Branches   449767   449771       +4     
==========================================
  Hits       372160   372160              
- Misses      54938    54941       +3     
- Partials    22669    22670       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lit(true),
));

#[allow(deprecated)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This attribute makes the current clippy --all-targets --workspace ... -D warnings job fail: Clippy 1.99 denies clippy::allow_attributes and explicitly suggests #[expect]. Replacing it with #[expect(deprecated)] preserves the intended scoped deprecation handling and also warns if the deprecation is later removed.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mikamikasuki thank you for your help i have made the changes now

@siddubakka

siddubakka commented Oct 8, 2026 •

Copy link
Copy Markdown
Author

@zhuqi-lucas fixed the clippy failure and the codecov issue (removed an unreachable branch that was dropping coverage). also added a test for it, so everything should be passing now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JoinSelection should skip HashJoinExec nodes that already carry a dynamic filter instead of failing in swap_inputs

4 participants