Skip to content

Remove schema alignment constraint from EquivalenceProperties::with_new_schema #20232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Sahitya0805 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Remove schema alignment constraint from EquivalenceProperties::with_new_schema #20232

Sahitya0805 wants to merge 3 commits into from
+173 −57

Conversation

@Sahitya0805
Copy link

@Sahitya0805 Sahitya0805 commented Feb 9, 2026 •
edited
Loading

Fixes #18933.

This PR removes the strict schema alignment constraint from EquivalenceProperties::with_new_schema and OrderingEquivalenceClass::with_new_schema.

Previously, these methods required schemas to be 'aligned' (same number of columns, same types at each position), which made it difficult to update properties when schemas evolved (e.g., in projections or join operations).

Key changes:

  • EquivalenceProperties: Now matches columns by name and index using the expression rewriter. If a column is missing in the new schema, it is gracefully dropped from its equivalence class while keeping the rest of the group.
  • OrderingEquivalenceClass: Orderings are now truncated at the first column that cannot be mapped to the new schema, ensuring that prefix orderings are preserved.
  • Improved Robustness: This change allows DataFusion to maintain as much physical property information as possible through schema transformations (renames, appends, and removals).

Added regression tests to verify renames, appends, and column removals.

@Sahitya0805
feat: Add SortOptions builder for better Expr::sort API
4f32396 
Introduces a new SortOptions struct with fluent builder methods
to eliminate boolean blindness in sort expressions.

Changes:
- Add SortOptions struct with desc(), asc(), nulls_first(), nulls_last()
- Add Expr::sort_by() method accepting SortOptions
- Keep existing Expr::sort() for backward compatibility
- Add test demonstrating both old and new APIs

Fixes apache#20227
@Sahitya0805
Make SortOptions::new() explicit about default values
93dd5ca 
Address review feedback to make the default values explicit
instead of hiding them behind Self::default().
@Sahitya0805
Remove schema alignment constraint from EquivalenceProperties::with_n…
7641268 
…ew_schema
@github-actions github-actions bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate labels Feb 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

core Core DataFusion crate logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Make EquivalenceProperties::new_with_schema not require aligned schema

1 participant

@Sahitya0805