Skip to content

fix: ignore exact empty files in ordering analysis - #24648

Open
xudong963 wants to merge 1 commit into
apache:mainfrom
xudong963:fix/ignore-empty-files-in-ordering
Open

fix: ignore exact empty files in ordering analysis#24648
xudong963 wants to merge 1 commit into
apache:mainfrom
xudong963:fix/ignore-empty-files-in-ordering

Conversation

@xudong963

@xudong963 xudong963 commented Aug 25, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

A file with exactly zero rows cannot violate a sort order, but schema-only Parquet files normally have neither sorting_columns ordering nor column min/max values. Treating those missing values as evidence against the non-empty files can add an unnecessary SortExec; with statistics-based file-group splitting enabled, it can also fail planning with statistics not found.

The Parquet footer row count is already available during listing, so exact empty files can be ignored without additional I/O. Files with absent or inexact row counts remain conservative.

What changes are included in this PR?

  • Ignore files with num_rows == Precision::Exact(0) when deriving a common file ordering.
  • Exclude exact empty files from MinMaxStatistics and sort-column NULL checks.
  • Preserve the mapping from min/max statistics rows to the original input file indices, so filtering an empty file cannot shift or misidentify later files.
  • Support the all-files-empty case with correctly typed zero-length arrays.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes. Scans over ordered Parquet datasets can retain their output ordering when exact empty files are present, avoiding an unnecessary sort, and statistics-based file-group splitting no longer fails on those files. There are no public API changes.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) catalog Related to the catalog crate datasource Changes to the datasource crate labels Aug 25, 2026
@xudong963
xudong963 force-pushed the fix/ignore-empty-files-in-ordering branch from fae6594 to aa07f9d Compare August 25, 2026 08:25
@xudong963
xudong963 marked this pull request as draft August 25, 2026 08:33
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.06299% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.44%. Comparing base (3ace5a0) to head (aa07f9d).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/datasource/src/statistics.rs 93.97% 0 Missing and 5 partials ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #24648    +/-   ##
========================================
  Coverage   81.44%   81.44%            
========================================
  Files        1118     1118            
  Lines      399560   399671   +111     
  Branches   399560   399671   +111     
========================================
+ Hits       325424   325526   +102     
- Misses      55137    55142     +5     
- Partials    18999    19003     +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xudong963
xudong963 marked this pull request as ready for review August 25, 2026 09:37
@xudong963

Copy link
Copy Markdown
Member Author

CC @asolimando in case you have time to review the stats improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalog Related to the catalog crate datasource Changes to the datasource crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Empty Parquet files invalidate file ordering and statistics-based grouping

2 participants