Repository navigation
feat(scan): read Parquet files by task range - #1032
Closed
kamcheungting-db wants to merge 6 commits into
Closed
kamcheungting-db wants to merge 6 commits into
kamcheungting-db wants to merge 6 commits into
Conversation
FileScanTaskReader previously rejected tasks that read part of a file. It now passes Parquet task ranges to the reader while keeping the full file size available for reading the footer. Use the first column's starting page to assign each row group to a range. This works when optional row-group offsets are missing and ensures that adjacent ranges read every row exactly once. Keep row positions and inherited row IDs relative to the whole file so deletes still remove the correct rows, even when page offsets are out of order. Recheck task bounds when opening a file, skip empty row groups, and reject invalid page offsets and partial tasks for unsupported formats. Add tests for split boundaries, deletes, row lineage, changed file sizes, and files with missing row-group offsets. Validation: all 306 tests in parquet_test and data_test pass. Formatting and whitespace checks pass. pre-commit could not run because it is not installed and PyPI DNS resolution failed during installation.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…alled InitReadContext advanced to the next row-group range before the output schema was built. If schema setup failed, a retried Next() skipped the first range. Opening a range no longer consumes it; ConsumeRowGroupRange() does that once the reader is in place. Co-authored-by: Isaac <no-reply@databricks.com>
Java Iceberg reads a split through parquet-java's withRange(), which keeps a row group when its midpoint (first-column start plus half the compressed size) falls in [start, start + length). Use the same rule so C++ and Java readers pick the same row groups for splits that cut through a row group. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allows
FileScanTaskReaderto read Parquet byte ranges created by the task-range API in #976. Each split selects row groups by midpoint, matching Java Iceberg, while_posand_row_idremain relative to the whole file so position deletes and row lineage continue to work.Supersedes #1002. Based on
main, which includes #976.The reader handles missing row-group offsets and noncontiguous selections, skips empty row groups, and validates page offsets and compressed sizes. Task bounds are checked again when opening; partial reads for other formats return
NotSupported. Removes the redundantFileScanTask::Range::file_sizefield and uses the data file's size for row estimates, with a guard for nonpositive sizes.Tests cover split coverage, midpoint boundaries, whole-file row positions and row IDs, deletes, missing and out-of-order offsets, invalid metadata, and changed file sizes.
Validation:
scan_testanddata_test; both suites passed (329 tests passed, 10 skipped).clang-format --dry-run --Werrorwith clang-format 20.1.8;git diff --checkpassed.parquet_testtarget is blocked by an unchanged test frommain:row_number_reader_test.cc:192usesstd::ranges::iota, which the local clang 18/libc++ toolchain does not provide. The Parquet suite has not run on this rebased version.pre-commit; the tool is unavailable locally and installation failed because PyPI DNS resolution failed, so its hooks have not run.