Skip to content

fix: allow empty V1 writes with non-nullable columns - #8394

Merged
Xuanwo merged 1 commit into
mainfrom
gatekeeper/fix-1914-1
Aug 9, 2026
Merged

fix: allow empty V1 writes with non-nullable columns#8394
Xuanwo merged 1 commit into
mainfrom
gatekeeper/fix-1914-1

Conversation

@lance-gatefixer

@lance-gatefixer lance-gatefixer Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • keep V1 page-statistic bounds nullable when the source column is non-nullable
  • cover writing and reading an empty batch with a non-nullable string column

Root cause

The V1 statistics collector copied source-column nullability onto its min_value and max_value fields. Empty string pages legitimately have no bounds, so Arrow rejected their null statistics values while constructing the non-nullable stats struct.

Scope

This removes the panic when creating an empty V1 table. IVF_PQ index creation still requires vectors for centroid training and will continue to reject an empty dataset.

Validation

  • cargo fmt --all -- --check
  • cargo test -p lance-file (153 unit tests and 4 doctests)
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #1914

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer bug Something isn't working labels Aug 7, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The change fixes the V1 empty-batch panic at the statistics-schema boundary: min/max bounds may be absent even when the data column is non-nullable. Keeping those bounds nullable uses the existing V1 statistics grammar and preserves one statistics row per data batch.

@cursor

cursor Bot commented Aug 9, 2026

Copy link
Copy Markdown

Review: LGTM — approve & merge

Empty V1 pages can have null min/max even when the data column is non-nullable; copying field nullability into the stats struct correctly panics on empty batches. Making stats bounds always nullable is the right V1-stats-schema fix and is backward compatible for readers of older files. Empty non-nullable Utf8 round-trip regression is sufficient.

@Xuanwo
Xuanwo merged commit 41be024 into main Aug 9, 2026
43 checks passed
@Xuanwo
Xuanwo deleted the gatekeeper/fix-1914-1 branch August 9, 2026 09:27
@Xuanwo Xuanwo added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic when creating empty table

1 participant