fix: return error instead of panicking on zero-dimension fixed-size-list columns by DanielMao1 · Pull Request #7247 · lance-format/lance

DanielMao1 · 2026-06-12T09:09:53Z

Problem

A fixed-size-list column with dimension 0 panics with attempt to divide by zero
(rust/lance-encoding/src/data.rs, FixedSizeListBlock::num_values). As of pylance 7.0.0
the panic fires on write for every storage version (stable/2.1/2.2), and reading
datasets persisted by older writers (which accepted such columns) panics as well.

Reproduction details are in the issue comment:
#5102 (comment)

Approach

Following the maintainer guidance in #5102 (error, not panic), this adds two small guards at
boundaries that already return Result, instead of changing DataBlock::num_values() to
return Result (the approach that made #5159 balloon across the whole encoding crate):

Write side: Schema::validate() rejects zero-dimension fixed-size-list fields
(including nested ones). validate() runs inside Schema::try_from(&ArrowSchema),
so every write entry point surfaces a clean schema error instead of a panic. Writes
currently panic on every storage version, so no working flow changes behavior.
Read side (defensive): the structural and legacy field-scheduler builders reject
zero-dimension fixed-size lists with an invalid-input error, so datasets persisted by
old writers fail cleanly at scheduling time instead of crashing the process.

How the guards sit in the data flow

Two facts that shape the design:

Schema::try_from(&ArrowSchema) calls validate() internally and every write path performs
this conversion, so guard 1 in one place covers all write entry points.
Guard 2 exists because writers up to ~2026-04 could still persist zero-dimension columns
under the stable (2.0) storage version; reading those files must not crash the process.

Tests

lance-core: Schema::try_from rejects zero-dim FSL at top level and nested in a struct;
positive dimensions still validate.
lance-encoding: the scheduler guard rejects zero-dim FSL, including FSL-nested-in-FSL,
and accepts positive dimensions.
Python: parametrized over legacy/stable/2.1, write_dataset now raises a clean
OSError (same mapping as other schema validation errors) instead of PanicException.

codecov · 2026-06-12T10:24:32Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

yanghua

Left two comments.

yanghua · 2026-06-13T02:23:51Z



+@pytest.mark.parametrize("data_storage_version", ["legacy", "stable", "2.1"])
+def test_write_zero_dimension_fixed_size_list(


Can we at least add one test case about reading existed zero dimension by the old writer logic?

Added as a Rust test rather than Python: once Schema::validate() rejects zero-dim FSL, no write path can produce such a dataset (and the encoder panics on zero-dim), so the fixture can't be generated by current code.
Instead, test_read_zero_dimension_fsl_errors_instead_of_panicking (in decoder.rs) drives create_structural_field_scheduler / create_legacy_field_scheduler — the read-plan builders the file reader calls against a stored schema — with a zero-dim FSL field, asserting a clean error instead of a panic.

yanghua · 2026-06-13T02:28:31Z

+            if let DataType::FixedSizeList(_, dimension) = field.data_type()
+                && dimension <= 0


Does it work for the FSL of FSL mode, e.g. FixedSizeList(FixedSizeList(Float32, 0), 4)?

Good catch — For my previous PR it did not. Because a FixedSizeList of a FixedSizeList over a primitive collapses into a single leaf field, the pre-order field walk never visits the inner list, and the original if let DataType::FixedSizeList(_, dimension) only matched the outermost dimension. So FixedSizeList(FixedSizeList(Float32, 0), 4) slipped through (outer dim 4 passed, inner dim 0 ignored).

I confirmed this with a probe test, then fixed it: the check now lives in a helper validate_fixed_size_list_dimensions() that recurses through nested list types, so an inner zero dimension is rejected too. Added FixedSizeList(FixedSizeList(Float32, 0), 4) (and a positive-dimension nested counterpart) to the schema test.

yanghua

Left two comments.

yanghua · 2026-06-16T02:48:13Z

+    /// so letting such a type through panics with a divide-by-zero during
+    /// scheduling. Such datasets can exist on disk because old writers did not
+    /// validate the dimension (see issue #5102).
+    fn validate_fsl_dimension(field: &Field, data_type: &DataType) -> Result<()> {


Is it duplicated with validate_fixed_size_list_dimensions?

Right，it may be deduplicated. Promoted validate_fixed_size_list_dimensions in lance-core to pub and call it from the decoder's scheduler builders, removing the duplicate validate_fsl_dimension. Both write and read paths now share one implementation.

yanghua · 2026-06-16T02:49:44Z

+    fn validate_fsl_dimension(field: &Field, data_type: &DataType) -> Result<()> {
+        if let DataType::FixedSizeList(inner, dimension) = data_type {
+            if *dimension <= 0 {
+                return Err(Error::invalid_input(format!(


Does Error::schema look better?

Agreed. The shared lance-core helper uses Error::schema, so after the dedup the read path now returns Error::schema too instead of invalid_input — consistent with the write path.

yanghua

+1 cc @Xuanwo

…ist columns A fixed-size-list column with dimension 0 used to panic with 'attempt to divide by zero' (rust/lance-encoding/src/data.rs) on every write path and when reading datasets persisted by older writers. Reject such columns with a clean schema error, following the maintainer guidance in lance-format#5102 (error, not panic). A shared helper `validate_fixed_size_list_dimensions` (in lance-core) is used on both paths: - Write path: called from Schema::validate(), which runs inside Schema::try_from(&ArrowSchema), so all write entry points are covered. - Read path: called from the decoder field-scheduler builders, so datasets persisted by old writers fail cleanly instead of panicking at scheduling time. A FixedSizeList of a FixedSizeList over a primitive collapses into a single leaf field, so the helper recurses through nested list types to also reject an inner zero dimension, e.g. FixedSizeList(FixedSizeList(Float32, 0), 4). Closes lance-format#5102

github-actions Bot added A-python Python bindings A-encoding Encoding, IO, file reader/writer bug Something isn't working labels Jun 12, 2026

DanielMao1 force-pushed the fix/zero-dim-fsl-panic branch 2 times, most recently from 143f1e1 to 240a45a Compare June 12, 2026 09:26

yanghua reviewed Jun 13, 2026

View reviewed changes

DanielMao1 force-pushed the fix/zero-dim-fsl-panic branch 2 times, most recently from 7827869 to 4d0d4b5 Compare June 15, 2026 11:02

DanielMao1 requested a review from yanghua June 15, 2026 11:13

yanghua reviewed Jun 16, 2026

View reviewed changes

DanielMao1 force-pushed the fix/zero-dim-fsl-panic branch 2 times, most recently from 002c5a6 to e1041bb Compare June 16, 2026 03:40

yanghua approved these changes Jun 16, 2026

View reviewed changes

DanielMao1 force-pushed the fix/zero-dim-fsl-panic branch from e1041bb to d758cd1 Compare June 16, 2026 05:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: return error instead of panicking on zero-dimension fixed-size-list columns#7247

fix: return error instead of panicking on zero-dimension fixed-size-list columns#7247
DanielMao1 wants to merge 1 commit into
lance-format:mainfrom
DanielMao1:fix/zero-dim-fsl-panic

DanielMao1 commented Jun 12, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

yanghua left a comment

Uh oh!

yanghua Jun 13, 2026

Uh oh!

DanielMao1 Jun 15, 2026 •

edited

Loading

Uh oh!

yanghua Jun 13, 2026

Uh oh!

DanielMao1 Jun 15, 2026

Uh oh!

yanghua left a comment

Uh oh!

yanghua Jun 16, 2026

Uh oh!

DanielMao1 Jun 16, 2026

Uh oh!

yanghua Jun 16, 2026

Uh oh!

DanielMao1 Jun 16, 2026

Uh oh!

yanghua left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		@pytest.mark.parametrize("data_storage_version", ["legacy", "stable", "2.1"])
		def test_write_zero_dimension_fixed_size_list(

		if let DataType::FixedSizeList(_, dimension) = field.data_type()
		&& dimension <= 0

Conversation

DanielMao1 commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Approach

How the guards sit in the data flow

Tests

Uh oh!

codecov Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yanghua left a comment

Choose a reason for hiding this comment

Uh oh!

yanghua Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

DanielMao1 Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yanghua Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

DanielMao1 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

yanghua left a comment

Choose a reason for hiding this comment

Uh oh!

yanghua Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

DanielMao1 Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

yanghua Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

DanielMao1 Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

yanghua left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DanielMao1 commented Jun 12, 2026 •

edited

Loading

codecov Bot commented Jun 12, 2026 •

edited

Loading

DanielMao1 Jun 15, 2026 •

edited

Loading