Skip to content

fix(arrow/extensions): reject Null Variant typed_value - #1243

Merged
zeroshade merged 3 commits into
apache:mainfrom
dgvj-work:fix/variant-null-typed-value
Sep 2, 2026
Merged

zeroshade merged 3 commits into
apache:mainfrom
dgvj-work:fix/variant-null-typed-value

Conversation

@dgvj-work

@dgvj-work dgvj-work commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

A Null typed_value column is not a valid shredded Variant type. Nulls are encoded in the value column as Variant null (00). A dedicated Null typed_value would also make “field present and null” indistinguishable from “field missing” for shredded objects.

See apache/arrow#50622 and apache/arrow#50810.

Fixes #1205

What changes are included in this PR?

  • NewVariantType rejects a Null typed_value field
  • Nested shredded fields with a Null typed_value are also rejected

Are these changes tested?

  • go test ./arrow/extensions

Are there any user-facing changes?

Yes — constructing a Variant extension type with a Null typed_value now returns arrow.ErrInvalid.

A Null typed_value column is not a valid shredded type. Variant nulls
belong in the value column.

Fixes apache#1205

Signed-off-by: Digvijay <digvijay.vaghela@yahoo.com>

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The direct Null typed_value rejection is incomplete in two ways.

First, NewShreddedVariantType(arrow.Null) now silently returns nil because its call to NewVariantType discards the new validation error. Second, nested typed_value fields can still wrap Null storage in an extension and bypass the new checks. Both cases were reproduced; details are inline.

Extension, race, vet, and focused pqarrow Variant tests otherwise pass. This head currently has no CI results.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

Comment thread arrow/extensions/variant.go
Comment thread arrow/extensions/variant.go Outdated
Treat Null like a missing shred type so NewShreddedVariantType never
returns nil. Unwrap extension storage before nested Null checks.

Signed-off-by: Digvijay <digvijay.vaghela@yahoo.com>

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two previously reported basic cases are fixed: bare Null now produces a usable default shredded type, and a single extension layer backed by Null is rejected in both nested forms.

Two related malformed/recursive extension cases remain. Recursive wrapping can still bypass validation and later panic during builder creation, while a public opaque extension with nil storage panics during validation itself. Details and fix directions are inline.

Extensions tests, race tests, and vet otherwise pass. CI is still running, and the two existing threads remain unresolved.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

Comment thread arrow/extensions/variant.go Outdated
Comment thread arrow/extensions/variant.go Outdated
@dgvj-work

Copy link
Copy Markdown
Contributor Author

Pushed a follow-up for the remaining unwrap cases.

storageType now walks extension layers (with cycle protection) and is nil-safe. Double-wrapped Null is rejected in the direct, one-field, two-field, and shredded paths. NewVariantType(NewOpaqueType(nil, ...)) returns arrow.ErrInvalid, and NewShreddedVariantType still returns the default type instead of panicking.

@zeroshade this should be ready for another look.

@dgvj-work
dgvj-work requested a review from zeroshade September 2, 2026 04:20
Reject double-wrapped Null and nil Opaque storage instead of
panicking or silently constructing an invalid type.

Signed-off-by: Digvijay <digvijay.vaghela@yahoo.com>
@dgvj-work
dgvj-work force-pushed the fix/variant-null-typed-value branch from 28c7d42 to a5adf0e Compare September 2, 2026 04:25

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM jsut waiting for CI to complete

@zeroshade
zeroshade merged commit d24b83e into apache:main Sep 2, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Parquet] Variant Null type typed_value field is supported

2 participants