fix(arrow/extensions): reject Null Variant typed_value - #1243
Conversation
A Null typed_value column is not a valid shredded type. Variant nulls belong in the value column. Fixes apache#1205 Signed-off-by: Digvijay <digvijay.vaghela@yahoo.com>
zeroshade
left a comment
There was a problem hiding this comment.
The direct Null typed_value rejection is incomplete in two ways.
First, NewShreddedVariantType(arrow.Null) now silently returns nil because its call to NewVariantType discards the new validation error. Second, nested typed_value fields can still wrap Null storage in an extension and bypass the new checks. Both cases were reproduced; details are inline.
Extension, race, vet, and focused pqarrow Variant tests otherwise pass. This head currently has no CI results.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
Treat Null like a missing shred type so NewShreddedVariantType never returns nil. Unwrap extension storage before nested Null checks. Signed-off-by: Digvijay <digvijay.vaghela@yahoo.com>
zeroshade
left a comment
There was a problem hiding this comment.
The two previously reported basic cases are fixed: bare Null now produces a usable default shredded type, and a single extension layer backed by Null is rejected in both nested forms.
Two related malformed/recursive extension cases remain. Recursive wrapping can still bypass validation and later panic during builder creation, while a public opaque extension with nil storage panics during validation itself. Details and fix directions are inline.
Extensions tests, race tests, and vet otherwise pass. CI is still running, and the two existing threads remain unresolved.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
|
Pushed a follow-up for the remaining unwrap cases.
@zeroshade this should be ready for another look. |
Reject double-wrapped Null and nil Opaque storage instead of panicking or silently constructing an invalid type. Signed-off-by: Digvijay <digvijay.vaghela@yahoo.com>
28c7d42 to
a5adf0e
Compare
zeroshade
left a comment
There was a problem hiding this comment.
LGTM jsut waiting for CI to complete
Rationale for this change
A Null
typed_valuecolumn is not a valid shredded Variant type. Nulls are encoded in thevaluecolumn as Variant null (00). A dedicated Nulltyped_valuewould also make “field present and null” indistinguishable from “field missing” for shredded objects.See apache/arrow#50622 and apache/arrow#50810.
Fixes #1205
What changes are included in this PR?
NewVariantTyperejects a Nulltyped_valuefieldtyped_valueare also rejectedAre these changes tested?
go test ./arrow/extensionsAre there any user-facing changes?
Yes — constructing a Variant extension type with a Null
typed_valuenow returnsarrow.ErrInvalid.