Typed column access with As<> / AsStrict<> - #541
Draft
slabko wants to merge 13 commits into
Draft
Conversation
ColumnArrayT/ColumnNullableT/ColumnTupleT/ColumnMapT::Wrap used to steal the source column's internals (rvalue-ref parameter consumed via std::move), leaving the original unusable. Wrap now shares the source's internals through shared_ptr: the returned typed column references the same underlying data, the original stays fully valid, and mutations are visible through both. All Wrap overloads take their argument by const reference (const <Column>& / const Column& / const ColumnRef&), so std::move is no longer required and lvalues and const sources are accepted. - nullable: wrap the nested column via WrapColumn (recursive share); include utils.h for it - array: share nested data and offsets - tuple: build element columns from the source via TupleFromColumn - map: share the backing array of key/value tuples - update doc-comments to describe the shared, non-destructive semantics - add tests for non-stealing behaviour and lvalue/const acceptance ColumnLowCardinalityT::Wrap is converted in the following commit.
Share the dedup map (unique_items_map_) via shared_ptr so a wrapped LowCardinality column shares dictionary, index and map with its source, keeping them coherent across both holders. Relax Wrap to const& (like the other typed columns) so it no longer steals from the source. Add tests covering non-stealing/shared-coherent semantics and lvalue acceptance.
Now that every wrappable column (including ColumnLowCardinalityT) exposes Wrap(const ColumnRef&), the WrapColumn helper no longer needs a non-const rvalue. Take the column by const reference and forward it without std::move; this also lets ColumnArrayT::Wrap pass col.data_ directly instead of copying it into a temporary ColumnRef.
Each typed column's Wrap now comes in two forms per input type: - Wrap(col, ValidationError* error): returns nullptr and, if error is non-null, fills *error on a type mismatch; it never throws. - Wrap(col): calls the two-argument form and throws the resulting ValidationError when it returns nullptr. The recursive WrapColumn helper mirrors the same pair. All Wrap mismatches now throw ValidationError consistently (previously Wrap(const Column&) and LowCardinality threw std::bad_cast). Add a default ValidationError constructor so an empty error can be created without a placeholder message, and add tests covering both the nullptr+error and throwing paths.
The Wrap-related helpers (SliceVector, HasWrapMethod, WrapColumn) depend only on declarations already provided by column.h, so fold them directly into column.h and drop the separate utils.h header. Since column.h is included everywhere, these utilities are now available without an extra include. Remove the now-redundant #include "utils.h" from the column sources and add a warning message when attempting to import "columns/utils.h"
Column::As<T>() and Column::AsStrict<T>() now recognise "wrappable" typed columns (those exposing a static Wrap method: ColumnArrayT, ColumnTupleT, ColumnMapT, ColumnNullableT, ColumnLowCardinalityT). For such a T that is not already an exact match, they Wrap the column into a storage-sharing typed view instead of failing: As returns nullptr when neither an exact cast nor a wrap is possible, and AsStrict throws ValidationError. Behaviour for non-wrappable T is unchanged, and an exact cast is always tried first to preserve object identity. The const As() overload deliberately does NOT wrap: wrapping would synthesize a mutable, storage-sharing view from a const column (via const_cast), which is not const-correct. It keeps the plain exact-downcast behaviour. The definitions are moved out-of-line below WrapColumn so the wrapping helpers are in scope. The ColumnLowCardinalityT::Wrap dictionary guard is pinned to a strict dynamic_pointer_cast because the constructor binds the dictionary via a reference dynamic_cast and therefore requires the exact stored type.
CreateColumnByType previously mapped LowCardinality(String)/LowCardinality( FixedString) to the strongly-typed ColumnLowCardinalityT<...>, unlike Array, Nullable, Tuple and Map, which all produce their base column type. Return the base ColumnLowCardinality here as well, so behaviour is consistent; callers can obtain the strongly-typed ColumnLowCardinalityT<...> view on demand via the wrapping Column::As<ColumnLowCardinalityT<...>>(). String and FixedString now share a single CreateColumnFromAst-based construction. The Nullable case keeps its own branch (documented) because it must select the ColumnLowCardinality(shared_ptr<ColumnNullable>) ctor overload that seeds the NULL item at dictionary index 0, and the default case keeps an explicit UnimplementedError for unsupported dictionary types. Add a CreateColumnByType.LowCardinality test and type-name round-trip cases.
The dictionary item type code was cached as a const member (type_) in the ColumnLowCardinalityT subclass. Store it in ColumnLowCardinality::item_type_code_ instead, computed once in Setup(), so the code is shared base state rather than per-subclass data. Remove the now-unused GetTypeCode helper.
The custom Swap of Array/Nullable/Tuple/Map rebound their member shared_ptr slots, so As<>/Wrap views (which hold their own handles to the same sub-columns) diverged after a swap. ColumnTuple::Clear had the same problem: columns_.clear() dropped the element handles, corrupting the tuple structure and leaving views with stale data (also the root cause of the Map clear bug). Swap the sub-columns' contents in place (via their own Swap) and clear each element in place, preserving sub-object identity so every view stays coherent, including through deep nesting. The typed wrapper overrides no longer swap their cached typed handles. Tuple Swap now guards against differing arity, and mismatched element types throw via the nested Swap instead of silently producing a Type()/data mismatch. LowCardinality Swap is intentionally left as-is; making it coherent needs the shared-state refactor and is handled separately.
ColumnLowCardinality::Swap and LoadBody rebound the index_column_ shared_ptr slot (and reassigned index_type_code_ by value), so As<>/Wrap views - which hold their own copies of those members - kept a stale index paired with the freshly swapped/loaded dictionary, returning wrong values or reading out of bounds. Unlike the dictionary and dedup map (mutated in place and thus already shared), the index column must be REPLACED because its numeric width can change, so it can't be swapped content-wise. Bundle the index column and its cached type code into an IndexState held behind a single shared_ptr; wrapped views share the bundle, so replacing the index (LoadBody) or swapping the bundle contents (Swap) is visible through every holder. The dictionary is still swapped in place to keep its object address stable for ColumnLowCardinalityT's typed_dictionary_. Adds wrap-then-Swap and wrap-then-Load alias-coherence tests.
The const overloads previously performed exact downcasts only, so col->As<ColumnArrayT<...>>() succeeded on a mutable handle but returned nullptr on a const handle for the very same column. Make them consistent: for a wrappable T, try an exact cast then fall back to Wrap, returning a read-only shared_ptr<const T> view. Wrapping never mutates the source; it uses an internal const_pointer_cast because Wrap builds a mutable wrapper. Add a matching const AsStrict() overload that throws on mismatch. Add tests covering const As() wrapping, const exact downcast, and const AsStrict() wrapping plus its throwing paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Column::As<T>()andColumn::AsStrict<T>()are now smarter for the typed columntemplates (
ColumnArrayT,ColumnTupleT,ColumnMapT,ColumnNullableT,ColumnLowCardinalityT). When the requestedTis one of these and the column is notalready exactly
T, they wrap the column into a strongly-typed, storage-sharing viewinstead of failing.
In practice this means you can take a column straight out of a
Selectresult and ask forthe convenient typed interface directly:
No manual reconstruction, no exact-type juggling.
Background: base columns vs. typed columns
ClickHouse composite types are represented by two layers in this library:
ColumnArrayColumnArrayT<Nested>At(),operator[],Append(container)ColumnNullableColumnNullableT<Nested>std::optional-basedAt()/operator[]ColumnTupleColumnTupleT<Cols...>std::tuple-basedAt()/operator[]ColumnMapColumnMapT<Key, Value>At(), iterationColumnLowCardinalityColumnLowCardinalityT<Dict>At(),operator[],AppendMany()When you
Select, the server-side columns are created by the factory as the basetypes (
ColumnArray,ColumnNullable,ColumnTuple,ColumnMap,ColumnLowCardinality).The
…T<>templates are the ergonomic, type-safe wrappers around them.Previously, to get a typed view you needed the column to already be that exact type, or
you had to build the typed column yourself. Now
As<>/AsStrict<>bridge the gapautomatically.
Behavior
A type is wrappable if it exposes a static
Wrapmethod. That is exactly the fivetemplates above (
ColumnArrayT,ColumnTupleT,ColumnMapT,ColumnNullableT,ColumnLowCardinalityT). Everything else (ColumnString,ColumnUInt64, …) isnon-wrappable and behaves as before.
TTAs<T>()(non-const)nullptron mismatch.nullptron mismatch.AsStrict<T>()ValidationErroron failure.As<T>() constnullptron mismatch.nullptron mismatch.Notes:
T, you get the very sameobject back (identity preserved, no new allocation).
Block::operator[]returns a non-constColumnRefby value, soblock[i]->As<…>()always uses the wrapping (non-const) overload — wrapping works out of the box for
Selectresults.
Wrapping is a storage-sharing view
Wrapping does not copy or move the underlying data. The typed wrapper shares the same
storage (dictionary, offsets, index, nested columns) with the source via
shared_ptr.Mutations through one are visible through the other, and the source stays fully valid.
Examples
LowCardinality
Array
Nullable
Tuple
Map
Nested types
Wrapping recurses, so nested typed columns work too:
AsvsAsStrictAsreturnsnullptron a type mismatch — always check it:AsStrictthrowsclickhouse::ValidationErrorinstead, which is handy when a mismatch is aprogramming error you want to surface immediately:
How it works (brief)
HasWrapMethod<T>, detects whetherTexposes a staticWrap.T,As/AsStricttrydynamic_pointer_cast<T>first, then fall back toWrapColumn<T>(...), which callsT::Wrap(...).T::Wrap(...)builds the typed wrapper by sharing the source column's internal storage(
shared_ptr), recursing into nested columns as needed.Wrap(col, ValidationError*)) used to implementAs, andthrowing one-argument forms used by
AsStrict.Limitations & gotchas
Check for
nullptr(or useAsStrict).Assignals a mismatch by returningnullptr; it does not throw for a plain type mismatch.Wrapping allocates. Unless the column already is
T(exact-match fast path), wrappingcreates a new wrapper object (with a distinct identity from the source, though it shares
storage). These methods are not
noexcept: allocation can still throwstd::bad_alloc, andthe throwing forms raise
ValidationErroron a mismatch.Numeric
LowCardinality(T)is unsupported by the column factory (its default-itemhandling is string-based).