[FLINK-40488][table] Support UUID as a comparison, ordering key by raminqaf · Pull Request #29128 · apache/flink

raminqaf · 2026-09-08T07:51:57Z

What is the purpose of the change

The main purpose is to make the UUID type usable as a key and as a value literal in Table API and SQL. This is a subtask of FLIP-604 (FLINK-40488).

UUID is stored internally as its 16-byte big-endian encoding, and the type defines ordering as the unsigned big-endian byte comparison of that encoding. This is the same ordering Flink's binary compare helpers already implement (SortUtil.compareBinary, SqlFunctionUtils.byteArrayCompare), and it matches the DataStream UuidComparator and how PostgreSQL orders UUIDs. The change routes UUID through those existing paths rather than adding new comparison logic. COUNT(DISTINCT uuid) is intentionally left out, consistent with BINARY/VARBINARY, and is deferred to a follow-up.

The PR also contains a related cleanup (FLINK-40616): it makes the sibling sealed built-in binary types, VARIANT and BITMAP, handle conversions consistently and removes redundant checks. It is kept as a separate commit.

Brief change log

UUID value literals (FLINK-40488):

ValueDataTypeConverter / ExpressionConverter / ValueLiteralExpression: UUID value literals and UUID '...' serialization.

UUID comparison and ordering (FLINK-40488):

TypeCheckUtils#isComparable: UUID is comparable.
GenerateUtils#generateCompare and ScalarOperatorGens#generateComparison: UUID uses the unsigned binary comparison.
SortCodeGenerator: UUID gets a fixed 16-byte normalized key.

BITMAP/VARIANT consistency (FLINK-40616):

BitmapType / VariantType: supports{Input,Output}Conversion use Class.isAssignableFrom, dropping the per-type whitelist sets.
ClassDataTypeConverter / ValueDataTypeConverter: both types resolve to their interface conversion class uniformly.
DataStructureConverters: both return a shared, stateless IdentityConverter.INSTANCE; BitmapBitmapConverter is removed.
DataFormatConverters.BitmapConverter: plain identity converter, matching VariantConverter.
BitmapSerializer.serialize: enforces the canonical form via RoaringBitmapData.toRoaringBitmapData, matching VariantSerializer.

Verifying this change

UUID (FLINK-40488):

SortCodeGeneratorTest: adds UUID to the randomized sort harness with 0x00… / 0x80… / 0xff… boundary values that make unsigned and signed byte orders disagree.
UuidSemanticTest (streaming and batch): equality, column comparison, a > UUID '...' literal filter, ORDER BY (as a bounded Top-N), GROUP BY, and a join on a UUID key.
ExpressionTest, ValueDataTypeConverterTest: UUID value-literal extraction and value-based type derivation.
CastFunctionITCase: UUID casts driven through lit(uuid) and UUID input fields.

BITMAP/VARIANT (FLINK-40616):

ClassDataTypeConverterTest, ValueDataTypeConverterTest, TypeInferenceExtractorTest, DataStructureConvertersTest, ExternalSerializerTest, DataFormatConvertersTest, BitmapSerializerTest, VariantSerializerTest.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving):
- ValueLiteralExpression is @PublicEvolving; UUID handling is additive within existing methods, with no signature or contract change.
- BitmapType and VariantType are @PublicEvolving; supports{Input,Output}Conversion now accept any subtype of the respective interface via isAssignableFrom instead ognature change.
The serializers: yes. BitmapSerializer.serialize now routes through RoaringBitmapData.toRoaringBitmapData; the wire format is
unchanged (byte-identical for RoaringBitmapData), o is added, consistent with VariantSerializer. Nostate-compatibility impact. No UUID serializer change.
The runtime per-record code paths (performance sees the existing unsigned binary comparison and afixed-length normalized key. BITMAP conversion drops a redundant per-record guard, and both binary types share a single stateless
identity converter. No other type's path changes.
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? y and as a value literal).
If yes, how is the feature documented? Documentation for the UUID type is tracked separately under FLINK-40494. The FLINK-40616 change is an internal consistency refactor with no us

Was generative AI tooling used to co-author thi

Yes (please specify the tool below)

Generated-by: Claude Code (Opus 4.8)

flinkbot · 2026-09-08T08:10:14Z

CI report:

cac1123 Azure: FAILURE
a5ce72e UNKNOWN

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

twalthr

Thank you @raminqaf. I added some comments just to not forget about this:
Please also update

ValueLiteralExpression
AbstractSqlCallContext#getLiteralValueAs and all related CallContext classes

twalthr · 2026-09-08T09:22:59Z

                && !isVariantType(type)
-                && !isBitmapType(type)
-                && !isUuidType(type);
+                && !isBitmapType(type);


Why did you remove it here?

Removed because the method isComparable returns true if the type is comparable. Before this PR UUID was not yet comparable.

twalthr · 2026-09-08T15:46:22Z

        } else if (value instanceof RoaringBitmapData) {
            convertedDataType = DataTypes.BITMAP();
+        } else if (value instanceof UUID) {
+            convertedDataType = DataTypes.UUID();


Something is off here. All 3 cases (Variant, Bitmap, UUID) actually belong to ClassDataTypeConverter.

That is true... The definition here is redundant. If we don't define the branch it should fall back to the else and get the defaultDataType from CalssDataTypeConverter.

Allow a UUID to be used as a value literal in the expression layer. ValueDataTypeConverter now derives DataTypes.UUID() from a java.util.UUID value (so Expressions.lit(uuid) works), and ExpressionConverter builds the literal RexNode via RexBuilder.makeUuidLiteral, since RexBuilder.makeLiteral has no generic UUID support. This also lets filter push-down round-trip a UUID predicate back into a RexNode, and lets a UUID literal be serialized as UUID '...'.

Enable UUID as an ordering, grouping and join key. UUID is stored as its 16-byte big-endian encoding and orders by the same unsigned byte-wise comparison as binary strings, so it is routed through the existing binary compare paths (SortUtil.compareBinary for sorting and SqlFunctionUtils.byteArrayCompare for predicates) at the code-generation sites that previously threw: the comparability gate, comparison and sort code generation, and the normalized-key computer (a fixed 16-byte, fully determining key consistent with the DataStream UuidComparator).

twalthr

Thank you @raminqaf. I added some last minor comments. Should be good to merge in the next iteration.

…istent BITMAP and VARIANT are sealed built-in binary types with a single canonical internal representation, RoaringBitmapData and BinaryVariant. Their handling had diverged across the type system, and the canonical-form invariant was enforced in several overlapping places for BITMAP but only in the serializer for VARIANT. This change makes the two types follow the same paths and enforces the canonical form once per type, in the serializer: - BitmapType and VariantType use Class.isAssignableFrom in supportsInputConversion and supportsOutputConversion, dropping the static whitelist sets. - ClassDataTypeConverter and ValueDataTypeConverter resolve both types to their interface conversion class uniformly. - DataStructureConverters returns a shared stateless IdentityConverter.INSTANCE for both types, removing the bespoke BitmapBitmapConverter and the redundant switch guard. - DataFormatConverters.BitmapConverter becomes a plain identity converter, matching VariantConverter. - BitmapSerializer.serialize enforces the canonical form via RoaringBitmapData.toRoaringBitmapData, matching VariantSerializer. There is no user-facing change for the supported implementations. A non-canonical conversion class is now rejected at the serialization boundary instead of by a plan-time whitelist.

twalthr reviewed Sep 8, 2026

View reviewed changes

raminqaf force-pushed the FLINK-40488-uuid-comparison branch from 261b275 to 8aee200 Compare September 8, 2026 14:28

raminqaf changed the title ~~[FLINK-40488][table] Support comparison and ordering for the UUID type~~ [FLINK-40488][table] Support UUID as a comparison, ordering, and aggregation key Sep 8, 2026

raminqaf force-pushed the FLINK-40488-uuid-comparison branch from 8aee200 to 283078f Compare September 8, 2026 14:58

raminqaf changed the title ~~[FLINK-40488][table] Support UUID as a comparison, ordering, and aggregation key~~ [FLINK-40488][table] Support UUID as a comparison, ordering key Sep 8, 2026

twalthr reviewed Sep 8, 2026

View reviewed changes

raminqaf force-pushed the FLINK-40488-uuid-comparison branch 2 times, most recently from 0a6fec0 to 3249efc Compare September 9, 2026 12:55

raminqaf added 2 commits September 9, 2026 14:58

raminqaf force-pushed the FLINK-40488-uuid-comparison branch from 3249efc to cac1123 Compare September 9, 2026 13:00

twalthr reviewed Sep 9, 2026

View reviewed changes

Comment thread flink-core/src/main/java/org/apache/flink/api/common/typeutils/base/BitmapSerializer.java Outdated

Comment thread ...common/src/test/java/org/apache/flink/table/types/extraction/TypeInferenceExtractorTest.java Outdated

raminqaf force-pushed the FLINK-40488-uuid-comparison branch from cac1123 to a5ce72e Compare September 9, 2026 15:53

Navigation Menu

Uh oh!

[FLINK-40488][table] Support UUID as a comparison, ordering key - #29128

[FLINK-40488][table] Support UUID as a comparison, ordering key#29128
raminqaf wants to merge 3 commits into
apache:masterfrom
raminqaf:FLINK-40488-uuid-comparison

raminqaf commented Sep 8, 2026 •

edited

Loading

Uh oh!

flinkbot commented Sep 8, 2026 •

edited

Loading

Uh oh!

twalthr left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twalthr Sep 8, 2026

Uh oh!

raminqaf Sep 8, 2026

Uh oh!

twalthr Sep 8, 2026

Uh oh!

raminqaf Sep 9, 2026

Uh oh!

twalthr left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

raminqaf commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author thi

Uh oh!

flinkbot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twalthr Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

twalthr Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

raminqaf commented Sep 8, 2026 •

edited

Loading

flinkbot commented Sep 8, 2026 •

edited

Loading