Skip to content

[FLINK-40120][table] Persist RowData field names in RowDataSerializerSnapshot#28717

Open
weiqingy wants to merge 2 commits into
apache:masterfrom
weiqingy:FLINK-40120-impl
Open

[FLINK-40120][table] Persist RowData field names in RowDataSerializerSnapshot#28717
weiqingy wants to merge 2 commits into
apache:masterfrom
weiqingy:FLINK-40120-impl

Conversation

@weiqingy

@weiqingy weiqingy commented Jul 11, 2026

Copy link
Copy Markdown

What is the purpose of the change

This pull request persists the top-level field names of a RowData state serializer in RowDataSerializerSnapshot, so they survive a save/restore round-trip. Today the snapshot stores the field types but not the names.

It is a self-contained, shared building block: the names are needed by schema-aware consumers — the State Catalog (FLIP-599) to expose a readable schema for keyed state, and name-based schema evolution (FLIP-527). They are written unconditionally and decoupled from any evolution logic, so whichever effort lands first establishes a single snapshot format.

Brief change log

  • Add a nullable top-level String[] fieldNames to RowDataSerializer, threaded through duplicate() and the snapshot round-trip; equals/hashCode are unchanged (names derive from the types).
  • Bump RowDataSerializerSnapshot from V3 to V4, appending the names block after the existing types / nested-serializer section. readSnapshot reads the names only for readVersion >= 4, so V4 readers still restore V3 snapshots (names absent -> null).
  • InternalSerializers builds the ROW serializer with its top-level field names (split from the structured-type case, which stays name-less).

Verifying this change

This change added tests and can be verified as follows:

  • V4 snapshot round-trip preserves the top-level names, and nested ROW field names remain recoverable from the child logical types.
  • Genuine V3 -> V4 backward compatibility: a hand-crafted V3-layout snapshot restores with fieldNames == null and the serializer still round-trips.
  • InternalSerializers.create(<a RowType>) yields a serializer whose snapshot carries the names unconditionally.
  • duplicate() preserves the names.
  • resolveSchemaCompatibility is unchanged: a field-type-array difference still returns incompatible() regardless of names.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: yes — RowDataSerializer / RowDataSerializerSnapshot, snapshot format V3 -> V4
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes — the RowData state snapshot format moves V3 -> V4. Backward compatible: V4 readers restore V3 snapshots (names absent -> null); no runtime behavior change.
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no — it persists metadata not yet consumed by any decision; no user-facing behavior change.
  • If yes, how is the feature documented? not applicable
Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code, Claude Opus 4.8)

…Snapshot

Add the top-level field names to RowDataSerializerSnapshot so they survive a
save/restore round-trip, as shared infrastructure for schema-aware consumers.

The names are written unconditionally and the change is behavior-neutral: no
compatibility or evolution logic is added. The snapshot format moves V3 to V4;
V4 readers still read V3 snapshots (names absent -> null), so existing savepoints
continue to restore.

Generated-by: Claude Code (Claude Opus 4.8)
@weiqingy weiqingy marked this pull request as ready for review July 11, 2026 05:25
@flinkbot

flinkbot commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@spuru9 spuru9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jul 11, 2026

@raminqaf raminqaf left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Left a nit @weiqingy

Comment on lines +365 to +366
if (readVersion >= 4) {
if (in.readBoolean()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can be simplified:

Suggested change
if (readVersion >= 4) {
if (in.readBoolean()) {
if (readVersion >= 4 && in.readBoolean()) {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, combined into the single short-circuit condition. Thanks!

@github-actions github-actions Bot added the community-reviewed-LGTM Applied if there are 2 non-committer approves on a PR. (The submitter cannot approve their own PR.) label Jul 13, 2026

@gyfora gyfora left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@github-actions github-actions Bot added community-reviewed PR has been reviewed by the community. and removed community-reviewed PR has been reviewed by the community. community-reviewed-LGTM Applied if there are 2 non-committer approves on a PR. (The submitter cannot approve their own PR.) labels Jul 13, 2026
Combine the nested version and field-names-present checks into a single
short-circuit condition; behavior is unchanged.

Generated-by: Claude Code (Claude Opus 4.8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants