Skip to content

NIFI-14210 Add record writer for Protobuf messages - #11499

Open
prankstrisse wants to merge 1 commit into
apache:mainfrom
prankstrisse:NIFI-14210
Open

NIFI-14210 Add record writer for Protobuf messages#11499
prankstrisse wants to merge 1 commit into
apache:mainfrom
prankstrisse:NIFI-14210

Conversation

@prankstrisse

@prankstrisse prankstrisse commented Aug 3, 2026

Copy link
Copy Markdown

Summary

NIFI-14210

Changes

  • StandardProtobufWriter (nifi-protobuf-services) — new SchemaRegistryService + RecordSetWriterFactory. Resolves the target Proto message either statically via a Message Name property or dynamically via a MessageNameResolver service, compiles the schema with the existing ProtobufSchemaCompiler, and serializes each Record with a new ProtobufDataSerializer. A single message is written per FlowFile, since concatenated Protobuf messages cannot be delimited on read.
  • ProtobufDataSerializer — maps NiFi Record fields onto Wire Schema/MessageType descriptors and encodes the binary payload. google.protobuf.Any-typed fields are written as plain nested messages rather than re-wrapped as Any.
  • WriteProtobufResultWithExternalSchema — orchestrates header + message-index + payload sequencing, mirroring the existing WriteAvroResultWithExternalSchema pattern: invokes SchemaReferenceWriter.writeHeader(...) when a Schema Reference Writer is configured, then an optional MessageIndexWriter.writeMessageIndex(...) step, then the Protobuf payload.
  • MessageIndexWriter (nifi-schema-registry-service-api) — new shared interface, the write-side inverse of MessageNameResolver: given a resolved message name and schema definition, encodes the path to that message and writes it to the output stream so a reader can resolve the same name back from the encoded path.
  • ConfluentProtobufMessageIndexWriter + ProtobufMessageIndexEncoder (nifi-confluent-protobuf-message-index-writer, new module) — Confluent implementation of MessageIndexWriter. Walks nested message declarations to find the target message's declaration-order path and zigzag-varint encodes it (including the [0] single-message optimization), mirroring the decoding done by ConfluentProtobufMessageNameResolver. Together with the unchanged, already-generic ConfluentEncodedSchemaReferenceWriter, this reproduces the Confluent wire format (5-byte header + message-index array + payload) on write.
  • VarintUtils — relocated from the message-name-resolver module to org.apache.nifi.confluent.schema and made non-package-private so it can be shared between the resolver (read) and index writer (write) modules.
  • Minor visibility/refactoring changes to ProtobufSchemaCompiler and ProtobufSchemaValidator to support reuse from the writer.

Testing

  • ProtobufMessageIndexEncoderTest, ConfluentProtobufMessageIndexWriterTest — unit tests for index-path encoding, including nested messages and the single-index optimization.
  • TestProtobufDataSerializer — Record-to-Protobuf field mapping and encoding.
  • TestStandardProtobufWriter — controller service property/validation behavior.
  • TestStandardProtobufWriterConfluentRoundTrip — end-to-end round trip: StandardProtobufWriter output (header + index + payload) is read back by StandardProtobufReader/ConfluentProtobufMessageNameResolver and compared against the original Records.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
  • Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
  • Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

  • Pull Request based on current revision of the main branch
  • Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • Build completed using ./mvnw clean install -P contrib-check
    • JDK 21
    • JDK 25

Licensing

  • New dependencies are compatible with the Apache License 2.0 according to the License Policy
  • New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • Documentation formatting appears as expected in rendered files

@prankstrisse
prankstrisse force-pushed the NIFI-14210 branch 3 times, most recently from 2bf36d3 to 936d100 Compare August 3, 2026 12:05
@prankstrisse
prankstrisse marked this pull request as ready for review August 3, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant