Add Kafka ingestion support for subset partitions

[xiangfu0]

Summary

Add support for Kafka partition-subset realtime ingestion so Pinot can assign and consume only selected topic partitions for a table.

Changes

• Add stream.kafka.partition.ids parser/validation utilities in pinot-kafka-base to interpret configured partition subsets.
• Update controller assignment logic (segment and instance selectors) to support partition-group assignment across subset partitions.
• Update Kafka metadata providers (pinot-kafka-3.0 and pinot-kafka-4.0) to:
  • honor configured partition subsets in partition counts/group metadata
  • validate configured IDs against topic metadata
  • support stable subset-based partition-group behavior
• Add unit tests for subset parsing and Kafka metadata-provider partition selection.
• Add quickstart examples for split-topic ingestion:
  • fineFoodReviews-part-0
  • fineFoodReviews-part-1

[Copilot]

Pull request overview

This PR adds support for configuring Kafka ingestion to consume only a subset of topic partitions via the stream.kafka.partition.ids configuration property. This enables multiple tables to share a single Kafka topic by consuming different partitions.

Changes:

• Added stream.kafka.partition.ids configuration property and parsing utilities
• Modified Kafka metadata providers (2.0 and 3.0) to validate and respect partition subsets
• Updated instance assignment logic to support non-contiguous partition IDs
• Added comprehensive unit tests and example configurations

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pinot-kafka-base/KafkaStreamConfigProperties.java	Defines new PARTITION_IDS constant for subset configuration
pinot-kafka-base/KafkaPartitionSubsetUtils.java	Implements parsing, validation, and deduplication of partition ID lists
pinot-kafka-base/KafkaPartitionSubsetUtilsTest.java	Comprehensive unit tests for partition ID parsing
pinot-kafka-base/KafkaPartitionLevelStreamConfig.java	Exposes stream config map for partition subset utilities
pinot-kafka-2.0/KafkaStreamMetadataProvider.java	Overrides partition methods to validate and return subset partitions
pinot-kafka-3.0/KafkaStreamMetadataProvider.java	Mirrors 2.0 implementation for Kafka 3.0 compatibility
pinot-kafka-2.0/KafkaPartitionLevelConsumerTest.java	Tests subset validation and partition count/ID fetching
pinot-kafka-2.0/README.md	Documents the partition subset feature
InstanceReplicaGroupPartitionSelector.java	Supports explicit partition IDs in instance assignment
ImplicitRealtimeTablePartitionSelector.java	Fetches and uses stream partition IDs for instance assignment
RealtimeSegmentAssignment.java	Updates segment assignment to handle non-contiguous partition IDs
InstanceAssignmentTest.java	Tests single-partition subset with non-zero ID
QuickStartBase.java	Adds fineFoodReviews-part-0 and fineFoodReviews-part-1 examples
examples/stream/subsetPartitions/*	Example configuration and documentation
examples/stream/fineFoodReviews-part-/	Demo tables consuming single partitions

[codecov-commenter]

Codecov Report

❌ Patch coverage is 65.75342% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.27%. Comparing base (f641a9c) to head (0d38946).
⚠️ Report is 9 commits behind head on master.

Files with missing lines	Patch %	Lines
...in/stream/kafka30/KafkaStreamMetadataProvider.java	41.86%	24 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #17587   +/-   ##
=========================================
  Coverage     63.26%   63.27%           
  Complexity     1466     1466           
=========================================
  Files          3190     3191    +1     
  Lines        192039   192102   +63     
  Branches      29421    29434   +13     
=========================================
+ Hits         121492   121545   +53     
- Misses        61026    61048   +22     
+ Partials       9521     9509   -12     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-11	63.24% <65.75%> (+0.02%)	⬆️
java-21	63.22% <65.75%> (-0.02%)	⬇️
temurin	63.27% <65.75%> (+<0.01%)	⬆️
unittests	63.26% <65.75%> (+<0.01%)	⬆️
unittests1	55.59% <ø> (+<0.01%)	⬆️
unittests2	34.27% <65.75%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.