[26.2] Kafka 4.x compatibility for non-Java clients#1781
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@nguyen-andrew @mattschumpert from https://redpandadata.atlassian.net/browse/DOC-2093 I'm not sure what the status is for Sarama, so I've left it out for now but please let me know if we can add it in. |
|
@nguyen-andrew @tyson-redpanda we have customers using Sarama and I think partial test coverage. Should we polish that off and claim support. or not? |
|
@mattschumpert @kbatuigas I think we can list Sarama. Older versions of Sarama pinned a protocol version instead of auto-negotiating, but turns out newer versions (I believe 1.46+) negotiate and clamp down to the broker versions like the other clients. I just did a quick manual test of Sarama v1.48.0 against Redpanda with its client config version set across 2.1, 2.6, 3.0, 4.0, and 4.2, and produce, consumer-group consume, and transactions worked at each level. I'll update our existing Sarama tests to declare a current version instead of keeping them pinned to 2.1/2.6. However, these tests are still smoke-level - we can add more coverage, but it'll need to be carved out as its own task. |
| == Kafka 4.x client compatibility | ||
|
|
||
| xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] are now available, making it possible to use durable cloud storage (S3, ADLS, GCS) as the primary backing store instead of local disk, eliminating over 90% of cross-AZ replication costs. This makes them ideal for latency-tolerant, high-throughput workloads such as observability streams, analytics pipelines, and AI/ML training data feeds, where cross-AZ networking charges are the dominant cost driver. | ||
| Redpanda version 26.2 validates a broader set of non-Java Apache Kafka® clients at their current versions that support Kafka 4.x, in addition to the Apache Kafka Java client. Newly validated clients include `confluent-kafka-go` and Sarama for Go, and `kafka-python` and `confluent-kafka-python` for Python. Validation confirms connectivity and correctness (for example, produce, consume, and transaction operations) but does not imply broker feature parity with Kafka. |
There was a problem hiding this comment.
@kbatuigas @pgellert @nguyen-andrew This framing does not inspire confidence. Its technically true (we don't have perfect 'API parity' mostly because of a bunch of things that aren't meaningful to us, like ZooKeeper/Kraft/Java runtime specific APIs, but im not sure we are really lacking a bunch of kafka 'features' that these clients would reasonably use (except perhaps a few schema registry things). What's a better way to say this? Can we either remove this sentence, or include just the positive statement as the caveat, saying 'validation is focused on core kafka APIs such as produce, consume, transactions, etc.' ?
There was a problem hiding this comment.
Especially in this era of mass Confluent migrations, I don't want people/LLMs reading this and thinking our support of their applications with these clients is a giant 'maybe'.
| == Kafka client compatibility | ||
|
|
||
| Clients developed for Kafka versions 0.11 or later are compatible with Redpanda. The Apache Kafka Java client at version 4.x is validated against Redpanda using the ducktape and chaos test suites; other clients in the table here remain compatible with the Kafka protocol versions they have always supported. Modern clients auto-negotiate protocol versions or use an earlier protocol version accepted by Redpanda brokers. | ||
| Redpanda validates the Apache Kafka Java client and a set of widely used non-Java clients, at their current versions that support Kafka 4.x, using the ducktape and chaos test suites. Validation confirms connectivity and correctness (for example, produce, consume, and transaction operations) at current client versions. It doesn't imply broker feature parity with Kafka. Modern clients auto-negotiate protocol versions or use an earlier protocol version accepted by Redpanda brokers. |
There was a problem hiding this comment.
Same comment as below @kbatuigas
There was a problem hiding this comment.
- I realized we missed removing this quotas bullet in 26.1. This is now false (as of 26.1, we DO support per user throughput quotas since 26.1). So we can just remove this bullet completely.
- Looking at this again, we should rename this section 'Minor differences between Redpanda and Kafka' because it's no longer really a list of missing features, esp 2/3 of the remaining items
e0e0c7e to
239f8c1
Compare
3 participants
Description
This pull request updates the Kafka client compatibility documentation to clarify which clients are validated with Redpanda and to improve accuracy and detail in the client compatibility table. The changes make it clearer which specific client versions are tested, expand the list of validated clients, and correct some references.
Documentation improvements:
confluentinc/librdkafka, addedconfluent-kafka-gofor Go, and added bothkafka-pythonandconfluent-kafka-pythonfor Python.confluent-pythonand focus onconfluent-kafka-dotnetas an example of a client based onlibrdkafka.Resolves https://redpandadata.atlassian.net/browse/DOC-2093
Review deadline: 13 July
Page previews
https://deploy-preview-1781--redpanda-docs-preview.netlify.app/streaming/26.2/develop/kafka-clients/
https://deploy-preview-1781--redpanda-docs-preview.netlify.app/streaming/26.2/get-started/release-notes/redpanda/
Checks