Skip to content

feat: add support for dynamic KRaft quorum scaling #1010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
razvan wants to merge 13 commits into
base: main
Choose a base branch
from
Open

feat: add support for dynamic KRaft quorum scaling #1010

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
571bf23
feat: add quorum-manager sidecar for dynamic KRaft voter membership
razvan Aug 18, 2026
059e7d9
feat: add startup/liveness/readiness probes for KRaft controllers
razvan Aug 18, 2026
f98c117
feat: use OrderedReady pod management for controller StatefulSets
razvan Aug 18, 2026
645061b
fix: reject and gracefully handle KRaft controllers scaled to zero
razvan Aug 18, 2026
930a497
test: cover KRaft controller scale-up/down/shutdown in kuttl operatio…
razvan Aug 18, 2026
3074345
docs: document KRaft dynamic voter membership and update CHANGELOG
razvan Aug 18, 2026
fc21e2e
Cleanup inline comments and documentation
razvan Aug 18, 2026
d02b920
Remove the custom bash trap functions
razvan Aug 19, 2026
62adbff
Comments abd documentation cleanups
razvan Aug 19, 2026
95839be
Merge remote-tracking branch 'origin/main' into feat/kraft-dynamic-vo…
razvan Aug 20, 2026
c4cb3a5
Cleanup CHANGELOG and KRaft docs.
razvan Aug 20, 2026
4b42def
Remove the kcat-prober container and consolidate container probes
razvan Aug 20, 2026
0596a0a
Cleanups
razvan Aug 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,5 @@ tilt_options.json
.envrc

.DS_Store

.worktrees/
20 changes: 20 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,17 @@ All notable changes to this project will be documented in this file.

### Changed

- The dynamic KRaft quorum created by the operator is now scaled automatically. Previously,
manual intervention was needed after every scale operation.
This change introduces a new side-car container (`quorum-manager`) to all controller pods
that adds the new controller to the voter list.
On termination, a new `preStop` hook on the controller container (`kafka`) removes the pod from
the voter list before shutdown.
The property `controller.quorum.bootstrap.servers` now contains the headless service names
of all controller role groups instead of individual peer host names. This prevevents the
restart controller from restarting all pods in the quorum when a new one is added/deleted.
The controller `StatefulSet` is now scaled using `OrderedBy` instead of the `Parallel` strategy
to ensure only one voter is added/removed at a time and thus keep the quorum healthy ([#1010]).
- Internal operator refactoring: introduce a build() step in the reconciler that
assembles all relevant Kubernetes resources before anything is applied ([#985]).
- Bump stackable-operator to 0.116.0 ([#994], [#1011]).
Expand All @@ -32,11 +43,20 @@ All notable changes to this project will be documented in this file.
which could cause problems with GitOps tools (e.g. ArgoCD) reporting a diff in the custom resources.
See [our internal issue](https://github.com/stackabletech/hdfs-operator/issues/626) and [the fix](https://github.com/kube-rs/kube/pull/2042) for details ([#998]).

### Removed

- BREAKING: The broker pod's separate `kcat-prober` sidecar container has been removed; its
`kcat`-based readiness probe now runs directly on the `kafka` container instead (`kcat` has
shipped in the same product image as `kafka` since #527, so the dedicated container/image was
no longer needed). The `kcat-prober` value is no longer accepted in a broker's
`logging.containers` CRD field ([#1010]).

[#985]: https://github.com/stackabletech/kafka-operator/pull/985
[#990]: https://github.com/stackabletech/kafka-operator/pull/990
[#994]: https://github.com/stackabletech/kafka-operator/pull/994
[#998]: https://github.com/stackabletech/kafka-operator/pull/998
[#1000]: https://github.com/stackabletech/kafka-operator/pull/1000
[#1010]: https://github.com/stackabletech/kafka-operator/pull/1010
[#1011]: https://github.com/stackabletech/kafka-operator/pull/1011

## [26.7.0] - 2026-07-21
Expand Down
37 changes: 32 additions & 5 deletions docs/modules/kafka/pages/usage-guide/kraft-controller.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,13 +85,36 @@ KRaft mode requires major configuration changes compared to ZooKeeper:
* `cluster-id`: This is set to the `metadata.name` of the KafkaCluster resource during initial formatting
* `node.id`: This is a calculated integer, hashed from the `role` and `rolegroup` and added `replica` id.
* `process.roles`: Will always only be `broker` or `controller`. Mixed `broker,controller` servers are not supported.
* The operator configures a static voter list containing the controller pods. Controllers are not dynamically managed.
* Each controller pod runs an additional `quorum-manager` sidecar container that, on startup, admits the pod into
the KRaft voter set (`kafka-metadata-quorum.sh add-controller`).
Removing the pod from the voter set again (`remove-controller`) on termination runs as the `kafka` container's
*own* `preStop` hook.
* Controller pods have a `startupProbe` (a plain TCP check on the KRaft listener port) and a `livenessProbe` that
combines that same TCP check with a check that the controller's local Raft state hasn't been stuck `unattached`
for an extended period — a symptom of a dynamically-joining controller resolving
`controller.quorum.bootstrap.servers` to its own pod address and getting wedged fetching from itself; a restart
forces a fresh DNS resolution attempt. A `readinessProbe` separately checks that the Raft state is one of
`leader`, `follower`, or `voted` via the controller's metrics endpoint, so a controller that cannot join or
rejoin the quorum is correctly reported as not ready instead of appearing healthy.
* Exactly one controller (the one with the numerically lowest `node.id` among all controller pod descriptors) formats
with `kafka-storage.sh format --standalone`, bootstrapping a single-node quorum by itself.
Every other controller formats with `--no-initial-controllers` and joins purely through the sidecar's `add-controller` call.
Brokers always format with `--no-initial-controllers` too; they are never voters.
the current replica count.
* `controller.quorum.bootstrap.servers` (used by the `kafka` process itself to find the controller quorum, by the
`quorum-manager` sidecar for its own `add-controller` calls, and by the `kafka` container's own `preStop` hook
for its `remove-controller` call) points at each controller role group's own headless Service DNS name, not
individual pod addresses.

== Known Issues

* Automatic migration from Apache ZooKeeper to KRaft is not supported.
* Scaling controller replicas might lead to unstable clusters.
* Kerberos is currently not supported for KRaft in all versions.
* The single controller chosen to bootstrap the quorum standalone is picked by a stable, deterministic rule
(lowest `node.id`), which is safe only for a cluster's *original* bootstrap. If that specific controller's
persistent volume is ever lost and needs to reformat after the cluster has already formed a quorum elsewhere,
reformatting it with `--standalone` would bootstrap a second, conflicting one-node quorum instead of rejoining
the existing one. If this happens, recovery requires manual intervention with Kafka's own KRaft tooling.

== Troubleshooting

Expand All @@ -108,10 +131,14 @@ Likely caused by controller resource starvation or unstable Kubernetes schedulin
Ensure Kafka version 3.9.x and higher and follow the official migration documentation.
The Stackable Kafka operator currently does not support the migration.

=== Scaling issues
=== Scaling controllers

The https://developers.redhat.com/articles/2024/11/27/dynamic-kafka-controller-quorum[Dynamic scaling] is only supported from Kafka version 3.9.0.
If you are using older versions, automatic scaling may not work properly (e.g. adding or removing controller replicas).
Controller replicas can be scaled up and down on a running cluster. A per-pod `quorum-manager` sidecar admits the
pod into the KRaft voter set on startup, and the `kafka` container's own `preStop` hook removes it again on
termination, as described under "Internal operator details" above.

Scaling more than one controller at a time is processed one pod at a time (`OrderedReady` pod management), not
in parallel, so that each pod's removal from the voter set can complete before the next one is terminated.

== Kraft migration guide

Expand Down
5 changes: 1 addition & 4 deletions docs/modules/kafka/partials/supported-versions.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,7 @@
* 3.9.2 (LTS)
* 3.9.1 (deprecated)

Support for clusters running in Kraft mode (which includes Apache Kafka 4.x.x) is experimental because it has not been thoroughly tested in production environments yet.
Support for clusters running in Kraft mode (which includes Apache Kafka >= 4.x) is experimental due to the following known issues:

Also there are some known issues such as:

* Controller scaling is not reliable.
* Kerberos authentication is not tested yet.
* Service exposition is not definitive.
160 changes: 0 additions & 160 deletions extra/crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,86 +181,6 @@ spec:
description: Configuration per logger
type: object
type: object
kcat-prober:
anyOf:
- required:
- custom
- {}
- {}
description: Log configuration of the container
properties:
console:
description: Configuration for the console appender
nullable: true
properties:
level:
description: |-
The log level threshold.
Log events with a lower log level are discarded.
enum:
- TRACE
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
- NONE
- null
nullable: true
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
nullable: true
type: string
type: object
file:
description: Configuration for the file appender
nullable: true
properties:
level:
description: |-
The log level threshold.
Log events with a lower log level are discarded.
enum:
- TRACE
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
- NONE
- null
nullable: true
type: string
type: object
loggers:
additionalProperties:
description: Configuration of a logger
properties:
level:
description: |-
The log level threshold.
Log events with a lower log level are discarded.
enum:
- TRACE
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
- NONE
- null
nullable: true
type: string
type: object
default: {}
description: Configuration per logger
type: object
type: object
vector:
anyOf:
- required:
Expand Down Expand Up @@ -717,86 +637,6 @@ spec:
description: Configuration per logger
type: object
type: object
kcat-prober:
anyOf:
- required:
- custom
- {}
- {}
description: Log configuration of the container
properties:
console:
description: Configuration for the console appender
nullable: true
properties:
level:
description: |-
The log level threshold.
Log events with a lower log level are discarded.
enum:
- TRACE
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
- NONE
- null
nullable: true
type: string
type: object
custom:
description: Log configuration provided in a ConfigMap
properties:
configMap:
description: ConfigMap containing the log configuration files
nullable: true
type: string
type: object
file:
description: Configuration for the file appender
nullable: true
properties:
level:
description: |-
The log level threshold.
Log events with a lower log level are discarded.
enum:
- TRACE
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
- NONE
- null
nullable: true
type: string
type: object
loggers:
additionalProperties:
description: Configuration of a logger
properties:
level:
description: |-
The log level threshold.
Log events with a lower log level are discarded.
enum:
- TRACE
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
- NONE
- null
nullable: true
type: string
type: object
default: {}
description: Configuration per logger
type: object
type: object
vector:
anyOf:
- required:
Expand Down
9 changes: 8 additions & 1 deletion rust/operator-binary/src/controller.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -607,6 +607,14 @@ pub(crate) mod test_support {

/// Runs the real validate step against a minimal (auth/OPA-free) fixture.
pub fn validated_cluster(kafka: &v1alpha1::KafkaCluster) -> ValidatedCluster {
validate_err(kafka).expect("validate should succeed for the test fixture")
}

/// Runs the real validate step against a minimal (auth/OPA-free) fixture, without unwrapping
/// the result -- for tests asserting on a specific validation failure.
pub fn validate_err(
kafka: &v1alpha1::KafkaCluster,
) -> Result<ValidatedCluster, super::validate::Error> {
validate(
kafka,
DereferencedObjects {
Expand All @@ -617,7 +625,6 @@ pub(crate) mod test_support {
},
&operator_environment(),
)
.expect("validate should succeed for the test fixture")
}
}

Expand Down
Loading
Loading