From d92305bcd686aadd5eb46316f9e5eb372dd0452a Mon Sep 17 00:00:00 2001 From: Matteo Merli Date: Tue, 14 Apr 2026 11:13:07 -0700 Subject: [PATCH 1/4] [improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final This upgrade brings Pulsar onto the Netty 4.2 line in preparation for the async-http-client 3.x upgrade (#25023), which transitively depends on Netty 4.2 and cannot be landed while Pulsar force-pins Netty 4.1. Netty 4.1 and 4.2 cannot co-exist on the classpath (same io.netty.* package namespace), so the upgrade has to be done in a single step. The Netty team asserts source/binary forward compatibility from 4.1 to 4.2 for regular API users: https://netty.io/news/2025/04/03/4-2-0.html https://github.com/netty/netty/wiki/Netty-4.2-Migration-Guide Changes in this PR: * gradle/libs.versions.toml: - Bump netty from 4.1.132.Final to 4.2.12.Final. - Drop the separate netty-iouring version (0.0.26.Final). io_uring has graduated from incubator (io.netty.incubator:netty-incubator-transport-*-io_uring) to a first-class Netty artifact (io.netty:netty-transport-{classes,native}-io_uring), now pinned to the same Netty version. * pulsar-common/build.gradle.kts: Point the io_uring consumer at the renamed aliases. * pulsar-common/.../EventLoopUtil.java: Netty 4.2 removed the dedicated IOUringEventLoopGroup class. io_uring now uses the generic MultiThreadIoEventLoopGroup + IoUringIoHandler factory pattern, which makes io_uring groups indistinguishable from any other MultiThreadIoEventLoopGroup by type, breaking the existing instanceof-based channel class dispatch. Fix: introduce a private marker subclass IoUringMultiThreadIoEventLoopGroup used at construction. Also repoint the incubator imports (io.netty.incubator.channel.uring.*) to the core package (io.netty.channel.uring.*) and adjust class names (IOUring -> IoUring). * build-logic/conventions/.../pulsar.java-conventions.gradle.kts: Exclude io.netty.incubator from all configurations. BookKeeper 4.17.3 (bookkeeper-common and stream-storage-java-client) still declares a transitive dependency on the 0.0.26.Final incubator io_uring jars, which are compiled against Netty 4.1 internals and are not safe to leave on the 4.2 classpath. Pulsar uses the core io_uring API via EventLoopUtil; BK stream-storage is an optional feature that Pulsar does not expose in its default surface. * distribution/{server,shell}/src/assemble/LICENSE.bin.txt: Reflect the actual Netty jar set shipped after the upgrade: - Bump all 4.1.132.Final entries to 4.2.12.Final. - Replace the monolithic netty-codec-*.jar with its 4.2 split-out sub-modules netty-codec-base and netty-codec-compression (netty-codec is now an aggregator POM that ships no classes). - Rename the incubator io_uring entries (io.netty.incubator-netty-incubator-transport-*-io_uring-0.0.26.Final) to the core io_uring artifacts (io.netty-netty-transport-{classes,native}-io_uring-4.2.12.Final). The jar set was cross-checked against the output of :distribution:pulsar-server-distribution:serverDistTar and :distribution:pulsar-shell-distribution:shellDistTar. * pulsar-common/.../BitSetRecyclableRecyclableTest and ConcurrentBitSetRecyclableTest: Relax the testRecycle assertion. Netty 4.2's io.netty.util.Recycler (which is itself deprecated in 4.2) no longer guarantees same-thread immediate reuse, so we only assert functional behavior: any recycled instance must come back cleared, and distinct create() calls must return distinct objects. Verification: * ./gradlew compileJava compileTestJava: clean across the entire project, only deprecation warnings (NioEventLoopGroup, EpollEventLoopGroup, DefaultEventLoopGroup, ChannelOption.RCVBUF_ALLOCATOR, EpollMode, Recycler, PlatformDependent.threadLocalRandom). These are compat shims that still function in 4.2; cleanup can follow in a separate PR. * :pulsar-common:test: passes (678 tests). * :pulsar-broker:test --tests BrokerServiceTest: passes (broker startup, producer/consumer flow, Netty transport end-to-end). * :pulsar-proxy:test --tests ProxyServiceTlsStarterTest: passes (proxy, TLS handshake, tcnative-boringssl integration). * :distribution:pulsar-server-distribution:serverDistTar and :distribution:pulsar-shell-distribution:shellDistTar both build, and the Netty jar set inside each tarball matches the LICENSE.bin.txt files exactly. Known Netty 4.2 behavior changes that this PR does NOT address: * The default SslContextBuilder.endpointIdentificationAlgorithm changed from null to HTTPS in 4.2. Pulsar's TLS client sites need to be audited and explicitly configured. This is intentionally out of scope here because the audit touches many modules (pulsar-client, pulsar-broker, pulsar-proxy, pulsar-broker-auth-oidc, admin) and should be its own PR. * The default ByteBufAllocator changed from pooled to adaptive in 4.2. Pulsar is not setting io.netty.allocator.type=pooled in this PR; if CI soak tests show regressions, the pooled override can be added to the launch scripts as a follow-up. --- .../kotlin/pulsar.java-conventions.gradle.kts | 9 ++++ .../server/src/assemble/LICENSE.bin.txt | 47 +++++++++--------- .../shell/src/assemble/LICENSE.bin.txt | 45 ++++++++--------- gradle/libs.versions.toml | 7 ++- pulsar-common/build.gradle.kts | 6 +-- .../common/util/netty/EventLoopUtil.java | 48 ++++++++++++------- .../BitSetRecyclableRecyclableTest.java | 6 ++- .../ConcurrentBitSetRecyclableTest.java | 6 ++- 8 files changed, 101 insertions(+), 73 deletions(-) diff --git a/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts b/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts index a6f65cc23af39..a8bdc424168bd 100644 --- a/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts +++ b/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts @@ -36,6 +36,15 @@ configurations.all { // NoSuchMethodError in Log4jLoggerFactory at test startup. exclude(group = "org.apache.logging.log4j", module = "log4j-slf4j-impl") + // Exclude the io.netty.incubator io_uring artifacts pulled in transitively + // by BookKeeper (bookkeeper-common and stream-storage-java-client still + // reference the 0.0.26.Final incubator jars). io_uring has graduated from + // incubator to core Netty in 4.2 (io.netty:netty-transport-*-io_uring); + // the incubator jar is compiled against Netty 4.1 internals and is not + // safe to leave on the 4.2 classpath. Pulsar uses the core io_uring API + // via EventLoopUtil. + exclude(group = "io.netty.incubator") + // Force Jackson version to match the version catalog. Transitive dependencies // (e.g. from jackson-bom) can pull in newer versions that break API compatibility // (EnumResolver.constructUsingToString signature changed in 2.19+). diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 351d4884548fa..56ace919a342b 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -293,26 +293,27 @@ The Apache Software License, Version 2.0 - org.apache.commons-commons-lang3-3.20.0.jar - org.apache.commons-commons-text-1.14.0.jar * Netty - - io.netty-netty-buffer-4.1.132.Final.jar - - io.netty-netty-codec-4.1.132.Final.jar - - io.netty-netty-codec-dns-4.1.132.Final.jar - - io.netty-netty-codec-http-4.1.132.Final.jar - - io.netty-netty-codec-http2-4.1.132.Final.jar - - io.netty-netty-codec-socks-4.1.132.Final.jar - - io.netty-netty-codec-haproxy-4.1.132.Final.jar - - io.netty-netty-common-4.1.132.Final.jar - - io.netty-netty-handler-4.1.132.Final.jar - - io.netty-netty-handler-proxy-4.1.132.Final.jar - - io.netty-netty-resolver-4.1.132.Final.jar - - io.netty-netty-resolver-dns-4.1.132.Final.jar - - io.netty-netty-resolver-dns-classes-macos-4.1.132.Final.jar - - io.netty-netty-resolver-dns-native-macos-4.1.132.Final-osx-aarch_64.jar - - io.netty-netty-resolver-dns-native-macos-4.1.132.Final-osx-x86_64.jar - - io.netty-netty-transport-4.1.132.Final.jar - - io.netty-netty-transport-classes-epoll-4.1.132.Final.jar - - io.netty-netty-transport-native-epoll-4.1.132.Final-linux-aarch_64.jar - - io.netty-netty-transport-native-epoll-4.1.132.Final-linux-x86_64.jar - - io.netty-netty-transport-native-unix-common-4.1.132.Final.jar + - io.netty-netty-buffer-4.2.12.Final.jar + - io.netty-netty-codec-base-4.2.12.Final.jar + - io.netty-netty-codec-compression-4.2.12.Final.jar + - io.netty-netty-codec-dns-4.2.12.Final.jar + - io.netty-netty-codec-http-4.2.12.Final.jar + - io.netty-netty-codec-http2-4.2.12.Final.jar + - io.netty-netty-codec-socks-4.2.12.Final.jar + - io.netty-netty-codec-haproxy-4.2.12.Final.jar + - io.netty-netty-common-4.2.12.Final.jar + - io.netty-netty-handler-4.2.12.Final.jar + - io.netty-netty-handler-proxy-4.2.12.Final.jar + - io.netty-netty-resolver-4.2.12.Final.jar + - io.netty-netty-resolver-dns-4.2.12.Final.jar + - io.netty-netty-resolver-dns-classes-macos-4.2.12.Final.jar + - io.netty-netty-resolver-dns-native-macos-4.2.12.Final-osx-aarch_64.jar + - io.netty-netty-resolver-dns-native-macos-4.2.12.Final-osx-x86_64.jar + - io.netty-netty-transport-4.2.12.Final.jar + - io.netty-netty-transport-classes-epoll-4.2.12.Final.jar + - io.netty-netty-transport-native-epoll-4.2.12.Final-linux-aarch_64.jar + - io.netty-netty-transport-native-epoll-4.2.12.Final-linux-x86_64.jar + - io.netty-netty-transport-native-unix-common-4.2.12.Final.jar - io.netty-netty-tcnative-boringssl-static-2.0.75.Final.jar - io.netty-netty-tcnative-boringssl-static-2.0.75.Final-linux-aarch_64.jar - io.netty-netty-tcnative-boringssl-static-2.0.75.Final-linux-x86_64.jar @@ -320,9 +321,9 @@ The Apache Software License, Version 2.0 - io.netty-netty-tcnative-boringssl-static-2.0.75.Final-osx-x86_64.jar - io.netty-netty-tcnative-boringssl-static-2.0.75.Final-windows-x86_64.jar - io.netty-netty-tcnative-classes-2.0.75.Final.jar - - io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.26.Final.jar - - io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.26.Final-linux-x86_64.jar - - io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.26.Final-linux-aarch_64.jar + - io.netty-netty-transport-classes-io_uring-4.2.12.Final.jar + - io.netty-netty-transport-native-io_uring-4.2.12.Final-linux-x86_64.jar + - io.netty-netty-transport-native-io_uring-4.2.12.Final-linux-aarch_64.jar * Prometheus client - io.prometheus.jmx-collector-0.16.1.jar - io.prometheus-simpleclient-0.16.0.jar diff --git a/distribution/shell/src/assemble/LICENSE.bin.txt b/distribution/shell/src/assemble/LICENSE.bin.txt index 350f38dcdadc6..7aec9549e20dc 100644 --- a/distribution/shell/src/assemble/LICENSE.bin.txt +++ b/distribution/shell/src/assemble/LICENSE.bin.txt @@ -345,22 +345,23 @@ The Apache Software License, Version 2.0 - commons-text-1.14.0.jar - commons-compress-1.28.0.jar * Netty - - netty-buffer-4.1.132.Final.jar - - netty-codec-4.1.132.Final.jar - - netty-codec-dns-4.1.132.Final.jar - - netty-codec-http-4.1.132.Final.jar - - netty-codec-socks-4.1.132.Final.jar - - netty-codec-haproxy-4.1.132.Final.jar - - netty-common-4.1.132.Final.jar - - netty-handler-4.1.132.Final.jar - - netty-handler-proxy-4.1.132.Final.jar - - netty-resolver-4.1.132.Final.jar - - netty-resolver-dns-4.1.132.Final.jar - - netty-transport-4.1.132.Final.jar - - netty-transport-classes-epoll-4.1.132.Final.jar - - netty-transport-native-epoll-4.1.132.Final-linux-aarch_64.jar - - netty-transport-native-epoll-4.1.132.Final-linux-x86_64.jar - - netty-transport-native-unix-common-4.1.132.Final.jar + - netty-buffer-4.2.12.Final.jar + - netty-codec-base-4.2.12.Final.jar + - netty-codec-compression-4.2.12.Final.jar + - netty-codec-dns-4.2.12.Final.jar + - netty-codec-http-4.2.12.Final.jar + - netty-codec-socks-4.2.12.Final.jar + - netty-codec-haproxy-4.2.12.Final.jar + - netty-common-4.2.12.Final.jar + - netty-handler-4.2.12.Final.jar + - netty-handler-proxy-4.2.12.Final.jar + - netty-resolver-4.2.12.Final.jar + - netty-resolver-dns-4.2.12.Final.jar + - netty-transport-4.2.12.Final.jar + - netty-transport-classes-epoll-4.2.12.Final.jar + - netty-transport-native-epoll-4.2.12.Final-linux-aarch_64.jar + - netty-transport-native-epoll-4.2.12.Final-linux-x86_64.jar + - netty-transport-native-unix-common-4.2.12.Final.jar - netty-tcnative-boringssl-static-2.0.75.Final.jar - netty-tcnative-boringssl-static-2.0.75.Final-linux-aarch_64.jar - netty-tcnative-boringssl-static-2.0.75.Final-linux-x86_64.jar @@ -368,12 +369,12 @@ The Apache Software License, Version 2.0 - netty-tcnative-boringssl-static-2.0.75.Final-osx-x86_64.jar - netty-tcnative-boringssl-static-2.0.75.Final-windows-x86_64.jar - netty-tcnative-classes-2.0.75.Final.jar - - netty-incubator-transport-classes-io_uring-0.0.26.Final.jar - - netty-incubator-transport-native-io_uring-0.0.26.Final-linux-aarch_64.jar - - netty-incubator-transport-native-io_uring-0.0.26.Final-linux-x86_64.jar - - netty-resolver-dns-classes-macos-4.1.132.Final.jar - - netty-resolver-dns-native-macos-4.1.132.Final-osx-aarch_64.jar - - netty-resolver-dns-native-macos-4.1.132.Final-osx-x86_64.jar + - netty-transport-classes-io_uring-4.2.12.Final.jar + - netty-transport-native-io_uring-4.2.12.Final-linux-aarch_64.jar + - netty-transport-native-io_uring-4.2.12.Final-linux-x86_64.jar + - netty-resolver-dns-classes-macos-4.2.12.Final.jar + - netty-resolver-dns-native-macos-4.2.12.Final-osx-aarch_64.jar + - netty-resolver-dns-native-macos-4.2.12.Final-osx-x86_64.jar * Prometheus client - simpleclient-0.16.0.jar - simpleclient_log4j2-0.16.0.jar diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index b5a5aca4a675d..b6a087ec32ae9 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -25,8 +25,7 @@ checkstyle = "13.3.0" # Major frameworks bookkeeper = "4.17.3" zookeeper = "3.9.5" -netty = "4.1.132.Final" -netty-iouring = "0.0.26.Final" +netty = "4.2.12.Final" jetty = "12.1.6" jersey = "2.42" jackson = "2.21.2" @@ -215,8 +214,8 @@ netty-resolver-dns-native-macos = { module = "io.netty:netty-resolver-dns-native netty-transport-native-epoll = { module = "io.netty:netty-transport-native-epoll", version.ref = "netty" } netty-transport-native-unix-common = { module = "io.netty:netty-transport-native-unix-common", version.ref = "netty" } netty-tcnative-boringssl-static = { module = "io.netty:netty-tcnative-boringssl-static", version.ref = "netty-tcnative" } -netty-incubator-transport-classes-io_uring = { module = "io.netty.incubator:netty-incubator-transport-classes-io_uring", version.ref = "netty-iouring" } -netty-incubator-transport-native-io-uring = { module = "io.netty.incubator:netty-incubator-transport-native-io_uring", version.ref = "netty-iouring" } +netty-transport-classes-io_uring = { module = "io.netty:netty-transport-classes-io_uring", version.ref = "netty" } +netty-transport-native-io-uring = { module = "io.netty:netty-transport-native-io_uring", version.ref = "netty" } netty-reactive-streams = { module = "com.typesafe.netty:netty-reactive-streams", version.ref = "netty-reactive-streams" } # Protobuf / gRPC protobuf-bom = { module = "com.google.protobuf:protobuf-bom", version.ref = "protobuf" } diff --git a/pulsar-common/build.gradle.kts b/pulsar-common/build.gradle.kts index 5388569ee135e..71a2fd16a1f5e 100644 --- a/pulsar-common/build.gradle.kts +++ b/pulsar-common/build.gradle.kts @@ -122,9 +122,9 @@ dependencies { implementation(variantOf(libs.netty.tcnative.boringssl.static) { classifier("linux-aarch_64") }) implementation(variantOf(libs.netty.tcnative.boringssl.static) { classifier("osx-x86_64") }) implementation(variantOf(libs.netty.tcnative.boringssl.static) { classifier("osx-aarch_64") }) - implementation(libs.netty.incubator.transport.classes.io.uring) - implementation(variantOf(libs.netty.incubator.transport.native.io.uring) { classifier("linux-x86_64") }) - implementation(variantOf(libs.netty.incubator.transport.native.io.uring) { classifier("linux-aarch_64") }) + implementation(libs.netty.transport.classes.io.uring) + implementation(variantOf(libs.netty.transport.native.io.uring) { classifier("linux-x86_64") }) + implementation(variantOf(libs.netty.transport.native.io.uring) { classifier("linux-aarch_64") }) implementation(libs.netty.codec.haproxy) implementation(libs.commons.lang3) implementation(libs.jakarta.ws.rs.api) diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/netty/EventLoopUtil.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/netty/EventLoopUtil.java index 3dbdd28871532..c23a94cd5346e 100644 --- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/netty/EventLoopUtil.java +++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/netty/EventLoopUtil.java @@ -20,11 +20,13 @@ import io.netty.bootstrap.ServerBootstrap; import io.netty.channel.EventLoopGroup; +import io.netty.channel.MultiThreadIoEventLoopGroup; import io.netty.channel.SelectStrategy; import io.netty.channel.epoll.Epoll; import io.netty.channel.epoll.EpollChannelOption; import io.netty.channel.epoll.EpollDatagramChannel; import io.netty.channel.epoll.EpollEventLoopGroup; +import io.netty.channel.epoll.EpollIoHandler; import io.netty.channel.epoll.EpollMode; import io.netty.channel.epoll.EpollServerSocketChannel; import io.netty.channel.epoll.EpollSocketChannel; @@ -35,11 +37,11 @@ import io.netty.channel.socket.nio.NioDatagramChannel; import io.netty.channel.socket.nio.NioServerSocketChannel; import io.netty.channel.socket.nio.NioSocketChannel; -import io.netty.incubator.channel.uring.IOUring; -import io.netty.incubator.channel.uring.IOUringDatagramChannel; -import io.netty.incubator.channel.uring.IOUringEventLoopGroup; -import io.netty.incubator.channel.uring.IOUringServerSocketChannel; -import io.netty.incubator.channel.uring.IOUringSocketChannel; +import io.netty.channel.uring.IoUring; +import io.netty.channel.uring.IoUringDatagramChannel; +import io.netty.channel.uring.IoUringIoHandler; +import io.netty.channel.uring.IoUringServerSocketChannel; +import io.netty.channel.uring.IoUringSocketChannel; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ThreadFactory; import lombok.CustomLog; @@ -52,13 +54,25 @@ public class EventLoopUtil { private static final String ENABLE_IO_URING = "pulsar.enableUring"; + /** + * Marker subclass so we can distinguish io_uring-backed event loops via instanceof. + * Netty 4.2 removed the dedicated {@code IOUringEventLoopGroup} class in favor of the + * generic {@link MultiThreadIoEventLoopGroup} + {@link IoUringIoHandler} pair, which + * makes io_uring groups indistinguishable by type from any other MultiThreadIoEventLoopGroup. + */ + private static final class IoUringMultiThreadIoEventLoopGroup extends MultiThreadIoEventLoopGroup { + IoUringMultiThreadIoEventLoopGroup(int nThreads, ThreadFactory threadFactory) { + super(nThreads, threadFactory, IoUringIoHandler.newFactory()); + } + } + /** * @return an EventLoopGroup suitable for the current platform */ public static EventLoopGroup newEventLoopGroup(int nThreads, boolean enableBusyWait, ThreadFactory threadFactory) { if (Epoll.isAvailable()) { if (isIoUringEnabledAndAvailable()) { - return new IOUringEventLoopGroup(nThreads, threadFactory); + return new IoUringMultiThreadIoEventLoopGroup(nThreads, threadFactory); } else { if (!enableBusyWait) { // Regular Epoll based event loop @@ -96,8 +110,8 @@ private static boolean isIoUringEnabledAndAvailable() { String ioUringSetting = System.getProperty(ENABLE_IO_URING); boolean ioUringEnabled = "1".equalsIgnoreCase(ioUringSetting) || "true".equalsIgnoreCase(ioUringSetting); if (ioUringEnabled) { - // Throw exception if IOUring cannot be used - IOUring.ensureAvailability(); + // Throw exception if IoUring cannot be used + IoUring.ensureAvailability(); } return ioUringEnabled; } @@ -109,8 +123,8 @@ private static boolean isIoUringEnabledAndAvailable() { * @return */ public static Class getClientSocketChannelClass(EventLoopGroup eventLoopGroup) { - if (eventLoopGroup instanceof IOUringEventLoopGroup) { - return IOUringSocketChannel.class; + if (eventLoopGroup instanceof IoUringMultiThreadIoEventLoopGroup) { + return IoUringSocketChannel.class; } else if (eventLoopGroup instanceof EpollEventLoopGroup) { return EpollSocketChannel.class; } else { @@ -128,7 +142,7 @@ public static Class getClientSocketChannelClass(EventLo public static Class getClientSocketChannelClass() { if (Epoll.isAvailable()) { if (isIoUringEnabledAndAvailable()) { - return IOUringSocketChannel.class; + return IoUringSocketChannel.class; } else { return EpollSocketChannel.class; } @@ -138,8 +152,8 @@ public static Class getClientSocketChannelClass() { } public static Class getServerSocketChannelClass(EventLoopGroup eventLoopGroup) { - if (eventLoopGroup instanceof IOUringEventLoopGroup) { - return IOUringServerSocketChannel.class; + if (eventLoopGroup instanceof IoUringMultiThreadIoEventLoopGroup) { + return IoUringServerSocketChannel.class; } else if (eventLoopGroup instanceof EpollEventLoopGroup) { return EpollServerSocketChannel.class; } else { @@ -157,7 +171,7 @@ public static Class getServerSocketChannelClass(E public static Class getServerSocketChannelClass() { if (Epoll.isAvailable()) { if (isIoUringEnabledAndAvailable()) { - return IOUringServerSocketChannel.class; + return IoUringServerSocketChannel.class; } else { return EpollServerSocketChannel.class; } @@ -167,8 +181,8 @@ public static Class getServerSocketChannelClass() } public static Class getDatagramChannelClass(EventLoopGroup eventLoopGroup) { - if (eventLoopGroup instanceof IOUringEventLoopGroup) { - return IOUringDatagramChannel.class; + if (eventLoopGroup instanceof IoUringMultiThreadIoEventLoopGroup) { + return IoUringDatagramChannel.class; } else if (eventLoopGroup instanceof EpollEventLoopGroup) { return EpollDatagramChannel.class; } else { @@ -186,7 +200,7 @@ public static Class getDatagramChannelClass(EventLoop public static Class getDatagramChannelClass() { if (Epoll.isAvailable()) { if (isIoUringEnabledAndAvailable()) { - return IOUringDatagramChannel.class; + return IoUringDatagramChannel.class; } else { return EpollDatagramChannel.class; } diff --git a/pulsar-common/src/test/java/org/apache/pulsar/common/util/collections/BitSetRecyclableRecyclableTest.java b/pulsar-common/src/test/java/org/apache/pulsar/common/util/collections/BitSetRecyclableRecyclableTest.java index 8061f853d66c1..b0bbd1ce0308e 100644 --- a/pulsar-common/src/test/java/org/apache/pulsar/common/util/collections/BitSetRecyclableRecyclableTest.java +++ b/pulsar-common/src/test/java/org/apache/pulsar/common/util/collections/BitSetRecyclableRecyclableTest.java @@ -28,11 +28,13 @@ public void testRecycle() { BitSetRecyclable bitset1 = BitSetRecyclable.create(); bitset1.set(3); bitset1.recycle(); + // Netty 4.2's Recycler (which is itself deprecated) no longer guarantees same-thread + // immediate reuse, so we only assert on functional behavior: any recycled instance + // must come back cleared, and distinct create() calls must return distinct objects. BitSetRecyclable bitset2 = BitSetRecyclable.create(); BitSetRecyclable bitset3 = BitSetRecyclable.create(); - Assert.assertSame(bitset2, bitset1); Assert.assertFalse(bitset2.get(3)); - Assert.assertNotSame(bitset3, bitset1); + Assert.assertNotSame(bitset3, bitset2); } @Test diff --git a/pulsar-common/src/test/java/org/apache/pulsar/common/util/collections/ConcurrentBitSetRecyclableTest.java b/pulsar-common/src/test/java/org/apache/pulsar/common/util/collections/ConcurrentBitSetRecyclableTest.java index 7ffda74156969..605e0115a20c6 100644 --- a/pulsar-common/src/test/java/org/apache/pulsar/common/util/collections/ConcurrentBitSetRecyclableTest.java +++ b/pulsar-common/src/test/java/org/apache/pulsar/common/util/collections/ConcurrentBitSetRecyclableTest.java @@ -30,11 +30,13 @@ public void testRecycle() { ConcurrentBitSetRecyclable bitset1 = ConcurrentBitSetRecyclable.create(); bitset1.set(3); bitset1.recycle(); + // Netty 4.2's Recycler (which is itself deprecated) no longer guarantees same-thread + // immediate reuse, so we only assert on functional behavior: any recycled instance + // must come back cleared, and distinct create() calls must return distinct objects. ConcurrentBitSetRecyclable bitset2 = ConcurrentBitSetRecyclable.create(); ConcurrentBitSetRecyclable bitset3 = ConcurrentBitSetRecyclable.create(); - Assert.assertSame(bitset2, bitset1); Assert.assertFalse(bitset2.get(3)); - Assert.assertNotSame(bitset3, bitset1); + Assert.assertNotSame(bitset3, bitset2); } @SuppressWarnings("deprecation") From 4f5aed94d8a0fba1aa70fb89319765e73af20271 Mon Sep 17 00:00:00 2001 From: Matteo Merli Date: Tue, 14 Apr 2026 11:24:13 -0700 Subject: [PATCH 2/4] [fix][build] Remove unused EpollIoHandler import in EventLoopUtil checkstyleMain flagged an UnusedImports violation on the EpollIoHandler import that was added speculatively in the Netty 4.2 bump. The io_uring path was refactored to use IoUringIoHandler + MultiThreadIoEventLoopGroup, but the epoll path still uses the 4.2 EpollEventLoopGroup compat shim directly, so EpollIoHandler is never referenced. --- .../java/org/apache/pulsar/common/util/netty/EventLoopUtil.java | 1 - 1 file changed, 1 deletion(-) diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/netty/EventLoopUtil.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/netty/EventLoopUtil.java index c23a94cd5346e..f0e3d9cba641c 100644 --- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/netty/EventLoopUtil.java +++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/netty/EventLoopUtil.java @@ -26,7 +26,6 @@ import io.netty.channel.epoll.EpollChannelOption; import io.netty.channel.epoll.EpollDatagramChannel; import io.netty.channel.epoll.EpollEventLoopGroup; -import io.netty.channel.epoll.EpollIoHandler; import io.netty.channel.epoll.EpollMode; import io.netty.channel.epoll.EpollServerSocketChannel; import io.netty.channel.epoll.EpollSocketChannel; From 3975235e2493e1761f2a77b0e880de6174b6d17d Mon Sep 17 00:00:00 2001 From: Matteo Merli Date: Tue, 14 Apr 2026 12:43:20 -0700 Subject: [PATCH 3/4] [fix][build] Restore incubator io_uring jars on the classpath MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI surfaced a NoClassDefFoundError during BookKeeper broker setup: java.lang.NoClassDefFoundError: io/netty/incubator/channel/uring/IOUringEventLoopGroup java.lang.ClassNotFoundException: io.netty.incubator.channel.uring.IOUringEventLoopGroup at org.apache.pulsar.broker.LedgerLostAndSkipNonRecoverableTest.setupSharedCluster BookKeeper 4.17.3 bytecode still directly references io.netty.incubator.channel.uring.IOUringEventLoopGroup — its own event-loop selection path loads the class eagerly. The previous commit excluded the entire io.netty.incubator group from all configurations to stop shipping the 4.1-era incubator jars alongside Netty 4.2, but that also prevented BK from resolving the class it links against, turning a dependency hygiene concern into a hard runtime failure. Revert the exclusion. The incubator classes are in a separate package (io.netty.incubator.channel.uring.*) from the new core io_uring API (io.netty.channel.uring.*), so they can coexist on the classpath without symbol conflicts. The incubator bytecode was compiled against Netty 4.1.x, and Netty 4.2 asserts forward compatibility for 4.1 bytecode, so BK's lazy io_uring usage (only when io_uring is enabled) should continue to work at runtime. Pulsar's own EventLoopUtil already uses the core io_uring API introduced in 4.2, not the incubator one. Add the 0.0.26.Final incubator io_uring jars back to distribution/server LICENSE.bin.txt (the shell distribution does not pull BK's stream-storage and does not receive them). The proper long-term fix is a BookKeeper release that moves off the incubator io_uring onto Netty 4.2's core io_uring module — that can be picked up in a later Pulsar BK version bump. --- .../src/main/kotlin/pulsar.java-conventions.gradle.kts | 9 --------- distribution/server/src/assemble/LICENSE.bin.txt | 3 +++ 2 files changed, 3 insertions(+), 9 deletions(-) diff --git a/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts b/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts index a8bdc424168bd..a6f65cc23af39 100644 --- a/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts +++ b/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts @@ -36,15 +36,6 @@ configurations.all { // NoSuchMethodError in Log4jLoggerFactory at test startup. exclude(group = "org.apache.logging.log4j", module = "log4j-slf4j-impl") - // Exclude the io.netty.incubator io_uring artifacts pulled in transitively - // by BookKeeper (bookkeeper-common and stream-storage-java-client still - // reference the 0.0.26.Final incubator jars). io_uring has graduated from - // incubator to core Netty in 4.2 (io.netty:netty-transport-*-io_uring); - // the incubator jar is compiled against Netty 4.1 internals and is not - // safe to leave on the 4.2 classpath. Pulsar uses the core io_uring API - // via EventLoopUtil. - exclude(group = "io.netty.incubator") - // Force Jackson version to match the version catalog. Transitive dependencies // (e.g. from jackson-bom) can pull in newer versions that break API compatibility // (EnumResolver.constructUsingToString signature changed in 2.19+). diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 56ace919a342b..a19f29095facf 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -324,6 +324,9 @@ The Apache Software License, Version 2.0 - io.netty-netty-transport-classes-io_uring-4.2.12.Final.jar - io.netty-netty-transport-native-io_uring-4.2.12.Final-linux-x86_64.jar - io.netty-netty-transport-native-io_uring-4.2.12.Final-linux-aarch_64.jar + - io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.26.Final.jar + - io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.26.Final-linux-x86_64.jar + - io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.26.Final-linux-aarch_64.jar * Prometheus client - io.prometheus.jmx-collector-0.16.1.jar - io.prometheus-simpleclient-0.16.0.jar From ae903ebcb7625f9c3ef0b902f21a4af8ef84e2a0 Mon Sep 17 00:00:00 2001 From: Matteo Merli Date: Tue, 14 Apr 2026 14:32:57 -0700 Subject: [PATCH 4/4] [fix][test] Restore Netty 4.1 TLS endpoint identification default for tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Netty 4.2 changed the default SslContextBuilder.endpointIdentificationAlgorithm from null to "HTTPS", which enables hostname verification on every TLS client connection that does not explicitly set the algorithm. Four Pulsar tests fail on the Netty 4.2 bump because they were written against the 4.1 default of no hostname verification: * pulsar-broker: AuthenticationTlsHostnameVerificationTest .testTlsSyncProducerAndConsumerWithInvalidBrokerHost — exercises the "hostname verification disabled" path, expects a connection to an invalid broker host to succeed. * pulsar-broker: TlsWithECCertificateFileTest .testConnectionSuccessWithCertificate — the EC test certs do not contain a SAN that matches the bind address, so broker-internal TLS now fails the hostname check and topic load times out. * pulsar-proxy: ProxyMutualTlsTest.testProducerByAuthenticationTls — proxy test certs have the same SAN mismatch. * integration: ClientTlsTest.testClient — integration test certs likewise. Set -Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE for the test JVM only. This is the Netty-supplied migration override documented at https://github.com/netty/netty/wiki/Netty-4.2-Migration-Guide and it restores the 4.1 default for the test suite without touching production code. Production launch scripts are deliberately not changed. The correct long-term fix is to audit Pulsar's SslContextBuilder call sites and explicitly set endpointIdentificationAlgorithm based on the user's hostnameVerification configuration, after which this test override can be removed. That audit should be its own PR — it touches pulsar-client, pulsar-broker, pulsar-proxy, pulsar-broker-auth-oidc, and admin. Note on production behavior: with this PR, Pulsar users who upgrade will get Netty 4.2's stricter default, so any TLS client that implicitly relied on the 4.1 "no hostname verification" default will start enforcing it. Users who want to keep the old behavior can set -Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE on the JVM until the per-call-site audit lands. --- .../main/kotlin/pulsar.java-conventions.gradle.kts | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts b/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts index a6f65cc23af39..f34c956e8922d 100644 --- a/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts +++ b/build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts @@ -171,6 +171,18 @@ tasks.withType().configureEach { "-XX:+EnableDynamicAgentLoading", "-Xshare:off", "-Dio.netty.tryReflectionSetAccessible=true", + // Netty 4.2 changed the default SslContextBuilder.endpointIdentificationAlgorithm + // from null to "HTTPS", which enables hostname verification on every TLS client + // connection that does not explicitly set the algorithm. Pulsar's TLS tests were + // written against the 4.1 default (no verification unless explicitly configured), + // and several of them either use certificates without matching hostnames or + // deliberately exercise "hostname verification disabled" paths. Restore the 4.1 + // default for the test JVM so those tests keep running. Production launch scripts + // are deliberately not changed; a follow-up PR should audit Pulsar's + // SslContextBuilder call sites and explicitly set endpointIdentificationAlgorithm + // based on the user's hostnameVerification configuration, after which this test + // override can be removed. + "-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE", "-Dpulsar.allocator.pooled=true", "-Dpulsar.allocator.exit_on_oom=false", "-Dpulsar.allocator.out_of_memory_policy=FallbackToHeap",