From 4274887879e60d75fd0115b1a4e8b7a17d353f06 Mon Sep 17 00:00:00 2001 From: Ross Lawley Date: Thu, 13 Aug 2026 15:27:09 +0100 Subject: [PATCH 1/2] Stop the buffer pool pruner thread when the pool is empty PROBLEM The driver starts a thread that removes idle buffers from PowerOfTwoBufferPool.DEFAULT. That thread never stops. A thread that runs forever keeps the class loader of all driver classes in memory. The static data of those classes also stays in memory. Then an application server cannot unload an application, and the memory of that application stays in use. Users report this behavior in GitHub issue 2029 and in JAVA-5643. CAUSE DEFAULT is a static field, and it calls enablePruning() during class initialization. That method schedules a periodic task. The executor starts the worker thread for the first task, but the pool is empty at that time. Therefore the thread has no work, and it continues to wake up forever. The thread keeps the class loader in memory because the JVM captures the class loader when it constructs the thread. The data that the thread holds is not the cause. For this reason, a change to the references of the thread cannot release the class loader. The thread must stop. SOLUTION enablePruning() now sets a flag. It does not schedule a task. The release() method schedules one prune when it puts a buffer into the pool. Each prune schedules the next prune, but only if the pool still holds a buffer. The pruner does not schedule the next prune when the pool becomes empty. The pruner also uses these settings on its executor: - allowCoreThreadTimeOut(true), so that the worker thread can stop - a keep-alive time of maxIdleTime / 2 - setRemoveOnCancelPolicy(true) The work queue becomes empty after the last prune. Then the keep-alive time expires, and the worker thread stops. A later call to release() schedules a new prune, and the executor starts a new thread. Two threads must not schedule a prune at the same time. A prune must also not stop while a different thread adds a buffer to the pool. The AtomicBoolean pruningScheduled prevents both conditions. The prune clears the flag, and then it examines the pool one more time before it stops. DRAWBACKS The pool releases the class loader about 90 seconds after the last buffer release. The default value of maxIdleTime is one minute. A buffer is old enough to remove only after two prunes, and the keep-alive time adds 30 seconds. The class loader stays in memory during that period. A tool that examines threads at the moment of an undeployment can still find a live thread. A pool that becomes idle and then busy starts a new thread. This adds a small cost. A test measures this cost. A busy pool keeps one thread, because new work arrives before the keep-alive time expires. prune() keeps its current behavior after an error. It writes a log message and throws the error again, and the pruner does not start again. This behavior is the same as before this change. PRIOR ART Netty has the same problem and uses the same solution. GlobalEventExecutor is a single-thread singleton. It starts its thread when work arrives, and it stops the thread when the task queue stays empty for a quiet period. The deprecated ThreadDeathWatcher class uses the same pattern. The steps that this change uses to clear and then examine the flag follow GlobalEventExecutor.TaskRunner. Netty also sets the context class loader of a new thread to null. See netty#7290 and JDK-7008595. That change corrects a different problem, which is a driver thread that keeps an application class loader in memory. This commit does not include that change. TESTS New tests in PowerOfTwoBufferPoolTest show three results. An empty pool starts no thread. The thread stops after the pruner empties the pool. The pruner starts again after the thread stops. A separate test harness measures class loader retention. That harness loads the driver into a child class loader, opens a MongoClient, closes it, and then waits for the class loader to become unreachable. Before this change, the class loader stayed in memory. After this change, the JVM collects it. The harness is a local development tool, and it is not part of this commit. JAVA-6279 --- .../connection/PowerOfTwoBufferPool.java | 112 +++++++++++++++++- .../connection/PowerOfTwoBufferPoolTest.java | 82 ++++++++++++- 2 files changed, 186 insertions(+), 8 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/PowerOfTwoBufferPool.java b/driver-core/src/main/com/mongodb/internal/connection/PowerOfTwoBufferPool.java index a8c7f87a24e..26ddd595de5 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/PowerOfTwoBufferPool.java +++ b/driver-core/src/main/com/mongodb/internal/connection/PowerOfTwoBufferPool.java @@ -28,9 +28,10 @@ import java.util.HashMap; import java.util.Map; import java.util.concurrent.ConcurrentLinkedDeque; -import java.util.concurrent.Executors; -import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.ScheduledThreadPoolExecutor; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; /** *

This class is not part of the public API and may be removed or changed at any time

@@ -40,6 +41,13 @@ public class PowerOfTwoBufferPool implements BufferProvider { /** * The global default pool. Pruning is enabled on this pool. Idle buffers are pruned after one minute. + * + *

The pruner thread does not run all the time. It starts when the pool holds a buffer. It stops when the pool + * becomes empty.

+ * + *

The pruner thread must stop. A thread that runs forever keeps the class loader of all driver classes in + * memory. The static data of those classes also stays in memory. Then an application server cannot unload the + * application. See JAVA-6279.

*/ public static final PowerOfTwoBufferPool DEFAULT = new PowerOfTwoBufferPool().enablePruning(); @@ -63,7 +71,13 @@ public ByteBuffer getBuffer() { private final Map powerOfTwoToPoolMap = new HashMap<>(); private final long maxIdleTimeNanos; - private final ScheduledExecutorService pruner; + private final ScheduledThreadPoolExecutor pruner; + /** + * True if the pruner has a scheduled prune. Two threads must not schedule a prune at the same time, and this flag + * prevents that. The method {@link #pruneAndRescheduleIfNeeded()} also uses this flag when it stops the pruner. + */ + private final AtomicBoolean pruningScheduled = new AtomicBoolean(); + private volatile boolean pruningEnabled; /** * Construct an instance with a highest power of two of 24. @@ -96,21 +110,51 @@ public ByteBuffer getBuffer() { powerOfTwo = powerOfTwo << 1; } maxIdleTimeNanos = timeUnit.toNanos(maxIdleTime); - pruner = Executors.newSingleThreadScheduledExecutor(new DaemonThreadFactory("BufferPoolPruner")); + pruner = new ScheduledThreadPoolExecutor(1, new DaemonThreadFactory("BufferPoolPruner")); + // The worker thread must stop when it has no more work. Then an idle pool holds no thread. + // + // These three settings are sufficient only because this class schedules one prune at a time. It schedules the + // next prune only if the pool is not empty. Then the work queue becomes empty and the keep-alive time expires. + // A periodic task stays in the work queue forever. Then the worker thread always has a task to wait for, and + // the keep-alive time never expires. + // + // The keep-alive time applies only after the last prune. While a prune is in the work queue, the worker thread + // waits for that prune. Because of this, a short keep-alive time does not change the interval between prunes. + // A short keep-alive time also decreases the time that an idle pool keeps our class loader in memory. + pruner.setKeepAliveTime(Math.max(1, maxIdleTimeNanos / 2), TimeUnit.NANOSECONDS); + pruner.allowCoreThreadTimeOut(true); + pruner.setRemoveOnCancelPolicy(true); } /** - * Call this method at most once to enable a background thread that prunes idle buffers from the pool + * Call this method one time only. It permits the pool to prune idle buffers. + * + *

This method does not start a thread. An empty pool has no buffers to prune. The pruner starts when you + * {@linkplain #release(ByteBuffer) release} a buffer. The pruner stops when the pool becomes empty.

*/ PowerOfTwoBufferPool enablePruning() { - pruner.scheduleAtFixedRate(this::prune, maxIdleTimeNanos, maxIdleTimeNanos / 2, TimeUnit.NANOSECONDS); + pruningEnabled = true; + if (!allPoolsEmpty()) { + // The pool can hold buffers from before this call, and those buffers also need a prune. An empty pool + // must not start a thread. + startPruningIfNeeded(); + } return this; } void disablePruning() { + pruningEnabled = false; pruner.shutdownNow(); } + /** + * @return The number of threads that the pruner uses. This method is package-private because the tests must show + * that no thread runs when the pool has no buffers to prune. JAVA-6279 is about that behavior. + */ + int prunerThreadCount() { + return pruner.getPoolSize(); + } + @Override public ByteBuf getBuffer(final int size) { return new PooledByteBufNIO(getByteBuffer(size)); @@ -136,7 +180,59 @@ public void release(final ByteBuffer buffer) { powerOfTwoToPoolMap.get(log2(roundUpToNextHighestPowerOfTwo(buffer.capacity()))); if (pool != null) { pool.release(new IdleTrackingByteBuffer(buffer)); + startPruningIfNeeded(); + } + } + + private void startPruningIfNeeded() { + if (pruningEnabled && pruningScheduled.compareAndSet(false, true)) { + schedulePrune(); + } + } + + private void schedulePrune() { + try { + pruner.schedule(this::pruneAndRescheduleIfNeeded, maxIdleTimeNanos / 2, TimeUnit.NANOSECONDS); + } catch (RejectedExecutionException e) { + // Another thread called `disablePruning` and stopped the executor. A release of a buffer must not fail + // because of this. + pruningScheduled.set(false); + } + } + + /** + * Prunes the pool. Then schedules the next prune, but only if the pool is not empty. + * + *

This method does not cancel a task to stop the pruner. It stops the pruner when it does not schedule the next + * prune. Then the work queue becomes empty and the pruner thread stops.

+ * + *

The steps below prevent a lost pruner. A thread that releases a buffer reads {@link #pruningScheduled}. If + * that flag is true, the thread does not schedule a prune, because it relies on this method to schedule the next + * prune. For this reason, this method clears the flag and then examines the pool one more time. If the pool is not + * empty, this method takes the next prune. If it cannot take the next prune, the other thread has taken it. The + * class {@code io.netty.util.concurrent.GlobalEventExecutor.TaskRunner} uses the same steps.

+ */ + private void pruneAndRescheduleIfNeeded() { + prune(); + if (allPoolsEmpty()) { + pruningScheduled.set(false); + if (allPoolsEmpty()) { + return; + } + if (!pruningScheduled.compareAndSet(false, true)) { + return; + } + } + schedulePrune(); + } + + private boolean allPoolsEmpty() { + for (BufferPool pool : powerOfTwoToPoolMap.values()) { + if (!pool.isEmpty()) { + return false; + } } + return true; } private void prune() { @@ -204,5 +300,9 @@ void prune() { long now = System.nanoTime(); available.removeIf(cur -> now - cur.getLastUsedNanos() >= maxIdleTimeNanos); } + + boolean isEmpty() { + return available.isEmpty(); + } } } diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/PowerOfTwoBufferPoolTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/PowerOfTwoBufferPoolTest.java index e2b439ba6c6..3cf50c80078 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/PowerOfTwoBufferPoolTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/PowerOfTwoBufferPoolTest.java @@ -22,10 +22,12 @@ import java.nio.ByteBuffer; import java.util.concurrent.TimeUnit; +import java.util.function.BooleanSupplier; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotSame; import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertTrue; public class PowerOfTwoBufferPoolTest { private PowerOfTwoBufferPool pool; @@ -75,7 +77,6 @@ public void testHugeBufferRequest() { assertNotSame(buf, pool.getBuffer((int) Math.pow(2, 10) + 1)); } - // Racy test @Test public void testPruning() throws InterruptedException { PowerOfTwoBufferPool pool = new PowerOfTwoBufferPool(10, 5, TimeUnit.MILLISECONDS) @@ -84,11 +85,88 @@ public void testPruning() throws InterruptedException { ByteBuf byteBuf = pool.getBuffer(256); ByteBuffer wrappedByteBuf = byteBuf.asNIO(); byteBuf.release(); - Thread.sleep(50); + // The pruner stops only after it empties the pool. Therefore a thread count of zero shows that the pruner + // removed the buffer. A wait for a fixed period would make this test racy. + assertTrue("the pruner must empty the pool", await(() -> pool.prunerThreadCount() == 0)); ByteBuf newByteBuf = pool.getBuffer(256); assertNotSame(wrappedByteBuf, newByteBuf.asNIO()); } finally { pool.disablePruning(); } } + + /** + * The pruner removes idle buffers, and an empty pool has no idle buffers. Therefore {@code enablePruning} must not + * start a thread. A thread that runs keeps the class loader of all driver classes in memory. See JAVA-6279. + */ + @Test + public void testEnablePruningStartsNoThreadWhileThePoolIsEmpty() { + PowerOfTwoBufferPool pool = new PowerOfTwoBufferPool(10, 5, TimeUnit.MILLISECONDS).enablePruning(); + try { + assertEquals(0, pool.prunerThreadCount()); + } finally { + pool.disablePruning(); + } + } + + /** + * The pruner empties the pool. Then it has no more work, and the thread must stop. The thread must not continue to + * wake up. This behavior is the correction for JAVA-6279. + */ + @Test + public void testPrunerThreadTerminatesOnceThePoolIsDrained() throws InterruptedException { + PowerOfTwoBufferPool pool = new PowerOfTwoBufferPool(10, 5, TimeUnit.MILLISECONDS).enablePruning(); + try { + pool.getBuffer(256).release(); + assertTrue("the pruner thread should terminate once the pool is drained", + await(() -> pool.prunerThreadCount() == 0)); + } finally { + pool.disablePruning(); + } + } + + /** + * The pruner must start again. A pool can become idle and then busy. If the pruner does not start again, the pool + * keeps the buffers that you release after the idle period. + */ + @Test + public void testPruningResumesAfterTheThreadHasTerminated() throws InterruptedException { + PowerOfTwoBufferPool pool = new PowerOfTwoBufferPool(10, 5, TimeUnit.MILLISECONDS).enablePruning(); + try { + pool.getBuffer(256).release(); + assertTrue("precondition: the pruner thread terminates once drained", + await(() -> pool.prunerThreadCount() == 0)); + + ByteBuf byteBuf = pool.getBuffer(256); + ByteBuffer wrapped = byteBuf.asNIO(); + byteBuf.release(); + assertTrue("a buffer released after termination should still be pruned", + await(() -> pool.getBuffer(256).asNIO() != wrapped)); + } finally { + pool.disablePruning(); + } + } + + /** A pool without pruning must not start a pruner thread. The number of buffers does not change this behavior. */ + @Test + public void testPruningDisabledPoolNeverStartsAThread() { + ByteBuf byteBuf = pool.getBuffer(256); + ByteBuffer wrapped = byteBuf.asNIO(); + byteBuf.release(); + // This assertion needs no wait. The executor creates its worker thread when it accepts a task, and not when it + // runs that task. Therefore a pool that schedules a prune has a thread before `release` returns. + assertEquals(0, pool.prunerThreadCount()); + assertSame("the pool must keep the buffer because it does not prune", wrapped, pool.getBuffer(256).asNIO()); + } + + private static boolean await(final BooleanSupplier condition) throws InterruptedException { + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (System.nanoTime() < deadline) { + if (condition.getAsBoolean()) { + return true; + } + Thread.sleep(5); + } + return condition.getAsBoolean(); + } } From 9a68670c9162d4fbb2173919e8399e690723dacc Mon Sep 17 00:00:00 2001 From: Ross Lawley Date: Thu, 13 Aug 2026 16:38:39 +0100 Subject: [PATCH 2/2] Included automated poc code and FINDINGS.md --- .gitignore | 1 + testing/java-6279-poc/FINDINGS.md | 668 +++++++++++++++ testing/java-6279-poc/classpath.gradle | 12 + testing/java-6279-poc/run.sh | 110 +++ .../src/java6279/ExecutorMechanism.java | 760 ++++++++++++++++++ testing/java-6279-poc/src/java6279/Poc.java | 666 +++++++++++++++ .../src/java6279/primer/Inert.java | 35 + .../primer/InheritsContextClassLoader.java | 48 ++ .../InheritsContextClassLoaderButNulled.java | 48 ++ .../InheritsContextClassLoaderNettyDance.java | 48 ++ .../primer/RegistersShutdownHook.java | 52 ++ .../RegistersShutdownHookNettyStyle.java | 58 ++ .../RegistersShutdownHookParentBody.java | 52 ++ .../src/java6279/primer/StartsOwnThread.java | 45 ++ .../primer/StartsOwnThreadNettyStyle.java | 54 ++ .../primer/StartsParentBuiltThread.java | 42 + .../primer/StaticSingletonExecutor.java | 70 ++ 17 files changed, 2769 insertions(+) create mode 100644 testing/java-6279-poc/FINDINGS.md create mode 100644 testing/java-6279-poc/classpath.gradle create mode 100755 testing/java-6279-poc/run.sh create mode 100644 testing/java-6279-poc/src/java6279/ExecutorMechanism.java create mode 100644 testing/java-6279-poc/src/java6279/Poc.java create mode 100644 testing/java-6279-poc/src/java6279/primer/Inert.java create mode 100644 testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoader.java create mode 100644 testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoaderButNulled.java create mode 100644 testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoaderNettyDance.java create mode 100644 testing/java-6279-poc/src/java6279/primer/RegistersShutdownHook.java create mode 100644 testing/java-6279-poc/src/java6279/primer/RegistersShutdownHookNettyStyle.java create mode 100644 testing/java-6279-poc/src/java6279/primer/RegistersShutdownHookParentBody.java create mode 100644 testing/java-6279-poc/src/java6279/primer/StartsOwnThread.java create mode 100644 testing/java-6279-poc/src/java6279/primer/StartsOwnThreadNettyStyle.java create mode 100644 testing/java-6279-poc/src/java6279/primer/StartsParentBuiltThread.java create mode 100644 testing/java-6279-poc/src/java6279/primer/StaticSingletonExecutor.java diff --git a/.gitignore b/.gitignore index 5581f51dc17..6534e4a9bc5 100644 --- a/.gitignore +++ b/.gitignore @@ -52,6 +52,7 @@ local.properties *.sh !.evergreen/*.sh !scripts/*.sh +!testing/java-6279-poc/*.sh # security-sensitive files *.gpg diff --git a/testing/java-6279-poc/FINDINGS.md b/testing/java-6279-poc/FINDINGS.md new file mode 100644 index 00000000000..b3e67a7bb9b --- /dev/null +++ b/testing/java-6279-poc/FINDINGS.md @@ -0,0 +1,668 @@ +# JAVA-6279 — class loader retention findings + +Internal working notes for [JAVA-6279](https://jira.mongodb.org/browse/JAVA-6279) (*Stop BufferPoolPruner thread when last MongoClient +closes*), [GitHub issue 2029](https://github.com/mongodb/mongo-java-driver/issues/2029) and +[JAVA-5643](https://jira.mongodb.org/browse/JAVA-5643). Touches JAVA-6240 (`CommonExecutor`) in §5. + +This directory began as a portable rework of Valentin Kovalenko's +[`primer` experiment](https://github.com/stIncMale/mongo-java-driver/commit/862b7d75fa0629e2b7c9cc4d6e8761b1678934dd), which hardcoded an +absolute path to one developer's `build/classes` directory and printed its results. It now resolves paths at run time, adds control +scenarios and a negative control, extends from synthetic classes to the driver itself, and compares every outcome against a stated +expectation so it can be run unattended. + +Claims are tagged **[executed]** = observed by running this code, **[code]** = read from source, **[unmeasured]** = not established either +way. + +Run it with `./testing/java-6279-poc/run.sh` (see [How to run](#how-to-run)). Nothing under any module's `src/main` is modified by the +harness. + +## 1. Verdict summary + +19 class loader scenarios and 12 executor mechanism checks, the latter under every JDK on the machine. Last full run: +all scenarios matched expectation, 50 PASS / 10 INFO / 0 FAIL. `PINNED` means the class loader was still strongly reachable after a window +of `System.gc()` nudges (10 s by default, `-Djava6279.gcWindowSeconds` to change it); +`COLLECTED` means its phantom reference was enqueued. + +### The two conclusions that matter + +1. **A non-terminated thread we start prevents the class loader of all driver classes — and therefore all of their static state — from being + collected.** **[executed]** +2. **`BufferPoolPruner` is that thread, and after `MongoClient.close()` it is the only thing left holding the loader.** + **[executed]** + +The second is load-bearing and is why `driver/OPEN_AND_CLOSE_CLIENT_THEN_DISABLE_PRUNING` exists. Knowing the pruner survives `close()` is +not enough — if anything *else* also survived, fixing the pruner would not release the loader. Nothing else does, so terminating it is +sufficient. + +### Driver scenarios + +| Scenario | Before the fix | After the fix | +|-----------------------------------------------------|----------------------------------------------------------|------------------------------------------------------------------------------| +| `driver/LOAD_ONLY` | COLLECTED | COLLECTED — control: loading driver classes leaks nothing | +| `driver/TOUCH_DEFAULT_POOL` | **PINNED** | **COLLECTED** — an empty pool now starts no thread | +| `driver/TOUCH_DEFAULT_POOL_THEN_DISABLE_PRUNING` | COLLECTED | COLLECTED | +| `driver/OPEN_AND_CLOSE_CLIENT` | **PINNED** | **COLLECTED after ~90 s** — the reported symptom, fixed | +| `driver/OPEN_AND_CLOSE_CLIENT_THEN_DISABLE_PRUNING` | COLLECTED | COLLECTED — before the fix, this was the proof the pruner was the *only* pin | +| `…_AND_TOUCH_COMMON_EXECUTOR` | skipped on `main`; **PINNED** on the backpressure branch | unchanged — `CommonExecutor` is a separate pin, see §5 | + +The ~90 s is inherent, not slack: with a one minute `maxIdleTime` a released buffer is only evictable after 60 s (two prune cycles at +`maxIdleTime / 2`), and the thread then times out after the keep-alive. **Any test of this fix must allow for that tail** — hence +`-Djava6279.driverGcWindowSeconds`, default 150. + +### Primer scenarios: what pins, and what does not + +| Scenario | Result | Isolates | +|-----------------------------------------------|------------|---------------------------------------------------------------------------------------------------------------------------------------| +| `primer/Inert` | COLLECTED | Control: the harness can observe a child loader being collected at all. | +| `primer/StartsOwnThread` | **PINNED** | A thread *constructed* in a child-loaded class's static initializer pins the loader, even with a parent-loaded `Runnable`. | +| `primer/StartsParentBuiltThread` | COLLECTED | Starting a `Thread` a *parent*-loaded class constructed does not pin. Isolates construction as the capture point. | +| `cclOnly/inherited` | **PINNED** | With **no child frame on the stack**, inheriting the child loader as context class loader pins it. A second, independent edge. | +| `cclOnly/nulled` | COLLECTED | Nulling that context class loader — *after* construction — closes that edge. | +| `primer/StartsOwnThreadNettyStyle` | **PINNED** | Netty's context class loader dance does not help when the thread's own class is in the loader. Confounded by the stack frame; see §2. | +| `primer/InheritsContextClassLoader` | **PINNED** | Confounded (stack frame present). Retained to show the stack capture dominates. | +| `primer/InheritsContextClassLoaderButNulled` | **PINNED** | Confounded, as above. | +| `primer/InheritsContextClassLoaderNettyDance` | **PINNED** | Confounded, as above — even nulling the *calling* thread's loader before construction cannot remove a stack frame. | +| `primer/RegistersShutdownHook` | **PINNED** | A shutdown hook pins even though the class starts no thread — see §7. | +| `primer/RegistersShutdownHookNettyStyle` | **PINNED** | Adding the context class loader nulling does not rescue it. | +| `primer/RegistersShutdownHookParentBody` | COLLECTED | The only non-pinning hook shape, and it cannot call driver code. | +| `primer/StaticSingletonExecutor` | **PINNED** | Models `CommonExecutor`, and shows the proposed `Cleaner` fix can never run — see §5. | + +### Executor mechanism checks + +Run at `--release 8` under **JDK 8, 11, 17, 23 (GraalVM) and 26**, because this leans on `ScheduledThreadPoolExecutor` +*implementation* behaviour rather than documented contract, and the driver's baseline is Java 8. + +| Check | Result | +|--------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------| +| The two settings alone do not reap the worker — a decision to stop is required | `poolSize=1, queue=1` while idle at 10× the keep-alive | +| One-shot scheduling self-terminates with no drain check (the `CommonExecutor` shape) | `poolSize=0` after the task runs, resurrects, back to 0 | +| A pending long one-shot delay survives a much shorter keep-alive | 3000 ms delay, 100 ms keep-alive → fired at 3002–3004 ms | +| Repeated resurrection is reliable over 500 create/reap cycles | 0 tasks lost, 500 threads created | +| Concurrent scheduling across resurrection loses nothing (8 × 250) | all ran, 0 rejections, **1 thread created** | +| **The conditional-reschedule design holds under contention** | 0 orphaned pools, 0 threads left — and **7/20 rounds orphaned with the re-check removed** | +| Reaps the worker once a periodic task is cancelled | `poolSize` 1 → 0 | +| Stays reusable and resurrects on re-scheduling | `isShutdown=false`, a *new* thread created | +| Self-cancellation from inside the task stops the repeat, 2000 round trips | 0 lost, 0 repeats not stopped | +| *(INFO)* The future needs safe publication | 0–13 per 2000 round trips throw a swallowed `NullPointerException` | +| *(INFO)* Cancelled task retained in the queue without `removeOnCancelPolicy` | `queue=1, poolSize=1` on most JDKs; JDK 8 reached 0 anyway | +| A generous keep-alive avoids thread churn, 25 cycles | 1 ms → 25 threads; 2 s → 1 thread | + +## 2. What pins the loader: two independent edges + +There are **two** distinct retaining edges. Conflating them wasted time here, and they have different fixes. + +### Edge A — capture at thread construction + +`primer/StartsOwnThread` (PINNED) versus `primer/StartsParentBuiltThread` (COLLECTED) isolates this to the moment +`new Thread(...)` runs: + +- Both run the same parent-loaded `Poc.SLEEPING_RUNNABLE`, so the executing thread holds no reference into the child loader by way of its + task. **[code]** +- In both, the thread's context class loader is the *application* loader, not the child loader — the harness prints it. So edge B is not + what is acting here. **[executed]** +- The only difference is which class was on the stack when the constructor ran. That alone flips the outcome. **[executed]** + +`primer/StartsOwnThread` deliberately uses `new Thread(null, runnable, name, 1, false)` — no thread group, no inherited thread locals — so +the pinning cannot be attributed to inherited state. **[code]** + +The exact retaining field inside `java.lang.Thread` was not identified. It was not needed, and it is a JDK implementation detail rather than +a contract. **[unmeasured]** + +**Consequence:** a thread constructed from driver code pins the driver's own loader, and driver code is on that stack by definition. Nothing +can be nulled or cleared away. **The thread has to actually terminate.** This is JAVA-6279. + +### Edge B — the inherited context class loader + +`cclOnly/*` isolates this by constructing the thread from `Poc`, with **no child-loaded frame on the stack**, while the calling thread's +context class loader is the child loader. So edge A is absent and only edge B can act: + +| Scenario | Result | +|---------------------------------------------------------------------|---------------| +| `cclOnly/inherited` | **PINNED** | +| `cclOnly/nulled` — context class loader nulled *after* construction | **COLLECTED** | + +So the context class loader is a genuine independent edge, and nulling it closes it. **Nulling after construction is sufficient**; Netty's +dance around the calling thread is not required for this edge. **[executed]** + +**Consequence:** this is the edge where a driver thread created on an application thread's behalf pins the *application's* loader. +`t.setContextClassLoader(null)` in `DaemonThreadFactory.newThread` closes it. That is a distinct bug from this ticket, and the change is +**not** in the tree — see the recommendation in §8. + +### Why the `primer/Inherits*` scenarios are retained but prove nothing about edge B + +Those three scenarios attempt edge B from inside a child-loaded class's ``, which necessarily puts a child frame on the stack — so +edge A is present too and dominates. All three are PINNED, including Netty's full null-the-caller-then-restore dance. **That says nothing +about the mitigation**, and an earlier version of these notes wrongly concluded from them that nulling after construction was "too late". +They are kept only as evidence that edge A dominates whenever it is present. `cclOnly/*` is the clean isolation. + +| Edge | Loader pinned | Closed by | +|------------------------------------|-----------------------|------------------------------------------| +| A — construction capture | the **driver's own** | the thread terminating (JAVA-6279) | +| B — inherited context class loader | the **application's** | nulling the CCL in `DaemonThreadFactory` | + +Both are worth fixing. Neither substitutes for the other. + +## 3. The driver path **[code]** + +`driver-core/src/main/com/mongodb/internal/connection/PowerOfTwoBufferPool.java`: + +```java +public static final PowerOfTwoBufferPool DEFAULT = new PowerOfTwoBufferPool().enablePruning(); + +PowerOfTwoBufferPool(final int highestPowerOfTwo, final long maxIdleTime, final TimeUnit timeUnit) { + // ... + pruner = Executors.newSingleThreadScheduledExecutor(new DaemonThreadFactory("BufferPoolPruner")); +} + +PowerOfTwoBufferPool enablePruning() { + pruner.scheduleAtFixedRate(this::prune, maxIdleTimeNanos, maxIdleTimeNanos / 2, TimeUnit.NANOSECONDS); + return this; +} + +void disablePruning() { + pruner.shutdownNow(); +} +``` + +Three details the scenarios confirm: + +- The executor's constructor does not start a thread; `ScheduledThreadPoolExecutor` starts its core worker lazily on first submission. It is + the `scheduleAtFixedRate` in `enablePruning()` that creates the thread. Consistent with + `driver/LOAD_ONLY` COLLECTED versus `driver/TOUCH_DEFAULT_POOL` PINNED. **[executed]** +- `DEFAULT` is a static field with pruning enabled at class initialization, and there is no public path to + `disablePruning()` — hence the reflective workaround in issue 2029, which + `driver/TOUCH_DEFAULT_POOL_THEN_DISABLE_PRUNING` reproduces and shows still works. **[executed]** +- The pruner runs every `maxIdleTimeNanos / 2` **forever**, including against a permanently empty pool. See §4 on why that is worth changing + beyond the leak itself. + +The `DaemonThreadFactory` marking is why this is a memory leak rather than a JVM-shutdown hang: the thread never blocks process exit, so +nothing surfaces until an application server undeploys or redeploys, or an OSGi bundle is refreshed. Then the whole driver class graph, plus +every application class in the same loader, stays in the heap. **[code]** + +`DEFAULT` is also **not owned by any `MongoClient`** — `driver-legacy`'s `DBCursor` and `DBCollection` and +`driver-benchmarks` use it with no client at all. **[code]** So the ticket's title, "stop the thread when the last +`MongoClient` closes", does not describe an implementable condition. Reference counting clients (community +[PR 2032](https://github.com/mongodb/mongo-java-driver/pull/2032), which triage deferred) would invent an ownership relation that does not +exist and still leave those consumers unaccounted for. + +## 4. The fix: a self-terminating, self-resurrecting pruner — IMPLEMENTED **[executed]** + +**Status: implemented on branch `JAVA-6279`** in `PowerOfTwoBufferPool`, with tests in `PowerOfTwoBufferPoolTest`. driver-core: 5820 tests, +0 failures, 0 errors; checkstyle and spotbugs clean. The harness now reports every +`driver/*` scenario COLLECTED, including `OPEN_AND_CLOSE_CLIENT`, which was the reported symptom. + +Key the thread's lifetime to the thing it exists to serve — whether the pool holds anything: + +- pool empty → nothing to prune → no thread should exist; +- a buffer is released → start pruning; +- `prune()` finds the pool drained → stop. + +This also satisfies the requirement raised on the ticket that shutting the pruner down must not leave the pool occupying memory: under "stop +only when drained", an empty pool *is* the termination condition, so there is nothing left to clear. + +### 4.1 Stop by not rescheduling, not by cancelling + +Replace `scheduleAtFixedRate` with a one-shot `schedule` that **reschedules itself only if the pool still holds something**: + +```java +// prune() +evictIdleBuffers(); +if( + +allPoolsEmpty()){ + pruningScheduled. + +set(false); + if( + +allPoolsEmpty()){ + return; // stop: queue empties, worker times out + } + if(!pruningScheduled. + +compareAndSet(false,true)){ + return; // a releaser owns the next run + } + } + +schedule(this::prune, maxIdleTimeNanos /2, NANOSECONDS); + +// release(buffer) +pool. + +addLast(buffer); +if(pruningScheduled. + +compareAndSet(false,true)){ + +schedule(this::prune, maxIdleTimeNanos /2, NANOSECONDS); +} +``` + +Three advantages over having `prune()` cancel its own periodic future: + +1. **The publication hazard (§4.4) cannot arise**, because the task never needs a reference to its own + `ScheduledFuture`. +2. **"Stop" becomes not doing something** rather than an action. No `cancel`, and `removeOnCancelPolicy` stops being load-bearing on the + stop path. +3. **No wakeups at all when idle**, where today the pruner wakes every `maxIdleTimeNanos / 2` forever. It also drops + `scheduleAtFixedRate`'s catch-up burst behaviour after a GC pause, which is undesirable for a pruner. + +An earlier version of these notes claimed a self-rescheduling one-shot "behaves the same way" as `scheduleAtFixedRate` +because the next run is queued before the current one ends. **That was wrong**: it is true only of *unconditional* +rescheduling. Conditional rescheduling lets the queue empty, which is the whole point. + +### 4.2 The executor recipe + +```java +ScheduledThreadPoolExecutor pruner = new ScheduledThreadPoolExecutor(1, factory); +pruner. + +setKeepAliveTime(keepAlive, unit); // must be > 0 +pruner. + +allowCoreThreadTimeOut(true); // let the core worker die when idle +pruner. + +setRemoveOnCancelPolicy(true); // recommended; see below +``` + +`pruner.shutdown()` when drained would be the obvious alternative and is a trap: a shut-down +`ScheduledThreadPoolExecutor` rejects further submissions, so resurrection would mean building a new executor per cycle, which means a +non-final field and a lock guarding it. Not shutting down avoids all of that. + +- **`setRemoveOnCancelPolicy(true)`** — recommended, but the evidence is weaker than an earlier version of these notes claimed. The intent + is that a cancelled task should not sit in the `DelayedWorkQueue` until its delay elapses. Most JDKs here show exactly that without it + (`queue=1, poolSize=1`), but JDK 8 reached `queue=0, poolSize=0` anyway, so the measurement is reported as `INFO`, not asserted. Set it + because it is free and because + `MongoScheduledThreadPoolExecutor` already does — not because of that measurement. +- **The keep-alive is a thread churn dial, not a correctness one.** See §4.2.1 for what a short value does and does + not cost. + +#### 4.2.1 What a short keep-alive costs **[executed]** + +The keep-alive is the one tuning decision in this fix, and the trade is not churn against correctness. It is churn +against **how long the class loader stays pinned**. A longer value means fewer threads and a longer leak tail. A +shorter value means prompter release and more thread creation. + +What a short keep-alive does **not** cost, all measured: + +| Concern | Result | +|---|---| +| Does it change the schedule? | **No.** The keep-alive applies only when the work queue is empty. While a task is pending, the worker parks on the delayed queue. Keep-alive 50 ms against a 500 ms period over 5 s: zero missed executions, one thread created, on JDK 8, 11 and 26. | +| Does it drop pending work? | **No.** A 3000 ms delay with a 100 ms keep-alive fired at 3002–3004 ms on all five JDKs. | +| Does it delay a scheduled task? | **No.** `ScheduledThreadPoolExecutor` creates the worker when it accepts the task, not when the task fires, so thread creation is paid at `schedule()` time rather than at the deadline. | +| Does it lose tasks in the die-and-resurrect race? | **No.** 500 forced create and reap cycles lost 0 tasks. 8 threads scheduling 250 tasks each lost 0 tasks and saw 0 rejections. | + +What it does cost: + +- **Thread churn, in one pattern only** — work that arrives at intervals *longer* than the keep-alive. Measured: a 1 ms + keep-alive over 25 cycles created **25 threads**; a 2 s keep-alive over the same 25 cycles created **1**. The cost per + thread is tens of microseconds, against a retry backoff of milliseconds or a prune interval of 30 seconds. The churn + had to be forced artificially to measure it at all. +- **Observability noise**, which is the less obvious cost. `DaemonThreadFactory` names threads from a monotonic + counter, so churn produces `CommonScheduler-1-thread-1`, `-2`, `-3` without bound. The 500-cycle check produced 500 + distinct thread names. Anything keyed on thread name — APM tools, JFR thread-start events, log correlation — + accumulates entries for threads that no longer exist. +- **A hard floor.** The keep-alive must be greater than zero, because `allowCoreThreadTimeOut(true)` rejects zero. This + is why `PowerOfTwoBufferPool` uses `Math.max(1, maxIdleTimeNanos / 2)`. A pool that is configured with a very small + idle time would otherwise fail during construction. + +Because the measured costs are microseconds and thread names, a short value is the better default. The pruner is safer +than the general case: prunes are `maxIdleTime / 2` apart and the queue stays non-empty during an active run, so churn +occurs only when a pool drains and then fills again after a gap. + +### 4.3 The stop-versus-release race, and testing it honestly **[executed]** + +The remaining risk: a releasing thread concludes "pruning is already scheduled" while `prune()` concurrently concludes +"drained, stopping". Get it wrong and buffers sit in the pool with no pruner — a class loader leak traded for a buffer leak. The check / +re-check above handles it, and Netty's `GlobalEventExecutor.TaskRunner` (§6) is the reference implementation. + +Prototyped as `conditionalRescheduleDesignHoldsUnderContention`, 20 rounds × 6 releasers releasing for 400 ms, with a deliberately widened +interleaving window, on all five JDKs: + +| Variant | Orphaned pools | Threads left | +|---------------------------------------------------|--------------------|--------------| +| Protocol intact | **0** | 0 | +| Re-check omitted (`-Djava6279.breakRecheck=true`) | **7 of 20 rounds** | 0 | + +**The negative control is the important number, and it took three attempts to get.** The first two versions passed *even with the protocol +deliberately broken* — first because the orphan window is nanoseconds wide, then because fixed-count producers finished before the first +`prune` ran, so the stop path never overlapped a releaser. Both dead ends are recorded in the source comments. + +> A concurrency test that has not been shown to fail against the broken implementation is not evidence. + +That applies directly to whatever test ships with the fix. + +### 4.4 The publication hazard, if you cancel a future anyway **[executed]** + +Recorded because it was found the hard way, and because anyone reaching for the cancel-self design will meet it. + +`scheduleAtFixedRate` **can begin running the task before it returns**, so a field assigned from its return value is not safely visible to +the task. The task reads null, dies with a `NullPointerException` — and a `ScheduledFuture` +*swallows* the throwable, so nothing is logged and nothing throws anywhere visible. The observable result is a pool holding buffers with no +pruner: a lost pruner, silently. + +Rate on an otherwise idle machine: **13/2000 round trips on JDK 8, 1–2/2000 on 11 through 26, and 0/2000 on some runs of the same JDK** — +load sensitive, roughly 0.5% at worst. A 200-round-trip sample reports green about a third of the time, which is exactly what happened: a +200-round-trip version passed repeatedly, then failed once during a full run, which is the only reason it was noticed. + +Reported as `INFO` and not gating the exit status: asserting that a race *does* reproduce is a flaky test by construction, and a run +observing 0 has disproved nothing. + +**If you take §4.1, this cannot arise at all.** If you cancel a future anyway, publish it under the same lock that guards start and stop. + +The *resurrect*-versus-die race, by contrast, could not be provoked, and `ThreadPoolExecutor.processWorkerExit` shows why: **[code]** + +```java +int min = allowCoreThreadTimeOut ? 0 : corePoolSize; +if(min ==0&&!workQueue. + +isEmpty()) +min =1; // don't leave a non-empty queue unattended +``` + +### 4.5 What the fix does and does not achieve + +Resurrection re-pins the loader, which is correct — the loader is in use again. The property that matters is that a quiet application +reaches a state with no live driver thread, bounded by roughly `maxIdleTime` to `1.5 × maxIdleTime` +after the last release (60–90 s with today's default). An undeployed application issues no further releases, so nothing resurrects. + +So the loader is **not** collectable at the instant of `close()`. Any test of the fix must widen its GC window past +`maxIdleTime` — `-Djava6279.gcWindowSeconds` exists for this. + +## 5. `CommonExecutor` (backpressure, JAVA-6240) **[code]** **[executed]** + +Read against [stIncMale PR 3](https://github.com/stIncMale/mongo-java-driver/pull/3), branch `sleepAsync` at +`13d9cc3ec4`. + +```java +public final class CommonExecutor { + private static final CommonExecutor INSTANCE = new CommonExecutor(); + private final MongoScheduledThreadPoolExecutor singleThreadScheduler; + + private CommonExecutor() { + singleThreadScheduler = new MongoScheduledThreadPoolExecutor(1, new DaemonThreadFactory("CommonScheduler")); + } +} +``` + +`singleThreadScheduler` is **never shut down anywhere in the PR** — no `close`, no `shutdown`, no +`allowCoreThreadTimeOut`. The only `allowCoreThreadTimeOut(true)` in the diff is on +`ownedExecutorBackingClientExecutor`, which *is* shut down in `close()`. **[code]** + +### Confirmed by running the harness against that branch **[executed]** + +`run.sh` takes `JAVA6279_DRIVER_REPO` so it can be pointed at another checkout: + +| Scenario | main | backpressure | +|----------------------------------------------|--------------------------|---------------| +| `OPEN_AND_CLOSE_CLIENT` | PINNED | PINNED | +| `OPEN_AND_CLOSE_CLIENT_THEN_DISABLE_PRUNING` | COLLECTED | **COLLECTED** | +| `…_AND_TOUCH_COMMON_EXECUTOR` | *skipped, no such class* | **PINNED** | + +1. `CommonExecutor` is an **independent second pin**: client closed *and* pruning disabled, loader still held, the only addition being one + `CommonExecutor.schedule` call. +2. The pin is **latent**. Creating and closing a client never touches `CommonExecutor`. Its only caller is + `DefaultAsyncClientExecutor.scheduleCompletion` via `sleepAsync`, whose only caller is + `RetryingAsyncCallbackSupplier` on a positive backoff — and only when the backing executor is *not* a + `ScheduledExecutorService`. So it needs an async client, a non-scheduled executor, and a real retry backoff. The sync driver never + reaches it, and it would not show up in a smoke test. + +That narrowness is why the pruner fix remains the higher-value one, and why this is still worth doing before merge: a leak that only appears +under retry backoff gets found in production rather than in CI. + +### The fix is two lines, because the scheduling is one-shot + +| | Scheduling | Does `allowCoreThreadTimeOut` suffice? | +|-------------------------------|----------------------------------------------------------|--------------------------------------------------------| +| `PowerOfTwoBufferPool` pruner | `scheduleAtFixedRate` — queue never empties | **No.** Needs a decision to stop, and with it the race | +| `CommonExecutor` | one-shot `schedule(...)` — queue empties after each task | **Yes.** Complete on its own | + +Verified patch in `CommonExecutor-JAVA-6240.patch`, built on that branch and re-run: the scenario flips **PINNED → COLLECTED at PT30.078s**, +i.e. released exactly when the 30 s keep-alive expired. **[executed]** + +```java +private static final Duration KEEP_ALIVE = Duration.ofSeconds(30); // MUST precede INSTANCE +private static final CommonExecutor INSTANCE = new CommonExecutor(); +// ...in the constructor: +singleThreadScheduler. + +setKeepAliveTime(KEEP_ALIVE.toNanos(),NANOSECONDS); + singleThreadScheduler. + +allowCoreThreadTimeOut(true); +``` + +**Declaration order is not cosmetic.** With `KEEP_ALIVE` after `INSTANCE`, the constructor runs during `` while +`KEEP_ALIVE` is still null: `NullPointerException: Cannot invoke "java.time.Duration.toNanos()"`. The first version of this patch had +exactly that bug and was only caught by running it. **[executed]** + +It composes with the existing close path: `ScheduledCallbackCompletion.reject()` already cancels the future on +`close()`, and with `removeOnCancelPolicy` set that cancellation empties the queue, so a client closed *mid-sleep* also lets the worker +exit. `MongoScheduledThreadPoolExecutor` is used only by `CommonExecutor` and its own test, so hosting the change in `CommonExecutor` keeps +a general-purpose class untouched. + +### Is repeated create/reap a problem? No — the worry inverts **[executed]** + +| Check | Result | +|----------------------------------------|-----------------------------------------------------------------------------| +| 8 threads × 250 concurrent schedules | all ran, 0 rejections, **1 thread ever created** | +| 500 *forced* create/reap cycles | **0 tasks lost**, 500 threads created | +| Long pending delay vs short keep-alive | 3000 ms delay, 100 ms keep-alive → fired at 3002–3004 ms, then `poolSize=0` | + +Under load the thread is **reused**, because work arrives before the keep-alive expires. Churn only occurs when calls are spaced further +apart than the keep-alive — precisely when an extra thread creation is irrelevant, and every caller here is already waiting out a backoff of +milliseconds against a thread creation of tens of microseconds. The churn had to be forced artificially to measure at all. + +The `~12 ms per cycle` the harness prints for the 500-cycle check is dominated by its own 10 ms `awaitPoolSize` poll granularity. **It is +not a measurement of thread creation cost.** + +### The `Cleaner` proposal in the `VAKOTODO` cannot work + +`CommonExecutor` carries: + +> `// VAKOTODO create a ticket and leave a TODO to use Cleaner when we are at Java SE 17 to shut down internal +> executors if the class is GCed.` + +`primer/StaticSingletonExecutor` models it exactly — static singleton, never-shut-down `ScheduledThreadPoolExecutor`, a +`java.lang.ref.Cleaner` registered on the singleton whose action shuts the executor down and holds no reference back. Result: **loader +PINNED, `cleaning action ran: false`.** **[executed]** + +The reachability is circular, so it cannot be otherwise: the cleaning action runs only when the singleton becomes unreachable; the singleton +is a static field, so it is reachable while the class is loaded; the class is loaded while the loader lives; and the loader lives because of +the thread the action was meant to stop. A `Cleaner` frees resources when an object is *forgotten*, and a static singleton is never +forgotten. Java 17 changes nothing. + +**Drop the TODO rather than filing it** — the ticket would be unimplementable as worded, and §4/§5 are the replacement. + +## 6. Prior art: how Netty solves this **[code]** **[executed]** + +From `netty-common` 4.2.9.Final sources, the version this repo already depends on. Netty hit this and arrived at the §4 design +independently, twice. + +`GlobalEventExecutor`: + +> Single-thread singleton `EventExecutor`. It starts the thread automatically and **stops it when there is no task +> pending in the task queue** for `io.netty.globalEventExecutor.quietPeriodSeconds` second (default is 1 second). + +`ThreadDeathWatcher` — deprecated, same shape: + +> When there is no thread to watch (i.e. all threads are dead), the daemon thread **will terminate itself, and a new +> daemon thread will be started again** when a new watch is added. + +Mechanically Netty drives its own loop over its own task queue, with a periodic no-op `quietPeriodTask` as the heartbeat that wakes the +runner to re-check for emptiness, because `GlobalEventExecutor` is not backed by a +`ScheduledThreadPoolExecutor`. Same idea, hand-rolled. + +### The race protocol worth copying + +```java +// in the worker, on finding the queue empty: +boolean stopped = started.compareAndSet(true, false); +assert stopped; + +// Check if there are pending entries added by execute() or schedule*() while we do CAS above. +if(taskQueue. + +isEmpty()){ + // A) No new task was added -> safe to terminate + // B) A new thread started and handled all the new tasks -> safe to terminate + break; + } + +// There are pending tasks added again. + if(!started. + +compareAndSet(false,true)){ + // startThread() started a new thread and set 'started' to true + // -> terminate this thread so the new one reads from taskQueue exclusively + break; + } +// New tasks were added, but this worker was faster to set 'started' to true +// -> keep this thread alive to handle them +``` + +Producer side is `addTask(task); if (!inEventLoop()) { startThread(); }`, with `startThread()` guarded by +`started.compareAndSet(false, true)`. + +This is the check / re-check §4.1 needs, battle-tested, with all three outcomes named. Follow it rather than re-deriving it. + +### Netty's class loader mitigation, and what it does cover + +`GlobalEventExecutor.startThread()` also nulls the context class loader around thread creation: + +```java +ClassLoader parentCCL = /* calling thread's context class loader */; + +// Avoid calling classloader leaking through Thread.inheritedAccessControlContext. +setContextClassLoader(callingThread, null); +try{ +final Thread t = threadFactory.newThread(taskRunner); + +// See https://github.com/netty/netty/issues/7290 and https://bugs.openjdk.org/browse/JDK-7008595 +setContextClassLoader(t, null); + +thread =t; + t. + +start(); +}finally{ + +setContextClassLoader(callingThread, parentCCL); +} +``` + +This addresses **edge B** (§2), not edge A. It stops a long-lived global thread from capturing whichever application loader happened to be +current when it started. It cannot help when the thread's own class is in the loader you want to unload — `primer/StartsOwnThreadNettyStyle` +applies the mitigation exactly and is still **PINNED**. + +Per `cclOnly/*`, the null-the-caller-then-restore dance is **not required** for edge B; nulling the new thread's context class loader after +construction is sufficient. + +## 7. Rejected alternatives + +### The two settings alone **[executed]** + +Adding `allowCoreThreadTimeOut(true)` and `setRemoveOnCancelPolicy(true)` to the existing pruner and changing nothing else does **nothing at +all**. `scheduleAtFixedRate` keeps a task in the `DelayedWorkQueue` permanently, so the worker always has something to wait for, `getTask` +never returns null and the keep-alive never expires. Measured with a period ten times the keep-alive: `poolSize=1, queue=1` while idle. +`removeOnCancelPolicy` is equally inert while nothing cancels. Somebody has to decide to stop; see §4.1. + +### A JVM shutdown hook **[executed]** + +Worse than useless. Four shapes measured: + +| Shape | Result | Can it stop the pruner? | +|---------------------------------------------------------------------------------------------|------------|-------------------------| +| Plain hook, child-loaded hook body | **PINNED** | yes | +| \+ context class loader nulled around creation | **PINNED** | yes | +| \+ nulled CCL *and* parent-loaded `Runnable`, `Thread` still constructed by the child class | **PINNED** | no | +| `Thread` **constructed** by a parent-loaded class | COLLECTED | **no** | + +Row 3 caught out a prediction: a parent-loaded `Runnable` and a nulled context class loader are still not enough, because edge A captures +the constructing class. Only building the `Thread` outside the loader removes the pin, and then the hook references nothing in the driver. +**The hook pins exactly to the extent that it can do its job.** + +Two further problems: + +- `ApplicationShutdownHooks` keeps hooks in a static map on a bootstrap-loaded class, so the hook thread, its + `Runnable`, that class and its loader are reachable until JVM exit. Registering a hook *creates* a pin where there was none. +- It fixes the wrong problem anyway. Hooks run at JVM exit, when the process and its memory are going away. This leak bites *before* exit — + undeploy, redeploy, OSGi refresh — while the JVM keeps running. The pruner is already a daemon thread, so it never delays exit. + +### Reference counting `MongoClient`s + +Community [PR 2032](https://github.com/mongodb/mongo-java-driver/pull/2032). Triage deferred it, and per §3 `DEFAULT` +has consumers with no `MongoClient` at all, so the ownership relation it needs does not exist. + +## 8. Recommendation + +1. ~~**`PowerOfTwoBufferPool`, on `main`, as JAVA-6279**~~ — §4. **DONE**, on branch `JAVA-6279`. One file plus its test, `driver-core` + only. +2. **`CommonExecutor`, on the backpressure branch, as JAVA-6240** — §5. Two lines plus deleting the `Cleaner` TODO. Cheap, no race, follows + precedent already in that PR, and stops a second pin landing. Narrower in effect than (1), but latent, so worth doing before merge. +3. **`DaemonThreadFactory` context class loader nulling** — §2 edge B. **Not applied**: tried, measured, then removed pending a decision. It + fixes a distinct, unfiled bug — the driver pinning an *application's* loader — but it needs a scoping decision first. Code on these + threads can no longer rely on a context class loader, `AsyncGetter` threads run application callbacks, and `ServiceLoader.load(Class)` + reads the context class loader implicitly. Netty's version is narrower and covers only its own global executor. If it lands it wants its + own ticket and its own commit, and it must not close this one. +4. **If JAVA-6279 stays blocked**, widening `disablePruning()`'s visibility costs nothing in public API terms (`PowerOfTwoBufferPool` is + already `com.mongodb.internal`) and removes the need for reflection in the issue-2029 workaround. A stopgap, not a fix — the default + still leaks. + +(1) and (2) are independent: disjoint files, no conflict. Backpressure does not touch `PowerOfTwoBufferPool` at all; the only shared file is +`DaemonThreadFactory`, where that PR adds `final` and tidies javadoc. + +**Whichever lands first, flip the harness's expectations as part of it**, or the suite goes red on success and gets ignored. + +## 9. Loose ends not pursued + +- In `driver/OPEN_AND_CLOSE_CLIENT_THEN_DISABLE_PRUNING` the thread listing taken immediately after `close()` still shows a `cluster-` + monitor thread, yet the loader is collected ~5 ms later. Either it terminated inside the GC window or it does not pin. Not investigated; + the verdict is the same either way. **[unmeasured]** +- The exact retaining field behind edge A. A JDK implementation detail, and not needed to act. **[unmeasured]** +- An adaptive prune delay — sleep until the oldest buffer becomes evictable rather than polling on a fixed cadence — would remove the + remaining wakeups. It needs the minimum `lastUsedNanos` across pools and changes eviction timing, so it does not belong in a bug fix. + **[unmeasured]** +- `SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder"` in the output is expected: the child loader gets the driver's compile + classpath, which has the SLF4J API but no binding. + +## How to run + +```bash +./testing/java-6279-poc/run.sh # everything +./testing/java-6279-poc/run.sh primer # synthetic class loader scenarios, no driver code +./testing/java-6279-poc/run.sh driver # the driver in a child class loader +./testing/java-6279-poc/run.sh executor # mechanism checks, under every JDK found +``` + +No MongoDB server is needed — the driver scenarios never issue an operation, so a `MongoClient` can be created and closed against an address +nothing is listening on. `run.sh` builds `:bson:jar :driver-core:jar :driver-sync:jar`, resolves third party jars from `driver-sync`'s own +runtime classpath so versions match the build, compiles the harness into a temporary directory and runs it. + +Options: + +| | | +|-----------------------------------------------|------------------------------------------------------------------------| +| `JAVA_HOMES=/a:/b` | override JDK discovery for the `executor` mode | +| `JAVA6279_DRIVER_REPO=/path` | run the `driver` scenarios against another checkout's jars | +| `POC_JAVA_ARGS=-Djava6279.gcWindowSeconds=45` | widen the GC window, e.g. past a fix's keep-alive | +| `POC_JAVA_ARGS=-Djava6279.breakRecheck=true` | negative control for the §4.3 race protocol; that check must then FAIL | + +To run the `driver` scenarios against the backpressure branch: + +```bash +git worktree add --detach /tmp/bp stIncMale/sleepAsync +(cd /tmp/bp && ./gradlew -q -PskipCryptVerify=true :bson:jar :driver-core:jar :driver-sync:jar) +JAVA6279_DRIVER_REPO=/tmp/bp ./testing/java-6279-poc/run.sh driver +``` + +`-PskipCryptVerify=true` is needed unless `gpg` is set up for the crypt library signature check. + +Exit status is 0 when every scenario matched the expectation in `Poc.java` and every gating check in +`ExecutorMechanism.java` passed. + +The `driver/*` expectations now describe **fixed** behaviour, so this suite is a regression check on JAVA-6279: if +`OPEN_AND_CLOSE_CLIENT` starts reporting PINNED again, the pruner has stopped terminating. The `primer/*` expectations describe properties +of the JVM rather than of the driver, and are not expected to change. diff --git a/testing/java-6279-poc/classpath.gradle b/testing/java-6279-poc/classpath.gradle new file mode 100644 index 00000000000..734900315c5 --- /dev/null +++ b/testing/java-6279-poc/classpath.gradle @@ -0,0 +1,12 @@ +// Gradle init script used by run.sh to discover module runtime classpaths without touching any +// build file. Applied with -I; adds no behaviour to a normal build. +allprojects { + tasks.register("printRuntimeCp") { + doLast { + def cp = project.configurations.findByName("runtimeClasspath") + if (cp != null) { + println "CP:" + cp.files.join(File.pathSeparator) + } + } + } +} diff --git a/testing/java-6279-poc/run.sh b/testing/java-6279-poc/run.sh new file mode 100755 index 00000000000..f5b15ac11c4 --- /dev/null +++ b/testing/java-6279-poc/run.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# Runs the JAVA-6279 proofs of concept. +# +# Usage: ./testing/java-6279-poc/run.sh [primer|driver|executor|all] +# +# primer synthetic classes in a throwaway child class loader -- reproduces the finding that a live thread +# started from a class's static initializer pins that class loader. No driver code. +# driver the driver loaded into a child class loader -- shows that BufferPoolPruner keeps that loader, and +# therefore every driver class and all its static state, strongly reachable after the client is closed. +# executor no driver code and no class loaders: checks that a self-terminating, self-resurrecting pruner is +# implementable on ScheduledThreadPoolExecutor without ever shutting it down. Run under every JDK found +# on the machine, because it leans on unspecified implementation behaviour and the driver's baseline is +# Java 8. Set JAVA_HOMES to a colon separated list to override JDK discovery. +# +# No MongoDB server is needed: the driver scenarios never issue an operation. +# Nothing under any module's src/main is modified. +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +POC_DIR="$REPO_ROOT/testing/java-6279-poc" +WHAT="${1:-all}" +cd "$REPO_ROOT" + +OUT="$(mktemp -d)" +trap 'rm -rf "$OUT"' EXIT +STATUS=0 + +# --------------------------------------------------------------------------------------------------------------------- +# executor mechanism: standalone, Java 8 clean, no driver jars needed +# --------------------------------------------------------------------------------------------------------------------- +if [ "$WHAT" = "executor" ] || [ "$WHAT" = "all" ]; then + echo "== compiling the executor mechanism checks at --release 8 ==" + mkdir -p "$OUT/mechanism" + javac -nowarn --release 8 -d "$OUT/mechanism" "$POC_DIR/src/java6279/ExecutorMechanism.java" + + # Discover JDKs. JAVA_HOMES wins; otherwise ask java_home on macOS; always include whatever java is on the PATH. + JDKS="" + if [ -n "${JAVA_HOMES:-}" ]; then + JDKS="$(echo "$JAVA_HOMES" | tr ':' '\n')" + elif [ -x /usr/libexec/java_home ]; then + JDKS="$(/usr/libexec/java_home -V 2>&1 | sed -n 's|.* \(/.*/Contents/Home\)$|\1|p' || true)" + fi + JDKS="$(printf '%s\n%s\n' "$JDKS" "$(dirname "$(dirname "$(command -v java)")")" | sed '/^$/d' | sort -u)" + + for JDK in $JDKS; do + [ -x "$JDK/bin/java" ] || continue + echo + echo "############ $("$JDK/bin/java" -version 2>&1 | head -1) ############" + "$JDK/bin/java" ${POC_JAVA_ARGS:-} -cp "$OUT/mechanism" java6279.ExecutorMechanism || STATUS=1 + done +fi + +# --------------------------------------------------------------------------------------------------------------------- +# class loader retention: needs the driver jars +# --------------------------------------------------------------------------------------------------------------------- +if [ "$WHAT" = "primer" ] || [ "$WHAT" = "driver" ] || [ "$WHAT" = "all" ]; then + need_jar() { + local pattern="$1" + local found + found="$(ls $pattern 2>/dev/null | grep -v -- '-sources' | grep -v -- '-javadoc' | head -1 || true)" + if [ -z "$found" ]; then + echo "missing jar: $pattern" >&2 + exit 1 + fi + echo "$found" + } + + echo + if [ -n "${JAVA6279_DRIVER_REPO:-}" ]; then + # Point the harness at another checkout -- e.g. the backpressure branch in a git worktree -- to see what that + # tree's threads do. The scenarios are branch agnostic; only the jars under test change. + echo "== using driver jars from $JAVA6279_DRIVER_REPO ==" + DRIVER_REPO="$JAVA6279_DRIVER_REPO" + else + echo "== building driver jars ==" + ./gradlew -q -PskipCryptVerify=true :bson:jar :driver-core:jar :driver-sync:jar + DRIVER_REPO="$REPO_ROOT" + fi + + # The classpath handed to the *child* loader. It must be self-contained down to the platform class loader, so it + # carries the third party jars as well, taken from driver-sync's own runtime classpath so versions match the build. + DRIVER_CP="$(need_jar "$DRIVER_REPO/bson/build/libs/bson-*.jar")" + DRIVER_CP="$DRIVER_CP:$(need_jar "$DRIVER_REPO/driver-core/build/libs/mongodb-driver-core-*.jar")" + DRIVER_CP="$DRIVER_CP:$(need_jar "$DRIVER_REPO/driver-sync/build/libs/mongodb-driver-sync-*.jar")" + + DEPS="$(./gradlew -q -I "$POC_DIR/classpath.gradle" :driver-sync:printRuntimeCp \ + | sed -n 's/^CP://p' | tr ':' '\n' | sort -u | grep -v -- '-SNAPSHOT.jar$' | paste -sd: -)" + if [ -z "$DEPS" ]; then + echo "could not resolve third party jars from Gradle" >&2 + exit 1 + fi + DRIVER_CP="$DRIVER_CP:$DEPS" + + echo "== compiling the class loader proof of concept ==" + find "$POC_DIR/src" -name '*.java' > "$OUT/sources.txt" + javac -nowarn -d "$OUT/classes" @"$OUT/sources.txt" + + # java6279.Poc goes on the application classpath; java6279.primer.* is also there, but the primer class loader + # defines it from $OUT/classes itself rather than delegating, which is what gives each scenario a fresh loader. + echo "== running ==" + SCENARIOS="$WHAT" + [ "$WHAT" = "all" ] && SCENARIOS="all" + java -cp "$OUT/classes" \ + -Djava6279.classesDir="$OUT/classes" \ + -Djava6279.driverCp="$DRIVER_CP" \ + ${POC_JAVA_ARGS:-} \ + java6279.Poc "$SCENARIOS" || STATUS=1 +fi + +exit $STATUS diff --git a/testing/java-6279-poc/src/java6279/ExecutorMechanism.java b/testing/java-6279-poc/src/java6279/ExecutorMechanism.java new file mode 100644 index 00000000000..7760c66e8e7 --- /dev/null +++ b/testing/java-6279-poc/src/java6279/ExecutorMechanism.java @@ -0,0 +1,760 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +/** + * Is a self-terminating, self-resurrecting {@code BufferPoolPruner} implementable without ever shutting the executor + * down? This decides the shape of the JAVA-6279 fix, so it is checked rather than assumed. + * + *

The alternative -- {@code shutdown()} when drained -- is terminal: a shut-down + * {@link ScheduledThreadPoolExecutor} rejects further submissions, so resurrection would mean building a new executor + * each cycle, which in turn means a non-final field and a lock guarding it. The recipe checked here avoids all of + * that:

+ * + *
+ *     ScheduledThreadPoolExecutor pruner = new ScheduledThreadPoolExecutor(1, factory);
+ *     pruner.setKeepAliveTime(keepAlive, unit);   // must be > 0
+ *     pruner.allowCoreThreadTimeOut(true);        // let the core worker die when idle
+ *     pruner.setRemoveOnCancelPolicy(true);       // so a cancelled periodic task leaves the queue empty
+ * 
+ * + *

{@code prune()} then cancels its own periodic future when it finds the pool drained, the worker times out and + * exits, and a later {@code scheduleAtFixedRate} on the same executor brings a worker back.

+ * + *

Deliberately Java 8 clean, with no dependency on the rest of this proof of concept, so that {@code run.sh} can + * compile it at {@code --release 8} and run it under every JDK on the machine. The driver's baseline is Java 8 and + * the behaviour being relied on is unspecified {@link ScheduledThreadPoolExecutor} implementation behaviour, not + * contract, so "it works on the developer's JDK" is not good enough.

+ */ +public final class ExecutorMechanism { + /** + * Big enough to be meaningful: the safe-publication hazard below shows up in roughly 0.5% of round trips, so a + * sample of 200 reports green about a third of the time. Sample sizes here are chosen against measured rates. + */ + private static final int ROUND_TRIPS = 2000; + + /** + * Set {@code -Djava6279.breakRecheck=true} to omit the check / re-check from + * {@link #conditionalRescheduleDesignHoldsUnderContention()}. That check must then FAIL; if it still passes, it is + * not sensitive enough to be evidence of anything. + */ + private static final boolean BREAK_RECHECK = Boolean.getBoolean("java6279.breakRecheck"); + + private ExecutorMechanism() { + } + + public static void main(final String... args) throws Exception { + System.out.printf("%s %s by %s%n", System.getProperty("java.vm.name"), System.getProperty("java.version"), + System.getProperty("java.vendor")); + List checks = new ArrayList(); + checks.add(settingsAloneDoNotReapTheWorker()); + checks.add(oneShotSchedulingSelfTerminatesWithNoDrainCheck()); + checks.add(pendingLongDelaySurvivesAShortKeepAlive()); + checks.add(repeatedResurrectionIsReliable()); + checks.add(concurrentSchedulingAcrossResurrection()); + checks.add(conditionalRescheduleDesignHoldsUnderContention()); + checks.add(reapsWorkerWhenPeriodicTaskCancelled()); + checks.add(staysReusableAndResurrects()); + checks.add(selfCancellationFromInsideTheTaskStopsTheRepeat()); + checks.add(safePublicationOfTheFutureIsRequired()); + checks.add(cancelledTaskLingersWithoutRemoveOnCancelPolicy()); + checks.add(generousKeepAliveAvoidsThreadChurn()); + report(checks); + } + + /** A pruner instrumented so the checks can see how many threads it ever created. */ + private static final class Pruner { + private final ScheduledThreadPoolExecutor executor; + private final AtomicInteger threadsCreated = new AtomicInteger(); + + Pruner(final long keepAlive, final TimeUnit unit, final boolean removeOnCancel) { + ThreadFactory factory = new ThreadFactory() { + @Override + public Thread newThread(final Runnable r) { + Thread thread = new Thread(r, "BufferPoolPruner-" + threadsCreated.incrementAndGet()); + thread.setDaemon(true); + return thread; + } + }; + executor = new ScheduledThreadPoolExecutor(1, factory); + executor.setKeepAliveTime(keepAlive, unit); + executor.allowCoreThreadTimeOut(true); + executor.setRemoveOnCancelPolicy(removeOnCancel); + } + + /** + * Schedules a periodic task that cancels itself on its first run, as a drained {@code prune()} would. + * + *

The {@code published} latch is not ceremony. {@code scheduleAtFixedRate} can start running the task + * before it returns the future, so a task that reads a field written *after* the call can see the unwritten + * value. See {@link #safePublicationOfTheFutureIsRequired()} — this cost an afternoon.

+ */ + ScheduledFuture scheduleSelfCancelling(final CountDownLatch ran) { + final AtomicReference> self = new AtomicReference>(); + final CountDownLatch published = new CountDownLatch(1); + ScheduledFuture future = executor.scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + try { + published.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + return; + } + self.get().cancel(false); + ran.countDown(); + } + }, 0, 5, TimeUnit.MILLISECONDS); + self.set(future); + published.countDown(); + return future; + } + + ScheduledFuture scheduleRepeating(final CountDownLatch ran) { + return executor.scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + ran.countDown(); + } + }, 0, 5, TimeUnit.MILLISECONDS); + } + + void shutdownNow() { + executor.shutdownNow(); + } + } + + // --------------------------------------------------------------------------------------------------------------- + // the checks + // --------------------------------------------------------------------------------------------------------------- + + /** + * The tempting one-line fix: add {@code allowCoreThreadTimeOut(true)} and {@code setRemoveOnCancelPolicy(true)} to + * the existing pruner and change nothing else. This checks whether that is sufficient. It is not. + * + *

{@code enablePruning()} uses {@code scheduleAtFixedRate}, so the periodic task sits in the + * {@code DelayedWorkQueue} permanently. The worker therefore always has something to wait for, {@code getTask} + * never returns null, and the keep-alive never expires — the settings are inert. {@code removeOnCancelPolicy} is + * likewise inert, because nothing ever cancels anything.

+ * + *

Note this is not specific to {@code scheduleAtFixedRate}: a self-rescheduling one-shot has the same property, + * since the next run is queued before the current one ends. The queue is only empty when pruning has genuinely + * stopped, which is the point — somebody has to decide to stop. That decision, the drained check, is the + * actual fix; these two settings are only what turns the decision into a dead thread.

+ */ + private static Check settingsAloneDoNotReapTheWorker() throws Exception { + Pruner pruner = new Pruner(100, TimeUnit.MILLISECONDS, true); + try { + final CountDownLatch ran = new CountDownLatch(1); + // A period long relative to the keep-alive, as the real one is: 1 minute idle time, 30 second period. + ScheduledFuture task = pruner.executor.scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + ran.countDown(); + } + }, 0, 2, TimeUnit.SECONDS); + if (!ran.await(5, TimeUnit.SECONDS)) { + return Check.fail("the two settings alone do not reap the worker", "the task never ran"); + } + // Idle for many multiples of the keep-alive, in the gap between two runs. + Thread.sleep(1000); + int poolSize = pruner.executor.getPoolSize(); + int queued = pruner.executor.getQueue().size(); + task.cancel(false); + int afterCancel = awaitPoolSize(pruner, 0); + // Passes by demonstrating that the settings are inert until something cancels. + return Check.of("the two settings alone do not reap the worker -- a decision to stop is required", + poolSize == 1 && queued == 1 && afterCancel == 0, + "idle 10x the keep-alive with the periodic task still scheduled: poolSize=" + poolSize + + ", queue=" + queued + " (thread alive, loader still pinned); " + + "poolSize=" + afterCancel + " only once the task stops being scheduled"); + } finally { + pruner.shutdownNow(); + } + } + + /** + * The distinction that decides how much work each fix is: one-shot scheduling needs no drain check at all. + * + *

{@code PowerOfTwoBufferPool.enablePruning()} uses {@code scheduleAtFixedRate}, so its queue is never empty and + * {@code allowCoreThreadTimeOut} can never fire — see {@link #settingsAloneDoNotReapTheWorker()}. But + * {@code CommonExecutor.schedule} uses one-shot {@code schedule(...)}, so once the scheduled task has run the queue + * really is empty, the worker times out on its own, and nothing has to decide to stop. For that shape the two + * settings ARE the whole fix, with none of the stop-versus-release race.

+ */ + private static Check oneShotSchedulingSelfTerminatesWithNoDrainCheck() throws Exception { + Pruner pruner = new Pruner(100, TimeUnit.MILLISECONDS, true); + try { + final CountDownLatch ran = new CountDownLatch(1); + pruner.executor.schedule(new Runnable() { + @Override + public void run() { + ran.countDown(); + } + }, 10, TimeUnit.MILLISECONDS); + if (!ran.await(5, TimeUnit.SECONDS)) { + return Check.fail("one-shot scheduling self-terminates with no drain check", "the task never ran"); + } + int afterRun = awaitPoolSize(pruner, 0); + // And it must still resurrect for the next scheduled task. + final CountDownLatch ranAgain = new CountDownLatch(1); + pruner.executor.schedule(new Runnable() { + @Override + public void run() { + ranAgain.countDown(); + } + }, 10, TimeUnit.MILLISECONDS); + boolean resurrected = ranAgain.await(5, TimeUnit.SECONDS); + int afterSecond = awaitPoolSize(pruner, 0); + return Check.of("one-shot scheduling self-terminates with no drain check (the CommonExecutor shape)", + afterRun == 0 && resurrected && afterSecond == 0 && pruner.threadsCreated.get() > 1, + "poolSize=" + afterRun + " after the one-shot ran, task ran again=" + resurrected + + ", poolSize=" + afterSecond + " after that, threads ever created=" + + pruner.threadsCreated.get()); + } finally { + pruner.shutdownNow(); + } + } + + /** + * The safety question for the {@code CommonExecutor} fix: with {@code allowCoreThreadTimeOut(true)} and a keep-alive + * much SHORTER than a pending one-shot delay, is that pending task still honoured, or can the worker time out and + * drop it? + * + *

This matters because {@code sleepAsync} delays are arbitrary — a retry backoff may be seconds while a sensible + * keep-alive is shorter. Losing a pending task would hang the callback, which is far worse than a leaked thread.

+ * + *

Safe by construction, per {@code ThreadPoolExecutor.processWorkerExit}: if the last worker exits while the + * queue is non-empty, a replacement is added. Checked anyway.

+ */ + private static Check pendingLongDelaySurvivesAShortKeepAlive() throws Exception { + Pruner pruner = new Pruner(100, TimeUnit.MILLISECONDS, true); + try { + long delayMillis = 3000; + final CountDownLatch ran = new CountDownLatch(1); + long scheduledAt = System.nanoTime(); + pruner.executor.schedule(new Runnable() { + @Override + public void run() { + ran.countDown(); + } + }, delayMillis, TimeUnit.MILLISECONDS); + boolean honoured = ran.await(delayMillis * 3, TimeUnit.MILLISECONDS); + long actualMillis = (System.nanoTime() - scheduledAt) / 1_000_000L; + int afterRun = awaitPoolSize(pruner, 0); + // Late is as bad as lost for a callback, so require it within a generous window of the requested delay. + boolean onTime = honoured && actualMillis < delayMillis * 2; + return Check.of("a pending long one-shot delay survives a much shorter keep-alive", + onTime && afterRun == 0, + "keep-alive=100ms, delay=" + delayMillis + "ms, ran=" + honoured + " after " + actualMillis + + "ms, poolSize=" + afterRun + " once it had run"); + } finally { + pruner.shutdownNow(); + } + } + + /** + * {@code CommonExecutor} is a singleton shared by every {@code MongoClient}, so with a short keep-alive its worker + * may be created and reaped over and over. Two questions: is that reliable, and what does it cost? + * + *

Each iteration schedules a one-shot, waits for it, then waits for the pool to drain to zero — so every + * iteration crosses the die/resurrect boundary deliberately, which is the worst case rather than the typical one.

+ */ + private static Check repeatedResurrectionIsReliable() throws Exception { + Pruner pruner = new Pruner(1, TimeUnit.MILLISECONDS, true); + try { + int iterations = 500; + int notRun = 0; + long startNanos = System.nanoTime(); + for (int i = 0; i < iterations; i++) { + final CountDownLatch ran = new CountDownLatch(1); + pruner.executor.schedule(new Runnable() { + @Override + public void run() { + ran.countDown(); + } + }, 0, TimeUnit.MILLISECONDS); + if (!ran.await(5, TimeUnit.SECONDS)) { + notRun++; + } + awaitPoolSize(pruner, 0); + } + long elapsedMicrosPerCycle = (System.nanoTime() - startNanos) / 1000L / iterations; + int threads = pruner.threadsCreated.get(); + return Check.of("repeated resurrection is reliable over " + iterations + " create/reap cycles", + notRun == 0 && threads > iterations / 2, + "tasks never run=" + notRun + ", threads ever created=" + threads + + " (churn really happened), ~" + elapsedMicrosPerCycle + "us per full cycle"); + } finally { + pruner.shutdownNow(); + } + } + + /** + * The multi-client case: several threads scheduling concurrently while the worker is dying. If resurrection lost a + * task here, a {@code sleepAsync} callback would never complete — a hang, not a leak. + */ + private static Check concurrentSchedulingAcrossResurrection() throws Exception { + final Pruner pruner = new Pruner(1, TimeUnit.MILLISECONDS, true); + try { + final int producers = 8; + final int perProducer = 250; + final CountDownLatch allRan = new CountDownLatch(producers * perProducer); + final CountDownLatch go = new CountDownLatch(1); + final AtomicInteger rejected = new AtomicInteger(); + Thread[] threads = new Thread[producers]; + for (int p = 0; p < producers; p++) { + threads[p] = new Thread(new Runnable() { + @Override + public void run() { + try { + go.await(); + for (int i = 0; i < perProducer; i++) { + try { + pruner.executor.schedule(new Runnable() { + @Override + public void run() { + allRan.countDown(); + } + }, 0, TimeUnit.MILLISECONDS); + } catch (java.util.concurrent.RejectedExecutionException e) { + rejected.incrementAndGet(); + allRan.countDown(); + } + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + }, "producer-" + p); + threads[p].start(); + } + go.countDown(); + boolean everythingRan = allRan.await(30, TimeUnit.SECONDS); + for (Thread t : threads) { + t.join(5000); + } + int afterwards = awaitPoolSize(pruner, 0); + return Check.of("concurrent scheduling across resurrection loses nothing (" + + producers + " threads x " + perProducer + ")", + everythingRan && rejected.get() == 0 && afterwards == 0, + "all tasks ran=" + everythingRan + ", rejections=" + rejected.get() + + ", outstanding=" + allRan.getCount() + ", poolSize afterwards=" + afterwards + + ", threads ever created=" + pruner.threadsCreated.get()); + } finally { + pruner.shutdownNow(); + } + } + + /** + * Prototypes the design this points to for {@code PowerOfTwoBufferPool}: replace {@code scheduleAtFixedRate} with a + * one-shot {@code schedule} that conditionally reschedules itself — next run only if the pool still holds + * something. Draining then makes the queue empty all by itself, so the worker times out; a later release schedules + * again. + * + *

Two advantages over cancelling one's own periodic future:

+ *
    + *
  • The task never needs a reference to its own {@code ScheduledFuture}, so the safe-publication hazard in + * {@link #safePublicationOfTheFutureIsRequired()} cannot arise at all.
  • + *
  • "Stop" becomes not doing something rather than an action, which is easier to reason about.
  • + *
+ * + *

The stop-versus-release race remains and still needs check / re-check. This models it with Netty's + * {@code GlobalEventExecutor} protocol and asserts the invariant that actually matters: once everything is + * quiescent, the pool must be empty (nothing was orphaned) and no thread may remain.

+ */ + private static Check conditionalRescheduleDesignHoldsUnderContention() throws Exception { + final Pruner pruner = new Pruner(50, TimeUnit.MILLISECONDS, true); + try { + int rounds = 20; + int orphanedRounds = 0; + int threadLeftRounds = 0; + for (int round = 0; round < rounds; round++) { + final java.util.concurrent.ConcurrentLinkedDeque pool = + new java.util.concurrent.ConcurrentLinkedDeque(); + final java.util.concurrent.atomic.AtomicBoolean pruningScheduled = + new java.util.concurrent.atomic.AtomicBoolean(); + final AtomicInteger pruneRuns = new AtomicInteger(); + final Runnable[] prune = new Runnable[1]; + prune[0] = new Runnable() { + @Override + public void run() { + pruneRuns.incrementAndGet(); + pool.clear(); // stands in for evicting idle buffers + if (pool.isEmpty()) { + // Widen the interleaving window deliberately. The orphan case needs a releaser to add an + // item AND fail its CAS in the gap between this emptiness check and the store below -- a + // window of nanoseconds in real code, which no realistic number of iterations would hit. + // Without this the negative control below passes and the whole check proves nothing. + for (int spin = 0; spin < 2000; spin++) { + Thread.yield(); + } + // Mark ourselves stopped, then RE-CHECK, exactly as Netty's TaskRunner does. + pruningScheduled.set(false); + if (BREAK_RECHECK) { + // Negative control: the naive "empty, so stop" with no re-check. Proves this check has + // teeth -- if omitting the protocol still passed, a PASS above would mean nothing. + return; + } + if (pool.isEmpty()) { + return; // safe to stop: nothing left + } + if (!pruningScheduled.compareAndSet(false, true)) { + return; // a releaser scheduled a run; it owns it now + } + } + pruner.executor.schedule(prune[0], 1, TimeUnit.MILLISECONDS); + } + }; + + // Releasers must keep going long enough to overlap the pruner's STOP path, otherwise the interleaving + // never occurs and the negative control passes. A fixed iteration count finishes in microseconds while + // the first prune is still 1ms away, which is exactly the mistake this replaced. + final int producers = 6; + final long releaseForNanos = TimeUnit.MILLISECONDS.toNanos(400); + final CountDownLatch go = new CountDownLatch(1); + Thread[] threads = new Thread[producers]; + for (int p = 0; p < producers; p++) { + threads[p] = new Thread(new Runnable() { + @Override + public void run() { + try { + go.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + return; + } + long until = System.nanoTime() + releaseForNanos; + while (System.nanoTime() < until) { + pool.addLast(new Object()); // "release(buffer)" + if (pruningScheduled.compareAndSet(false, true)) { + pruner.executor.schedule(prune[0], 1, TimeUnit.MILLISECONDS); + } + Thread.yield(); + } + } + }, "releaser-" + p); + threads[p].start(); + } + go.countDown(); + for (Thread t : threads) { + t.join(10_000); + } + // Let things settle: any pruner still scheduled must get a chance to drain the pool. + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (System.nanoTime() < deadline && !(pool.isEmpty() && pruner.executor.getPoolSize() == 0)) { + Thread.sleep(10); + } + if (!pool.isEmpty()) { + orphanedRounds++; // buffers left with no pruner coming -- the failure that matters + } + if (pruner.executor.getPoolSize() != 0) { + threadLeftRounds++; // thread never died -- the class loader would stay pinned + } + } + return Check.of("the conditional-reschedule design holds under contention (" + rounds + + " rounds x 6 releasers releasing for 400ms)", + orphanedRounds == 0 && threadLeftRounds == 0, + "rounds leaving an orphaned non-empty pool=" + orphanedRounds + + ", rounds leaving a live thread=" + threadLeftRounds); + } finally { + pruner.shutdownNow(); + } + } + + /** The load-bearing one: does the core worker actually die, leaving nothing to pin the class loader? */ + private static Check reapsWorkerWhenPeriodicTaskCancelled() throws Exception { + Pruner pruner = new Pruner(1, TimeUnit.MILLISECONDS, true); + try { + CountDownLatch ran = new CountDownLatch(1); + ScheduledFuture task = pruner.scheduleRepeating(ran); + if (!ran.await(5, TimeUnit.SECONDS)) { + return Check.fail("reaps the worker when the periodic task is cancelled", "the task never ran"); + } + int poolSizeWhileScheduled = pruner.executor.getPoolSize(); + task.cancel(false); + int poolSize = awaitPoolSize(pruner, 0); + return Check.of("reaps the worker when the periodic task is cancelled", + poolSizeWhileScheduled == 1 && poolSize == 0, + "poolSize " + poolSizeWhileScheduled + " while scheduled, " + poolSize + " after cancel"); + } finally { + pruner.shutdownNow(); + } + } + + /** Resurrection has to work on the same executor, or the fix needs a mutable field and a lock. */ + private static Check staysReusableAndResurrects() throws Exception { + Pruner pruner = new Pruner(1, TimeUnit.MILLISECONDS, true); + try { + CountDownLatch firstRan = new CountDownLatch(1); + ScheduledFuture first = pruner.scheduleRepeating(firstRan); + boolean firstOk = firstRan.await(5, TimeUnit.SECONDS); + first.cancel(false); + awaitPoolSize(pruner, 0); + boolean neverShutDown = !pruner.executor.isShutdown(); + + CountDownLatch secondRan = new CountDownLatch(1); + ScheduledFuture second = pruner.scheduleRepeating(secondRan); + boolean secondOk = secondRan.await(5, TimeUnit.SECONDS); + second.cancel(false); + int threads = pruner.threadsCreated.get(); + return Check.of("stays reusable and resurrects a worker on re-scheduling", + firstOk && secondOk && neverShutDown && threads > 1, + "isShutdown=" + pruner.executor.isShutdown() + ", threads ever created=" + threads + + " (>1 proves the first worker died rather than lingering)"); + } finally { + pruner.shutdownNow(); + } + } + + /** + * {@code prune()} has no handle on its own future unless one is stashed for it, and cancelling a periodic task + * from inside its own run is exactly the case the JDK documents least clearly. Round-tripped to probe the + * resurrect-versus-die race in {@code ThreadPoolExecutor.processWorkerExit}. + */ + private static Check selfCancellationFromInsideTheTaskStopsTheRepeat() throws Exception { + Pruner pruner = new Pruner(1, TimeUnit.MILLISECONDS, true); + try { + int neverRan = 0; + int repeatNotStopped = 0; + for (int i = 0; i < ROUND_TRIPS; i++) { + CountDownLatch ran = new CountDownLatch(1); + ScheduledFuture task = pruner.scheduleSelfCancelling(ran); + if (!ran.await(5, TimeUnit.SECONDS)) { + neverRan++; // a lost pruner: buffers would sit in the pool with nobody to prune them + continue; + } + Thread.sleep(5); // give the repeat a chance to misfire + if (!task.isCancelled()) { + repeatNotStopped++; + } + } + int threads = pruner.threadsCreated.get(); + return Check.of("self-cancellation from inside the task stops the repeat, " + ROUND_TRIPS + " round trips", + neverRan == 0 && repeatNotStopped == 0 && threads > 1, + "lost pruners=" + neverRan + ", repeats not stopped=" + repeatNotStopped + + ", threads ever created=" + threads); + } finally { + pruner.shutdownNow(); + } + } + + /** + * A hazard the fix itself has to handle, found the hard way while writing these checks. + * + *

If {@code prune()} is to cancel its own periodic future, it needs a reference to that future — but + * {@code scheduleAtFixedRate} can begin running the task before it returns, so a field assigned from its return + * value is not safely visible to the task. The task then reads null, dies with a {@link NullPointerException}, + * and the executor cancels the repeat. Worse, a {@code ScheduledFuture} swallows the throwable: nothing is logged + * and nothing throws where it would be noticed. The observable result is a pool with buffers in it and no pruner + * — a lost pruner, which is the exact failure mode the fix must not have.

+ * + *

Measured at roughly 0.5% of round trips on an otherwise idle machine, which is low enough that a small test + * sample will happily report green.

+ * + *

So the fix must publish the future under whatever lock already guards start/stop, not simply assign it after + * scheduling.

+ */ + private static Check safePublicationOfTheFutureIsRequired() throws Exception { + Pruner pruner = new Pruner(1, TimeUnit.MILLISECONDS, true); + try { + int taskThrew = 0; + for (int i = 0; i < ROUND_TRIPS; i++) { + final CountDownLatch ran = new CountDownLatch(1); + final AtomicReference> self = new AtomicReference>(); + final AtomicReference thrown = new AtomicReference(); + // Deliberately UNSAFE: no publication barrier, exactly the naive implementation. + ScheduledFuture task = pruner.executor.scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + try { + self.get().cancel(false); + } catch (Throwable t) { + thrown.set(t); + } finally { + ran.countDown(); + } + } + }, 0, 5, TimeUnit.MILLISECONDS); + self.set(task); + ran.await(2, TimeUnit.SECONDS); + if (thrown.get() != null) { + taskThrew++; + } + task.cancel(false); + } + // Informational, never a failure. Asserting that a race *does* reproduce is a flaky test by construction: + // the rate varies by JDK and by machine load, and a run that happens to observe 0 has not disproved + // anything. The requirement stands on the reasoning plus the runs that did observe it; this number is here + // to show the hazard is not hypothetical and to give a sense of how easily a small sample misses it. + return Check.informational( + "the future needs safe publication, or prune() reads null and the repeat dies silently", + taskThrew + "/" + ROUND_TRIPS + " round trips threw NullPointerException inside the task, " + + "swallowed by the ScheduledFuture" + + (taskThrew == 0 ? " (not observed on this run -- the hazard is still real, see FINDINGS)" + : "")); + } finally { + pruner.shutdownNow(); + } + } + + /** + * Why {@code setRemoveOnCancelPolicy(true)} is part of the recipe and not a nicety: without it a cancelled + * periodic task stays in the {@code DelayedWorkQueue} until its delay elapses, so the queue is not empty, so the + * worker has something to wait on and does not time out. With a realistic one minute period that is a thread + * loitering for up to a minute past the drain -- which is most of what the ticket is trying to avoid. + */ + private static Check cancelledTaskLingersWithoutRemoveOnCancelPolicy() throws Exception { + Pruner pruner = new Pruner(1, TimeUnit.MILLISECONDS, false); + try { + CountDownLatch ran = new CountDownLatch(1); + // A long period, so that "still queued" and "already elapsed" are distinguishable. + ScheduledFuture task = pruner.executor.scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + ran.countDown(); + } + }, 0, 10, TimeUnit.SECONDS); + if (!ran.await(5, TimeUnit.SECONDS)) { + return Check.fail("cancelled task lingers in the queue without removeOnCancelPolicy", + "the task never ran"); + } + task.cancel(false); + Thread.sleep(300); + int queued = pruner.executor.getQueue().size(); + int poolSize = pruner.executor.getPoolSize(); + // Informational, for the same reason as safePublicationOfTheFutureIsRequired: this observes + // implementation behaviour that varies by JDK. JDK 8 has been seen to reach queue=0, poolSize=0 here + // anyway. The recommendation to set the policy does not rest on this measurement -- it rests on the + // deterministic checks above, all of which pass WITH the policy set on every JDK, and on the policy being + // free. MongoScheduledThreadPoolExecutor already sets it. + return Check.informational("cancelled task retained in the queue without removeOnCancelPolicy", + "queue=" + queued + ", poolSize=" + poolSize + + (queued > 0 || poolSize > 0 + ? " -- the worker is still there, hence setRemoveOnCancelPolicy(true)" + : " -- this JDK dropped it anyway; the policy is still recommended, see FINDINGS")); + } finally { + pruner.shutdownNow(); + } + } + + /** + * The keep-alive is a thread churn dial, not a correctness one. A 1 ms keep-alive creates one thread per + * schedule/cancel cycle; a generous one lets a busy application reuse a single worker while a quiet one still + * eventually drops to zero. Set it to something on the order of {@code maxIdleTime}. + */ + private static Check generousKeepAliveAvoidsThreadChurn() throws Exception { + Pruner churny = new Pruner(1, TimeUnit.MILLISECONDS, true); + Pruner calm = new Pruner(2, TimeUnit.SECONDS, true); + try { + int cycles = 25; + for (int i = 0; i < cycles; i++) { + roundTrip(churny); + roundTrip(calm); + } + int churnyThreads = churny.threadsCreated.get(); + int calmThreads = calm.threadsCreated.get(); + return Check.of("a generous keep-alive avoids thread churn over " + cycles + " cycles", + calmThreads < churnyThreads, + "1ms keep-alive created " + churnyThreads + " threads, 2s keep-alive created " + calmThreads); + } finally { + churny.shutdownNow(); + calm.shutdownNow(); + } + } + + // --------------------------------------------------------------------------------------------------------------- + // helpers + // --------------------------------------------------------------------------------------------------------------- + + private static void roundTrip(final Pruner pruner) throws Exception { + CountDownLatch ran = new CountDownLatch(1); + ScheduledFuture task = pruner.scheduleSelfCancelling(ran); + ran.await(5, TimeUnit.SECONDS); + task.cancel(false); + Thread.sleep(5); + } + + private static int awaitPoolSize(final Pruner pruner, final int target) throws InterruptedException { + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (System.nanoTime() < deadline) { + if (pruner.executor.getPoolSize() == target) { + return target; + } + Thread.sleep(10); + } + return pruner.executor.getPoolSize(); + } + + private static final class Check { + private final String name; + private final boolean passed; + private final boolean informational; + private final String detail; + + private Check(final String name, final boolean passed, final boolean informational, final String detail) { + this.name = name; + this.passed = passed; + this.informational = informational; + this.detail = detail; + } + + static Check of(final String name, final boolean passed, final String detail) { + return new Check(name, passed, false, detail); + } + + static Check fail(final String name, final String detail) { + return new Check(name, false, false, detail); + } + + /** Reports a measurement without gating the exit status. For hazards whose reproduction rate is stochastic. */ + static Check informational(final String name, final String detail) { + return new Check(name, true, true, detail); + } + + String label() { + return informational ? "INFO" : passed ? "PASS" : "FAIL"; + } + } + + private static void report(final List checks) { + boolean allPassed = true; + System.out.println(); + for (Check check : checks) { + allPassed &= check.passed; + System.out.printf(" %-4s %s%n", check.label(), check.name); + System.out.printf(" %s%n", check.detail); + } + System.out.println(); + System.out.println(allPassed + ? "the self-terminating, self-resurrecting pruner is implementable on this JDK" + : "AT LEAST ONE CHECK FAILED ON THIS JDK -- the mechanism cannot be relied on"); + if (!allPassed) { + System.exit(1); + } + } +} diff --git a/testing/java-6279-poc/src/java6279/Poc.java b/testing/java-6279-poc/src/java6279/Poc.java new file mode 100644 index 00000000000..f8d8b17c466 --- /dev/null +++ b/testing/java-6279-poc/src/java6279/Poc.java @@ -0,0 +1,666 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279; + +import java.io.File; +import java.lang.ref.Cleaner; +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.net.URLClassLoader; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Set; +import java.util.TreeSet; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * JAVA-6279 proof of concept: does a non-terminated driver thread pin the class loader that loaded the driver? + * + *

This is a portable rework of the experiment in + * + * Valentin Kovalenko's {@code primer} commit. That version hardcoded an absolute path to one developer's + * {@code build/classes} directory; here the class file directory and the driver classpath are supplied as system + * properties by {@code run.sh}, so it runs anywhere.

+ * + *

Two groups of scenarios:

+ *
    + *
  • {@code primer} — synthetic classes in a throwaway child class loader, reproducing the original finding: a + * live thread started from a class's static initializer keeps that class and its whole class loader strongly + * reachable, even though the thread's {@code Runnable} is defined by a parent-loaded class and so + * references nothing in the child loader.
  • + *
  • {@code driver} — the real case behind JAVA-6279: load the driver into a child class loader, touch (or open + * and close) it, then check whether that loader can be collected while {@code BufferPoolPruner} is alive.
  • + *
+ * + *

Every scenario reports {@code COLLECTED} or {@code PINNED}; nothing is asserted, because "the GC did not get + * around to it" and "something holds a strong reference" are not distinguishable in the general case. The bounded + * wait plus the explicit {@link System#gc()} nudges make {@code PINNED} strong evidence in practice, and the control + * scenarios establish that the harness can observe a collection at all.

+ * + *

Not wired into Gradle: it needs {@link System#gc()}, custom class loaders and multi-second GC windows, none of + * which belong in the normal test suite.

+ */ +public final class Poc { + /** + * How long the threads started by the primer classes sleep. Must comfortably outlive {@link #GC_WINDOW} so that + * a {@code PINNED} verdict is attributable to a live thread rather than to one that already finished. + */ + private static final Duration THREAD_LIFETIME = Duration.ofSeconds(30); + + /** + * How long we nudge the GC before declaring a referent unreachable-or-not. Override with + * {@code -Djava6279.gcWindowSeconds=...} when a fix under test terminates its thread on a timer longer than the + * default — e.g. verifying a {@code CommonExecutor} keep-alive of 30s needs a window comfortably beyond it. + */ + private static final Duration GC_WINDOW = + Duration.ofSeconds(Long.getLong("java6279.gcWindowSeconds", 10L)); + + /** + * The driver scenarios need a longer window than the primer ones. With JAVA-6279 fixed the pruner stops on a timer + * derived from {@code maxIdleTime} (one minute by default) and the thread then times out after the keep-alive, so + * the loader is released roughly 90s after the last buffer release rather than immediately. Override with + * {@code -Djava6279.driverGcWindowSeconds=...}. + */ + private static final Duration DRIVER_GC_WINDOW = + Duration.ofSeconds(Long.getLong("java6279.driverGcWindowSeconds", 150L)); + + private static final String PRIMER_PACKAGE = "java6279.primer."; + + /** + * The body of every thread the primer classes start. Deliberately defined here, in a class the child + * loader delegates to its parent, so the running thread's {@code Runnable} has no reference of any kind into the + * child loader. That is the whole point of the experiment. + */ + public static final Runnable SLEEPING_RUNNABLE = new Runnable() { + @Override + public void run() { + try { + log("thread %s is sleeping for %s", Thread.currentThread().getName(), THREAD_LIFETIME); + Thread.sleep(THREAD_LIFETIME.toMillis()); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } finally { + log("thread %s terminated", Thread.currentThread().getName()); + } + } + }; + + /** + * A hook thread both constructed and bodied by this parent-loaded class, before any child loader exists. The + * child class only calls {@code addShutdownHook} on it. This is the only shutdown-hook shape that does not pin — + * and, being unable to reference driver code, the only one that cannot stop the pruner. + */ + public static final Thread PARENT_BUILT_HOOK_THREAD = new Thread(new Runnable() { + @Override + public void run() { + log("the fully parent-built shutdown hook ran"); + } + }, "java6279-shutdown-hook-parent-built"); + + private Poc() { + } + + public static void main(final String... args) throws Exception { + log("%s %s by %s", System.getProperty("java.vm.name"), System.getProperty("java.version"), + System.getProperty("java.vendor")); + String what = args.length > 0 ? args[0] : "all"; + List results = new ArrayList<>(); + if (what.equals("primer") || what.equals("all")) { + results.addAll(primerScenarios()); + } + if (what.equals("driver") || what.equals("all")) { + results.addAll(driverScenarios()); + } + report(results); + } + + // --------------------------------------------------------------------------------------------------------------- + // primer scenarios + // --------------------------------------------------------------------------------------------------------------- + + private static List primerScenarios() throws Exception { + Path classesDir = Paths.get(requireProperty("java6279.classesDir")); + List results = new ArrayList<>(); + results.add(primerScenario(classesDir, "Inert", + "control: a child-loaded class that starts no thread", Expectation.COLLECTED)); + results.add(primerScenario(classesDir, "StartsOwnThread", + "a child-loaded class that constructs and starts a thread in its static initializer", + Expectation.PINNED)); + results.add(primerScenario(classesDir, "StartsOwnThreadNettyStyle", + "as above, but with Netty's GlobalEventExecutor mitigation: null the context class loader around " + + "thread creation (netty#7290, JDK-7008595)", + Expectation.PINNED)); + results.add(primerScenario(classesDir, "StartsParentBuiltThread", + "a child-loaded class that starts a Thread object constructed by a parent-loaded class", + Expectation.UNKNOWN)); + results.add(primerScenario(classesDir, "InheritsContextClassLoader", + "a parent-built thread that INHERITS the child loader as its context class loader -- the leak edge " + + "Netty's mitigation targets, i.e. a driver thread pinning an application's loader", + Expectation.PINNED)); + results.add(primerScenario(classesDir, "InheritsContextClassLoaderButNulled", + "the same, context class loader nulled after construction. CONFOUNDED: this class's is on " + + "the stack, and the stack capture pins regardless -- see cclOnly/* for the clean isolation", + Expectation.PINNED)); + results.add(primerScenario(classesDir, "InheritsContextClassLoaderNettyDance", + "the same, nulling the CALLING thread's context class loader before construction as Netty does. " + + "Also CONFOUNDED by the stack frame: the stack capture cannot be nulled away", + Expectation.PINNED)); + // Isolate the context class loader edge with NO child-loaded class on the stack at construction time. The + // scenarios above cannot do this: a primer class's static initializer is necessarily on the stack, and the + // stack capture dominates, masking whatever the context class loader does. + results.add(contextClassLoaderOnlyScenario(classesDir, false)); + results.add(contextClassLoaderOnlyScenario(classesDir, true)); + results.add(primerScenario(classesDir, "RegistersShutdownHook", + "the shutdown-hook alternative: register a hook to stop the pruner, start no thread. " + + "ApplicationShutdownHooks holds hooks in a static map until JVM exit", + Expectation.PINNED)); + results.add(primerScenario(classesDir, "RegistersShutdownHookNettyStyle", + "shutdown hook + Netty's context class loader nulling, hook body still child-loaded", + Expectation.PINNED)); + results.add(primerScenario(classesDir, "RegistersShutdownHookParentBody", + "shutdown hook + nulled context class loader + parent-loaded hook body -- collectable, but the hook " + + "cannot reference driver code, so it cannot stop the pruner", + Expectation.COLLECTED)); + results.add(primerScenario(classesDir, "StaticSingletonExecutor", + "models CommonExecutor: a static singleton whose executor is never shut down, with the Cleaner its " + + "VAKOTODO proposes -- see whether the cleaning action can ever run", + Expectation.PINNED)); + return results; + } + + /** + * The clean isolation of the context class loader edge. Everything happens in this parent-loaded class: a child + * class is initialized and returned from, then the thread is constructed with only {@code Poc} frames on the stack + * while the calling thread's context class loader is the child loader. So the only possible edge into that loader + * is the context class loader, and {@code nullContextClassLoader} decides whether it exists. + * + *

This is Netty's configuration, and the one the {@code DaemonThreadFactory} change is aimed at: a driver thread + * created on behalf of an application, pinning the application's loader.

+ */ + private static Result contextClassLoaderOnlyScenario(final Path classesDir, final boolean nullContextClassLoader) + throws Exception { + String name = "cclOnly/" + (nullContextClassLoader ? "nulled" : "inherited"); + banner(name); + PhantomReachableWatch watch = loadAndForgetCclOnly(classesDir, nullContextClassLoader); + boolean collected = watch.awaitCollected(GC_WINDOW); + return new Result(name, + nullContextClassLoader + ? "thread constructed with NO child frame on the stack and the context class loader nulled" + : "thread constructed with NO child frame on the stack, inheriting the child loader as its " + + "context class loader", + nullContextClassLoader ? Expectation.COLLECTED : Expectation.PINNED, collected, watch.elapsed()); + } + + private static PhantomReachableWatch loadAndForgetCclOnly(final Path classesDir, + final boolean nullContextClassLoader) throws Exception { + ClassLoader loader = new PrimerClassLoader(classesDir); + Class.forName(PRIMER_PACKAGE + "Inert", true, loader); + Thread callingThread = Thread.currentThread(); + ClassLoader previous = callingThread.getContextClassLoader(); + callingThread.setContextClassLoader(loader); + try { + Thread thread = new Thread(SLEEPING_RUNNABLE, + "java6279-cclOnly-" + (nullContextClassLoader ? "nulled" : "inherited")); + log("built %s with inherited context class loader %s", thread.getName(), thread.getContextClassLoader()); + if (nullContextClassLoader) { + thread.setContextClassLoader(null); + } + thread.start(); + } finally { + callingThread.setContextClassLoader(previous); + } + return new PhantomReachableWatch(loader, loader.toString()); + } + + /** + * Loads {@code java6279.primer.} in a fresh child loader, forgets every strong reference to the + * loader and to the classes it defined, and reports whether the loader became phantom reachable. + */ + private static Result primerScenario(final Path classesDir, final String simpleName, final String description, + final Expectation expectation) throws Exception { + banner("primer/" + simpleName); + // Loading the trigger class and dropping the references happens in a separate frame so that no local variable + // in this frame keeps the loader alive while we wait for the GC. + CLEANER_RAN.set(false); + PhantomReachableWatch watch = loadAndForgetPrimer(classesDir, simpleName); + boolean collected = watch.awaitCollected(GC_WINDOW); + if (simpleName.equals("StaticSingletonExecutor")) { + log("cleaning action ran: %s (the CommonExecutor VAKOTODO proposes relying on this)", CLEANER_RAN.get()); + } + return new Result("primer/" + simpleName, description, expectation, collected, watch.elapsed()); + } + + private static PhantomReachableWatch loadAndForgetPrimer(final Path classesDir, final String simpleName) + throws Exception { + ClassLoader loader = new PrimerClassLoader(classesDir); + // Initialize the trigger class, then also load an inert sibling. The sibling is what the original experiment + // used to show the pinning is loader-wide and not specific to the class that started the thread. + Class.forName(PRIMER_PACKAGE + simpleName, true, loader); + Class.forName(PRIMER_PACKAGE + "Inert", true, loader); + return new PhantomReachableWatch(loader, loader.toString()); + } + + /** + * A {@code Thread} object constructed while this class is initialized -- that is, before any primer class loader + * exists. {@code java6279.primer.StartsParentBuiltThread} only calls {@code start()} on it. Being a field rather + * than the result of a factory method matters: if the object were constructed on demand, a child-loaded class + * would be on the stack at construction time and the distinction the scenario is drawing would be lost. + * + *

{@code App.THREAD} in the original experiment.

+ */ + public static final Thread PARENT_BUILT_THREAD = new Thread(SLEEPING_RUNNABLE, "java6279-parent-built"); + + /** + * Builds and starts a thread from this parent-loaded class, so there is no construction-site pin, and the + * only possible edge into a child loader is the inherited context class loader. With {@code nullContextClassLoader} + * this applies the mitigation added to {@code DaemonThreadFactory}; without it, the thread keeps whatever the + * calling thread's context class loader was. + */ + public static void buildAndStartThreadInheritingCcl(final boolean nullContextClassLoader) { + Thread thread = new Thread(SLEEPING_RUNNABLE, + "java6279-ccl-" + (nullContextClassLoader ? "nulled" : "inherited")); + log("built %s with inherited context class loader %s", thread.getName(), thread.getContextClassLoader()); + if (nullContextClassLoader) { + thread.setContextClassLoader(null); + } + thread.start(); + } + + /** + * Netty's full mitigation: null the calling thread's context class loader BEFORE constructing, then + * restore it. The distinction from {@link #buildAndStartThreadInheritingCcl} matters — construction captures the + * creating thread's context, so nulling the new thread's field afterwards is too late. + */ + public static void buildAndStartThreadNettyDance() { + Thread callingThread = Thread.currentThread(); + ClassLoader parentCcl = callingThread.getContextClassLoader(); + callingThread.setContextClassLoader(null); + try { + Thread thread = new Thread(SLEEPING_RUNNABLE, "java6279-ccl-netty-dance"); + thread.setContextClassLoader(null); + log("built %s with context class loader %s", thread.getName(), thread.getContextClassLoader()); + thread.start(); + } finally { + callingThread.setContextClassLoader(parentCcl); + } + } + + public static void log(final String format, final Object... args) { + System.err.printf("[java6279] " + format + "%n", args); + } + + /** + * Registers a cleaning action, as {@code CommonExecutor}'s {@code VAKOTODO} proposes doing to shut its executor + * down. The {@link Cleaner} lives here, in a parent-loaded class, so that its own thread is not itself a reason + * for a child loader to be pinned. + */ + public static void registerCleaner(final Object referent, final Runnable action) { + CLEANER.register(referent, () -> { + CLEANER_RAN.set(true); + action.run(); + }); + } + + private static final Cleaner CLEANER = Cleaner.create(); + + /** Whether any cleaning action registered via {@link #registerCleaner} has run. */ + private static final AtomicBoolean CLEANER_RAN = new AtomicBoolean(); + + // --------------------------------------------------------------------------------------------------------------- + // driver scenarios + // --------------------------------------------------------------------------------------------------------------- + + private static List driverScenarios() throws Exception { + URL[] driverClasspath = driverClasspath(); + List results = new ArrayList<>(); + results.add(driverScenario(driverClasspath, DriverAction.LOAD_ONLY, + "control: driver classes loaded but PowerOfTwoBufferPool.DEFAULT never initialized", + Expectation.COLLECTED)); + results.add(driverScenario(driverClasspath, DriverAction.TOUCH_DEFAULT_POOL, + "PowerOfTwoBufferPool.DEFAULT initialized. Before JAVA-6279 this alone started the pruner and " + + "pinned the loader; an empty pool must now start no thread", Expectation.COLLECTED)); + results.add(driverScenario(driverClasspath, DriverAction.TOUCH_DEFAULT_POOL_THEN_DISABLE_PRUNING, + "the same, then disablePruning() called reflectively -- the workaround from GitHub issue 2029", + Expectation.COLLECTED)); + results.add(driverScenario(driverClasspath, DriverAction.OPEN_AND_CLOSE_CLIENT, + "MongoClients.create(...) followed by close() -- the symptom as reported. With JAVA-6279 fixed the " + + "pruner drains the pool, stops, and its thread times out, so the loader is released", + Expectation.COLLECTED)); + results.add(driverScenario(driverClasspath, DriverAction.OPEN_AND_CLOSE_CLIENT_THEN_DISABLE_PRUNING, + "the same, plus disablePruning() -- shows the pruner is the only remaining pin after close()", + Expectation.COLLECTED)); + results.add(driverScenario(driverClasspath, + DriverAction.OPEN_AND_CLOSE_CLIENT_THEN_DISABLE_PRUNING_AND_TOUCH_COMMON_EXECUTOR, + "the same, plus starting CommonExecutor's thread as an async retry backoff would -- backpressure " + + "branch only, isolates CommonExecutor as a second independent pin", + Expectation.UNKNOWN)); + return results; + } + + private static Result driverScenario(final URL[] driverClasspath, final DriverAction action, + final String description, final Expectation expectation) throws Exception { + banner("driver/" + action); + PhantomReachableWatch watch = loadAndForgetDriver(driverClasspath, action); + boolean collected = watch.awaitCollected(DRIVER_GC_WINDOW); + if (!collected) { + log("live non-JVM threads after the GC window: %s", nonJvmThreadNames()); + } + if (action.skipped) { + return new Result("driver/" + action, "SKIPPED (not applicable to this branch) -- " + description, + Expectation.UNKNOWN, collected, watch.elapsed()); + } + return new Result("driver/" + action, description, expectation, collected, watch.elapsed()); + } + + private static PhantomReachableWatch loadAndForgetDriver(final URL[] driverClasspath, final DriverAction action) + throws Exception { + // Parent is the platform class loader, not the application one, so the driver classes on this classpath are + // genuinely defined by the child loader -- exactly the situation of an application server or OSGi container + // loading the driver as part of a redeployable unit. + URLClassLoader loader = new URLClassLoader("java6279-driver", driverClasspath, + ClassLoader.getPlatformClassLoader()); + action.run(loader); + // The loader is deliberately *not* closed. close() only releases jar file handles, it does not affect + // reachability, and doing it here would make any still-running driver thread fail with NoClassDefFoundError + // and cloud the result. + return new PhantomReachableWatch(loader, loader.toString()); + } + + private enum DriverAction { + LOAD_ONLY { + @Override + void run(final ClassLoader loader) throws Exception { + Class.forName("com.mongodb.internal.connection.PowerOfTwoBufferPool", false, loader); + log("loaded PowerOfTwoBufferPool without initializing it"); + } + }, + TOUCH_DEFAULT_POOL { + @Override + void run(final ClassLoader loader) throws Exception { + defaultPool(loader); + } + }, + TOUCH_DEFAULT_POOL_THEN_DISABLE_PRUNING { + @Override + void run(final ClassLoader loader) throws Exception { + disablePruning(defaultPool(loader)); + } + }, + OPEN_AND_CLOSE_CLIENT { + @Override + void run(final ClassLoader loader) throws Exception { + openAndCloseClient(loader); + } + }, + OPEN_AND_CLOSE_CLIENT_THEN_DISABLE_PRUNING { + @Override + void run(final ClassLoader loader) throws Exception { + openAndCloseClient(loader); + disablePruning(defaultPool(loader)); + } + }, + + /** + * Only meaningful on the backpressure branch, where {@code CommonExecutor} exists. Starts its + * {@code CommonScheduler} thread the way an async retry backoff would, then closes the client and disables + * pruning -- so a PINNED result isolates {@code CommonExecutor} as a second, independent pin. + */ + OPEN_AND_CLOSE_CLIENT_THEN_DISABLE_PRUNING_AND_TOUCH_COMMON_EXECUTOR { + @Override + void run(final ClassLoader loader) throws Exception { + Class commonExecutorClass; + try { + commonExecutorClass = Class.forName("com.mongodb.internal.thread.CommonExecutor", true, loader); + } catch (ClassNotFoundException e) { + log("CommonExecutor is not on this branch -- scenario skipped"); + skipped = true; + return; + } + openAndCloseClient(loader); + disablePruning(defaultPool(loader)); + Object commonExecutor = commonExecutorClass.getMethod("commonExecutor").invoke(null); + // `schedule` is package private, as `disablePruning` is. + java.lang.reflect.Method schedule = commonExecutorClass.getDeclaredMethod( + "schedule", Runnable.class, java.time.Duration.class, java.util.concurrent.Executor.class); + schedule.setAccessible(true); + java.util.concurrent.Executor direct = Runnable::run; + schedule.invoke(commonExecutor, (Runnable) () -> { }, java.time.Duration.ofMillis(1), direct); + log("scheduled on CommonExecutor; live non-JVM threads: %s", nonJvmThreadNames()); + } + }; + + /** Set when a scenario cannot apply to the branch under test. */ + boolean skipped; + + abstract void run(ClassLoader loader) throws Exception; + + static Object defaultPool(final ClassLoader loader) throws Exception { + Class poolClass = Class.forName("com.mongodb.internal.connection.PowerOfTwoBufferPool", true, loader); + Object pool = poolClass.getField("DEFAULT").get(null); + log("initialized PowerOfTwoBufferPool.DEFAULT; live non-JVM threads: %s", nonJvmThreadNames()); + return pool; + } + + /** {@code disablePruning} is package private, hence the reflection. This is the workaround from issue 2029. */ + static void disablePruning(final Object pool) throws Exception { + java.lang.reflect.Method disablePruning = pool.getClass().getDeclaredMethod("disablePruning"); + disablePruning.setAccessible(true); + disablePruning.invoke(pool); + log("called PowerOfTwoBufferPool.disablePruning() reflectively"); + } + + static void openAndCloseClient(final ClassLoader loader) throws Exception { + Class mongoClients = Class.forName("com.mongodb.client.MongoClients", true, loader); + Object client = mongoClients.getMethod("create", String.class) + .invoke(null, System.getProperty("org.mongodb.test.uri", "mongodb://localhost:27017")); + log("created %s", client.getClass().getName()); + // No operation is issued, so no server is needed: the point is the client's own background threads. A + // failed heartbeat against an absent server is expected and harmless here. + ((AutoCloseable) client).close(); + log("closed the MongoClient; live non-JVM threads: %s", nonJvmThreadNames()); + } + } + + private static URL[] driverClasspath() throws MalformedURLException { + String raw = requireProperty("java6279.driverCp"); + List urls = new ArrayList<>(); + for (String entry : raw.split(File.pathSeparator)) { + if (!entry.isEmpty()) { + urls.add(Paths.get(entry).toUri().toURL()); + } + } + return urls.toArray(new URL[0]); + } + + // --------------------------------------------------------------------------------------------------------------- + // reachability plumbing + // --------------------------------------------------------------------------------------------------------------- + + /** + * The child class loader for the primer scenarios. Defines only {@code java6279.primer.*} itself and delegates + * everything else -- including {@link Poc} -- to its parent, so the primer classes can call back into + * parent-loaded code without that code becoming child-loaded. + */ + private static final class PrimerClassLoader extends ClassLoader { + private static int instanceCount; + private final Path classesDir; + private final String name; + + PrimerClassLoader(final Path classesDir) { + super(Poc.class.getClassLoader()); + this.classesDir = classesDir; + this.name = "java6279-primer-" + (++instanceCount); + } + + @Override + protected Class loadClass(final String name, final boolean resolve) throws ClassNotFoundException { + synchronized (getClassLoadingLock(name)) { + Class result = findLoadedClass(name); + if (result == null) { + result = name.startsWith(PRIMER_PACKAGE) ? findClass(name) : getParent().loadClass(name); + } + if (resolve) { + resolveClass(result); + } + return result; + } + } + + @Override + protected Class findClass(final String name) throws ClassNotFoundException { + if (!name.startsWith(PRIMER_PACKAGE)) { + throw new ClassNotFoundException(name); + } + byte[] classFile; + try { + classFile = Files.readAllBytes(classesDir.resolve(name.replace('.', '/') + ".class")); + } catch (IOException e) { + throw new ClassNotFoundException(name, e); + } + return defineClass(name, classFile, 0, classFile.length); + } + + @Override + public String toString() { + return name; + } + } + + /** + * Watches one referent and reports whether it becomes phantom reachable within a bounded window, nudging the + * collector as it goes. Holds no strong reference to the referent. + */ + private static final class PhantomReachableWatch { + private final java.lang.ref.PhantomReference reference; + private final java.lang.ref.ReferenceQueue queue = new java.lang.ref.ReferenceQueue<>(); + private final String description; + private final long startNanos = System.nanoTime(); + private Duration elapsed = Duration.ZERO; + + PhantomReachableWatch(final Object referent, final String description) { + this.description = description; + this.reference = new java.lang.ref.PhantomReference<>(referent, queue); + } + + boolean awaitCollected(final Duration window) throws InterruptedException { + long deadlineNanos = startNanos + window.toNanos(); + while (System.nanoTime() < deadlineNanos) { + System.gc(); + if (queue.remove(100) != null) { + elapsed = Duration.ofNanos(System.nanoTime() - startNanos); + log("%s became phantom reachable in %s", description, elapsed); + reference.clear(); + return true; + } + } + elapsed = Duration.ofNanos(System.nanoTime() - startNanos); + log("%s is still strongly reachable after %s", description, elapsed); + return false; + } + + Duration elapsed() { + return elapsed; + } + } + + // --------------------------------------------------------------------------------------------------------------- + // reporting + // --------------------------------------------------------------------------------------------------------------- + + private enum Expectation { + COLLECTED, PINNED, UNKNOWN + } + + private static final class Result { + private final String name; + private final String description; + private final Expectation expectation; + private final boolean collected; + private final Duration elapsed; + + Result(final String name, final String description, final Expectation expectation, final boolean collected, + final Duration elapsed) { + this.name = name; + this.description = description; + this.expectation = expectation; + this.collected = collected; + this.elapsed = elapsed; + } + + String observed() { + return collected ? "COLLECTED" : "PINNED"; + } + + boolean asExpected() { + return expectation == Expectation.UNKNOWN + || (expectation == Expectation.COLLECTED) == collected; + } + } + + private static void report(final List results) { + System.err.println(); + System.err.println("================================ results ================================"); + System.err.printf("%-9s %-9s %-50s %s%n", "OBSERVED", "EXPECTED", "SCENARIO", "AFTER"); + boolean allAsExpected = true; + for (Result result : results) { + allAsExpected &= result.asExpected(); + System.err.printf("%-9s %-9s %-50s %s%n", result.observed(), result.expectation, result.name, + result.elapsed); + System.err.printf("%33s%s%n", "", result.description); + } + System.err.println("========================================================================="); + System.err.println(allAsExpected + ? "every scenario matched its expectation" + : "AT LEAST ONE SCENARIO DID NOT MATCH ITS EXPECTATION -- see the table above"); + // The exit status reflects only whether the harness observed what it expected, so run.sh can be used in a + // pipeline. A PINNED result for the pruner scenarios is the bug, and is the *expected* outcome today. + if (!allAsExpected) { + System.exit(1); + } + } + + private static Set nonJvmThreadNames() { + Set names = new TreeSet<>(); + for (Thread thread : Thread.getAllStackTraces().keySet()) { + ThreadGroup group = thread.getThreadGroup(); + if (group != null && !"system".equals(group.getName()) && !thread.getName().equals("main")) { + names.add(thread.getName()); + } + } + return names.isEmpty() ? Collections.unmodifiableSet(new TreeSet<>(Arrays.asList("(none)"))) : names; + } + + private static String requireProperty(final String name) { + String value = System.getProperty(name); + if (value == null || value.isEmpty()) { + throw new IllegalStateException("system property " + name + " must be set; use run.sh"); + } + return value; + } + + private static void banner(final String scenario) { + System.err.println(); + System.err.println("---------------- " + scenario + " ----------------"); + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/Inert.java b/testing/java-6279-poc/src/java6279/primer/Inert.java new file mode 100644 index 00000000000..3ab098af327 --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/Inert.java @@ -0,0 +1,35 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * Starts nothing. Loaded in every primer scenario alongside the scenario's trigger class, so that the "pinning is + * loader-wide, not class-specific" part of the finding is visible: when a sibling class starts a thread, this class + * cannot be collected either. + * + *

Class {@code D} in the original experiment.

+ */ +final class Inert { + static { + Poc.log("%s is being initialized by %s", Inert.class.getName(), Inert.class.getClassLoader()); + } + + private Inert() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoader.java b/testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoader.java new file mode 100644 index 00000000000..89f5e99f0d4 --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoader.java @@ -0,0 +1,48 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * Isolates the context class loader edge, which is the one Netty's mitigation -- and the + * {@code DaemonThreadFactory.newThread} change of the same shape -- actually addresses. + * + *

Models a driver thread created while an application's class loader is current: the thread is constructed by a + * parent-loaded class (so there is no construction-site pin, per {@code StartsParentBuiltThread}), but the calling + * thread's context class loader is this child loader, so the new thread inherits it.

+ * + *

This is the inverse of JAVA-6279's own problem: the loader at risk here is the application's, pinned by a + * driver thread, rather than the driver's own.

+ */ +final class InheritsContextClassLoader { + static { + Poc.log("%s is being initialized by %s", InheritsContextClassLoader.class.getName(), InheritsContextClassLoader.class.getClassLoader()); + Thread callingThread = Thread.currentThread(); + ClassLoader previous = callingThread.getContextClassLoader(); + // Pretend an application thread with its own class loader is the one calling into the driver. + callingThread.setContextClassLoader(InheritsContextClassLoader.class.getClassLoader()); + try { + Poc.buildAndStartThreadInheritingCcl(false); + } finally { + callingThread.setContextClassLoader(previous); + } + } + + private InheritsContextClassLoader() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoaderButNulled.java b/testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoaderButNulled.java new file mode 100644 index 00000000000..bf0e0f62dab --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoaderButNulled.java @@ -0,0 +1,48 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * Isolates the context class loader edge, which is the one Netty's mitigation -- and the + * {@code DaemonThreadFactory.newThread} change of the same shape -- actually addresses. + * + *

Models a driver thread created while an application's class loader is current: the thread is constructed by a + * parent-loaded class (so there is no construction-site pin, per {@code StartsParentBuiltThread}), but the calling + * thread's context class loader is this child loader, so the new thread inherits it.

+ * + *

This is the inverse of JAVA-6279's own problem: the loader at risk here is the application's, pinned by a + * driver thread, rather than the driver's own. This variant applies the mitigation, so it shows whether nulling the context class loader closes that edge.

+ */ +final class InheritsContextClassLoaderButNulled { + static { + Poc.log("%s is being initialized by %s", InheritsContextClassLoaderButNulled.class.getName(), InheritsContextClassLoaderButNulled.class.getClassLoader()); + Thread callingThread = Thread.currentThread(); + ClassLoader previous = callingThread.getContextClassLoader(); + // Pretend an application thread with its own class loader is the one calling into the driver. + callingThread.setContextClassLoader(InheritsContextClassLoaderButNulled.class.getClassLoader()); + try { + Poc.buildAndStartThreadInheritingCcl(true); + } finally { + callingThread.setContextClassLoader(previous); + } + } + + private InheritsContextClassLoaderButNulled() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoaderNettyDance.java b/testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoaderNettyDance.java new file mode 100644 index 00000000000..42a39b79ecc --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/InheritsContextClassLoaderNettyDance.java @@ -0,0 +1,48 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * Isolates the context class loader edge, which is the one Netty's mitigation -- and the + * {@code DaemonThreadFactory.newThread} change of the same shape -- actually addresses. + * + *

Models a driver thread created while an application's class loader is current: the thread is constructed by a + * parent-loaded class (so there is no construction-site pin, per {@code StartsParentBuiltThread}), but the calling + * thread's context class loader is this child loader, so the new thread inherits it.

+ * + *

This is the inverse of JAVA-6279's own problem: the loader at risk here is the application's, pinned by a + * driver thread, rather than the driver's own. This variant nulls the CALLING thread's context class loader before construction, the way Netty does, rather than nulling the new thread's field afterwards.

+ */ +final class InheritsContextClassLoaderNettyDance { + static { + Poc.log("%s is being initialized by %s", InheritsContextClassLoaderNettyDance.class.getName(), InheritsContextClassLoaderNettyDance.class.getClassLoader()); + Thread callingThread = Thread.currentThread(); + ClassLoader previous = callingThread.getContextClassLoader(); + // Pretend an application thread with its own class loader is the one calling into the driver. + callingThread.setContextClassLoader(InheritsContextClassLoaderNettyDance.class.getClassLoader()); + try { + Poc.buildAndStartThreadNettyDance(); + } finally { + callingThread.setContextClassLoader(previous); + } + } + + private InheritsContextClassLoaderNettyDance() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/RegistersShutdownHook.java b/testing/java-6279-poc/src/java6279/primer/RegistersShutdownHook.java new file mode 100644 index 00000000000..b59449c1071 --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/RegistersShutdownHook.java @@ -0,0 +1,52 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * Registers a JVM shutdown hook that would stop the pruner, and starts no thread of its own. + * + *

This checks the other tempting fix: leave the pruner thread running and shut it down from a + * {@link Runtime#addShutdownHook(Thread)} hook. The hook body is a child-loaded class, as a real one would be, since + * the whole point would be to call driver code.

+ * + *

Expected to be worse than useless. {@code ApplicationShutdownHooks} keeps registered hooks in a static map held + * by a bootstrap-loaded class, so the hook thread — and through it its {@code Runnable}, this class, and this class + * loader — is strongly reachable until the JVM exits. Registering the hook therefore creates a permanent pin + * in a class that otherwise had none.

+ */ +final class RegistersShutdownHook { + static { + Poc.log("%s is being initialized by %s", RegistersShutdownHook.class.getName(), + RegistersShutdownHook.class.getClassLoader()); + // The Runnable is an instance of this child-loaded class, exactly as a hook that called driver code would be. + Runtime.getRuntime().addShutdownHook(new Thread(new HookBody(), "java6279-shutdown-hook")); + Poc.log("%s registered a shutdown hook and started no thread of its own", + RegistersShutdownHook.class.getName()); + } + + private static final class HookBody implements Runnable { + @Override + public void run() { + Poc.log("the shutdown hook ran"); + } + } + + private RegistersShutdownHook() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/RegistersShutdownHookNettyStyle.java b/testing/java-6279-poc/src/java6279/primer/RegistersShutdownHookNettyStyle.java new file mode 100644 index 00000000000..040eb52ef4a --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/RegistersShutdownHookNettyStyle.java @@ -0,0 +1,58 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * The combination: a shutdown hook, registered with Netty's context class loader nulling applied around the creation + * of the hook thread. The hook body is still a child-loaded class, because a hook that is going to call + * {@code disablePruning()} has to reference driver code. + * + *

Expected to remain pinned. Nulling the context class loader removes one edge; it does nothing about the + * {@code Runnable}, and {@code ApplicationShutdownHooks} holds the hook thread — and therefore its {@code Runnable}, + * and therefore this class and its loader — in a static map until the JVM exits.

+ */ +final class RegistersShutdownHookNettyStyle { + static { + Poc.log("%s is being initialized by %s", RegistersShutdownHookNettyStyle.class.getName(), + RegistersShutdownHookNettyStyle.class.getClassLoader()); + Thread callingThread = Thread.currentThread(); + ClassLoader parentCcl = callingThread.getContextClassLoader(); + callingThread.setContextClassLoader(null); + try { + Thread hook = new Thread(null, new HookBody(), "java6279-shutdown-hook-netty-style", 1, false); + hook.setContextClassLoader(null); + Runtime.getRuntime().addShutdownHook(hook); + Poc.log("%s registered a shutdown hook with context class loader %s", + RegistersShutdownHookNettyStyle.class.getName(), hook.getContextClassLoader()); + } finally { + callingThread.setContextClassLoader(parentCcl); + } + } + + /** Child-loaded, as any hook that called driver code would have to be. */ + private static final class HookBody implements Runnable { + @Override + public void run() { + Poc.log("the Netty-style shutdown hook ran"); + } + } + + private RegistersShutdownHookNettyStyle() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/RegistersShutdownHookParentBody.java b/testing/java-6279-poc/src/java6279/primer/RegistersShutdownHookParentBody.java new file mode 100644 index 00000000000..28e18f3ca41 --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/RegistersShutdownHookParentBody.java @@ -0,0 +1,52 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * The only shutdown hook shape that does not pin: the hook thread is CONSTRUCTED by a parent-loaded class, not + * merely bodied by one. Registering a hook thread that this class constructs pins the loader even with a + * parent-loaded {@code Runnable} and a null context class loader — measured, and consistent with + * {@code StartsOwnThread} versus {@code StartsParentBuiltThread}. + * + *

Expected to be collected — and to be useless, which is the point. A hook that references nothing in the driver's + * class loader cannot call driver code, so it cannot stop the pruner. The two properties are in direct tension: the + * hook pins exactly to the extent that it is capable of doing its job.

+ */ +final class RegistersShutdownHookParentBody { + static { + Poc.log("%s is being initialized by %s", RegistersShutdownHookParentBody.class.getName(), + RegistersShutdownHookParentBody.class.getClassLoader()); + Thread callingThread = Thread.currentThread(); + ClassLoader parentCcl = callingThread.getContextClassLoader(); + callingThread.setContextClassLoader(null); + try { + // Both CONSTRUCTED and bodied by a parent-loaded class. Constructing it here instead -- even with a + // parent-loaded Runnable and a null context class loader -- pins the loader, because thread construction + // captures the constructing class (see StartsOwnThread vs StartsParentBuiltThread). + Runtime.getRuntime().addShutdownHook(Poc.PARENT_BUILT_HOOK_THREAD); + Poc.log("%s registered a fully parent-built shutdown hook", + RegistersShutdownHookParentBody.class.getName()); + } finally { + callingThread.setContextClassLoader(parentCcl); + } + } + + private RegistersShutdownHookParentBody() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/StartsOwnThread.java b/testing/java-6279-poc/src/java6279/primer/StartsOwnThread.java new file mode 100644 index 00000000000..1d555bec837 --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/StartsOwnThread.java @@ -0,0 +1,45 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * Constructs and starts a thread from its own static initializer. This is the shape of the driver's own code: a + * class in the driver's class loader creates a thread that outlives the work that prompted it. + * + *

The {@code Runnable} comes from {@link Poc}, which the primer class loader delegates to its parent, so the + * running thread holds no reference into this class loader by way of its task. The thread is nonetheless expected to + * keep this class loader strongly reachable.

+ * + *

Class {@code C} in the original experiment, in its {@code new Thread(null, runnable, name, 1, false)} form.

+ */ +final class StartsOwnThread { + static { + Poc.log("%s is being initialized by %s", StartsOwnThread.class.getName(), + StartsOwnThread.class.getClassLoader()); + // The 5-argument constructor is the one the original experiment found sufficient to pin the loader: it takes + // no thread group and does not inherit thread locals, so the pinning cannot be explained by inherited state. + Thread thread = new Thread(null, Poc.SLEEPING_RUNNABLE, "java6279-own-thread", 1, false); + thread.start(); + Poc.log("%s started %s with context class loader %s", StartsOwnThread.class.getName(), thread.getName(), + thread.getContextClassLoader()); + } + + private StartsOwnThread() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/StartsOwnThreadNettyStyle.java b/testing/java-6279-poc/src/java6279/primer/StartsOwnThreadNettyStyle.java new file mode 100644 index 00000000000..01ead4c2b3b --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/StartsOwnThreadNettyStyle.java @@ -0,0 +1,54 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * {@link StartsOwnThread}, but applying Netty's class loader mitigation from + * {@code GlobalEventExecutor.startThread()}: null the creating thread's context class loader around the + * {@code new Thread(...)} call, null the new thread's context class loader, then restore. + * + *

Netty does this citing netty#7290 and + * JDK-7008595, with the comment "Avoid calling classloader + * leaking through Thread.inheritedAccessControlContext".

+ * + *

The question this scenario answers: does that mitigation also release our case — a thread whose own + * class lives in the loader we want collected — or does it only address the different edge Netty cares about, a + * long-lived global thread pinning whichever application class loader happened to be current when it started?

+ */ +final class StartsOwnThreadNettyStyle { + static { + Poc.log("%s is being initialized by %s", StartsOwnThreadNettyStyle.class.getName(), + StartsOwnThreadNettyStyle.class.getClassLoader()); + Thread callingThread = Thread.currentThread(); + ClassLoader parentCcl = callingThread.getContextClassLoader(); + callingThread.setContextClassLoader(null); + try { + Thread thread = new Thread(null, Poc.SLEEPING_RUNNABLE, "java6279-netty-style", 1, false); + thread.setContextClassLoader(null); + thread.start(); + Poc.log("%s started %s with context class loader %s", StartsOwnThreadNettyStyle.class.getName(), + thread.getName(), thread.getContextClassLoader()); + } finally { + callingThread.setContextClassLoader(parentCcl); + } + } + + private StartsOwnThreadNettyStyle() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/StartsParentBuiltThread.java b/testing/java-6279-poc/src/java6279/primer/StartsParentBuiltThread.java new file mode 100644 index 00000000000..478e9335ee4 --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/StartsParentBuiltThread.java @@ -0,0 +1,42 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +/** + * Starts a thread whose {@code Thread} object was constructed by a parent-loaded class ({@link Poc}) rather than by + * this class. Only the {@code start()} call happens with this class on the stack. + * + *

The original experiment noted that this variant does not pin the loader, which is what isolates thread + * construction, rather than thread execution or the thread's context class loader, as the point at which the + * loader is captured. The harness records this scenario's outcome without asserting it, since it is the one result + * that plausibly varies by JVM and JDK version.

+ */ +final class StartsParentBuiltThread { + static { + Poc.log("%s is being initialized by %s", StartsParentBuiltThread.class.getName(), + StartsParentBuiltThread.class.getClassLoader()); + Thread thread = Poc.PARENT_BUILT_THREAD; + thread.start(); + Poc.log("%s started %s with context class loader %s", StartsParentBuiltThread.class.getName(), + thread.getName(), thread.getContextClassLoader()); + } + + private StartsParentBuiltThread() { + } +} diff --git a/testing/java-6279-poc/src/java6279/primer/StaticSingletonExecutor.java b/testing/java-6279-poc/src/java6279/primer/StaticSingletonExecutor.java new file mode 100644 index 00000000000..8e046843fba --- /dev/null +++ b/testing/java-6279-poc/src/java6279/primer/StaticSingletonExecutor.java @@ -0,0 +1,70 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java6279.primer; + +import java6279.Poc; + +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; + +/** + * Models {@code com.mongodb.internal.thread.CommonExecutor} from the backpressure work + * (stIncMale PR 3, JAVA-6240): a static singleton + * holding a {@code ScheduledThreadPoolExecutor} that nothing ever shuts down. + * + *

Also registers the {@link java.lang.ref.Cleaner} that {@code CommonExecutor}'s {@code VAKOTODO} proposes as the + * eventual fix — "use Cleaner when we are at Java SE 17 to shut down internal executors if the class is GCed" — so + * that the proposal can be checked rather than assumed. The cleaning action holds no reference to the singleton, as + * {@code Cleaner} requires.

+ * + *

The expectation is that the cleaning action never runs, because the reachability is circular: the singleton is + * reachable from its class, the class from its class loader, and the loader is pinned by the very thread the cleaning + * action was supposed to stop.

+ */ +final class StaticSingletonExecutor { + /** As {@code CommonExecutor.INSTANCE} is: a static field, so reachable for as long as the class is loaded. */ + private static final StaticSingletonExecutor INSTANCE = new StaticSingletonExecutor(); + + private final ScheduledThreadPoolExecutor singleThreadScheduler; + + static { + Poc.log("%s is being initialized by %s", StaticSingletonExecutor.class.getName(), + StaticSingletonExecutor.class.getClassLoader()); + // Capture the executor, not the singleton: a cleaning action that referenced INSTANCE would pin it by itself + // and the check would prove nothing. + final ScheduledThreadPoolExecutor executor = INSTANCE.singleThreadScheduler; + Poc.registerCleaner(INSTANCE, () -> { + Poc.log("the cleaning action ran; shutting the executor down"); + executor.shutdownNow(); + }); + // CommonExecutor starts its thread lazily, on the first schedule() call. Model that. + INSTANCE.singleThreadScheduler.scheduleAtFixedRate( + () -> { }, 0, 50, TimeUnit.MILLISECONDS); + Poc.log("%s scheduled a task, starting the CommonScheduler-equivalent thread", + StaticSingletonExecutor.class.getName()); + } + + private StaticSingletonExecutor() { + singleThreadScheduler = new ScheduledThreadPoolExecutor(1, + runnable -> { + Thread thread = new Thread(runnable, "java6279-CommonScheduler"); + thread.setDaemon(true); + return thread; + }); + singleThreadScheduler.setRemoveOnCancelPolicy(true); + } +}