Regression in 2.52.0: hangs from certain exceptions, rather than errors propagating to callers

[eager-signal]

Describe the bug

We have an integration test that shuts down the DynamoDBLocal server and then makes a call that is expected to fail. Starting with 2.52.0, that test times out, rather than getting the expected error. It appears to have been a subtle case not handled by #7197.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Scenarios like a remote server being unavailable should be visible to callers via standard exception mechanisms.

Current Behavior

Certain errors are not propagated back to callers.

Reproduction Steps

/**
 * Regression test for a hang introduced in 2.52.0 by "Fix adaptive retries implementation" (#7197).
 *
 * <p>{@code AsyncRetryableStage.maybeAttemptExecute} used to run synchronously, so anything it threw was caught by
 * {@code maybeRetryExecute}'s {@code catch (Throwable t) { future.completeExceptionally(t); }}. In 2.52.0 the body moved
 * inside a {@code whenComplete} callback, and {@code whenComplete} does not propagate to its caller -- the throwable is
 * captured into the returned (discarded) stage. That catch block is now unreachable, so the response future is never
 * completed and the caller waits forever.
 *
 * <p>The throw used here is the {@code RejectedExecutionException} from
 * {@code scheduledExecutor.schedule(() -> attemptExecute(future), ...)}, which is what a shut-down SDK scheduled
 * executor produces (e.g. a request racing client {@code close()}, or a caller-supplied
 * {@code ClientOverrideConfiguration#scheduledExecutorService} that rejects).
 *
 * <p>Expected: the future completes exceptionally (passes on 2.51.4).
 * Actual on 2.52.0+: the future never completes, and {@code get} times out.
 */
public class AsyncRetryableStageRejectedScheduleTest {

    @Test
    void execute_retryBackoffScheduleRejected_completesFutureExceptionally() throws Exception {
        // A scheduled executor that rejects everything, as a shut-down one does.
        ScheduledExecutorService rejectingExecutor = Executors.newScheduledThreadPool(1);
        rejectingExecutor.shutdown();

        RetryToken token = mock(RetryToken.class);
        RetryStrategy retryStrategy = mock(RetryStrategy.class);

        // First attempt: allowed immediately, so it runs inline and never touches the scheduler.
        when(retryStrategy.acquireInitialTokenAsync(any()))
            .thenReturn(CompletableFuture.completedFuture(AcquireInitialTokenResponse.create(token, Duration.ZERO)));

        // The attempt failed and the strategy says "retry" -- which makes the stage schedule the next attempt.
        when(retryStrategy.refreshRetryTokenAsync(any()))
            .thenReturn(CompletableFuture.completedFuture(RefreshRetryTokenResponse.create(token, Duration.ZERO)));

        RequestPipeline<SdkHttpFullRequest, CompletableFuture<Response<SdkResponse>>> delegate = mock(RequestPipeline.class);
        when(delegate.execute(any(), any()))
            .thenReturn(CompletableFutureUtils.failedFuture(new IOException("connection reset")));

        AsyncRetryableStage<SdkResponse> stage =
            new AsyncRetryableStage<>(mock(TransformingAsyncResponseHandler.class),
                                      HttpClientDependencies.builder()
                                                            .clientConfiguration(clientConfig(retryStrategy,
                                                                                              rejectingExecutor))
                                                            .build(),
                                      delegate);

        SdkHttpFullRequest request = SdkHttpFullRequest.builder()
                                                      .method(SdkHttpMethod.GET)
                                                      .uri(URI.create("https://my-service.amazonaws.com"))
                                                      .build();

        CompletableFuture<Response<SdkResponse>> executeFuture = stage.execute(request, requestExecutionContext());

        // On 2.52.0+ this reports TimeoutException instead: the future is never completed at all.
        ExecutionException e = assertThrows(ExecutionException.class, () -> executeFuture.get(2, TimeUnit.SECONDS));
        assertInstanceOf(RejectedExecutionException.class, e.getCause());
    }

    private static SdkClientConfiguration clientConfig(RetryStrategy retryStrategy, ScheduledExecutorService scheduler) {
        return SdkClientConfiguration.builder()
                                     .option(SdkClientOption.RETRY_STRATEGY, retryStrategy)
                                     .option(SdkClientOption.SCHEDULED_EXECUTOR_SERVICE, scheduler)
                                     .build();
    }

    private static RequestExecutionContext requestExecutionContext() {
        ExecutionAttributes attrs =
            ExecutionAttributes.builder()
                               .put(SdkInternalExecutionAttribute.NEW_RETRIES_2026_ENABLED, true)
                               .build();

        return RequestExecutionContext.builder()
                                      .originalRequest(mock(SdkRequest.class))
                                      .executionContext(ExecutionContext.builder()
                                                                        .metricCollector(NoOpMetricCollector.create())
                                                                        .executionAttributes(attrs)
                                                                        .build())
                                      .build();
    }
}

Possible Solution

In AsyncRetryableStage#attemptFirstExecute and AsyncRetryableStage#maybeAttemptExecute, add a catch to the logic inside retryableStageHelper.tryRefreshTokenAsync(suggestedDelay()).whenComplete( … ) that calls future.completeExceptionally(t)

Additional Information/Context

No response

AWS Java SDK version used

2.52.0

JDK version used

25.0.3

Operating System and version

Linux

[dagnir]

Hi @eager-signal thank you for the thorough report. I've put up a fix pending review #7315.

[dagnir]

#7315 will be included in tomorrow's release.

[eager-signal]

We have an integration test that shuts down the DynamoDBLocal server and then makes a call that is expected to fail.

Confirmed that the integration test passes again with 2.54.4.

[dagnir]

Awesome! Thanks for confirming. Going ahead and closing this.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.