feat: add automatic S3 request retry with exponential backoff

[allanrogerr]

Description

Implements SDK-level automatic retry with full-jitter exponential backoff, matching the behaviour specified in minio-go's retry-mechanism spec. All requests through BaseS3Client.executeAsync() now automatically retry on retryable S3 error codes, retryable HTTP status codes, and transient network/IO errors up to maxRetries attempts (default 10). The retry policy lives in a new package-private Retry class; the execution loop is wired into BaseS3Client.executeWithRetry() using a daemon ScheduledExecutorService for Java 8–compatible backoff delays.

Motivation and Context

The SDK had no SDK-level retry capability. Transient failures — server-side throttling (SlowDown, InternalError), gateway errors (502, 503, 504), request timeouts (408), and connection resets — propagated directly to callers, requiring every consumer to implement its own retry loop. The Go client has had this capability since its inception; this PR closes the gap for the Java SDK. It does not address #1700 since error code BucketNotEmpty (which maps to http code 409) is not natively retriable in minio-go.
Retryable S3 codes (matching minio-go retryableS3Codes): RequestError, RequestTimeout, Throttling, ThrottlingException, RequestLimitExceeded, RequestThrottled, InternalError, ExpiredToken, ExpiredTokenException, SlowDown, SlowDownWrite, SlowDownRead.
Retryable HTTP status codes (matching minio-go retryableHTTPStatusCodes): 408, 429, 499, 500, 502, 503, 504, 520.
Retryable IO errors: all IOException except TLS handshake failures and HTTP/HTTPS protocol mismatch — matching minio-go's isRequestErrorRetryable.
Seekability gate: non-seekable bodies (okhttp3.RequestBody) get exactly one attempt; seekable bodies (byte[], ByteBuffer, RandomAccessFile) retry up to maxRetries — matching minio-go's body-seeker check.
Backoff: rand(0, min(1 s, 200 ms × 2ⁿ)) before the nth retry (full jitter, capped at 1 s), matching minio-go's DefaultRetryUnit/DefaultRetryCap/MaxJitter defaults.
RetryHead guard: the "RetryHead" S3 error code is explicitly excluded from retry so executeHeadAsync region-redirect logic is unaffected.

How to test this PR?

Run ./gradlew :api:check. The 43 JUnit 4 tests in api/src/test/java/io/minio/RetryTest.java cover:

• all 12 retryable S3 codes,
• all 8 retryable HTTP codes,
• IOException classification (retryable vs. SSL/protocol-mismatch),
• isRetryable() dispatch for all four exception types,
• backoff bounds for attempts 0–5+, and
• MockWebServer integration tests for
  • 500-then-success,
  • 503-then-success,
  • S3-InternalError-then-success,
  • S3-SlowDown-then-success,
  • 429/502/504-then-success,
  • no-retry-on-404/403,
  • exhausted-retries-throws-last-error,
  • maxRetries=1 disabling retry,
  • setMaxRetries post-construction,
  • three-attempt sequence,
  • seekable-body retry (PUT with byte[]),
  • path consistency across retries, and
• the InvalidResponseException.responseCode() accessor.

OR build a local mint against the code in this branch the run mint e.g.

docker run --rm --network host -e SERVER_ENDPOINT="127.0.0.1:9000" -e ACCESS_KEY="minioadmin" -e SECRET_KEY="minioadmin" -e ENABLE_HTTPS=0 -e MINT_MODE="full" minio/mint:local minio-java
Running with
SERVER_ENDPOINT:      127.0.0.1:9000
ACCESS_KEY:           minioadmin
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0
RUN_ON_FAIL:          0

To get logs, run 'docker cp :/mint/log /tmp/mint-logs'

(1/1) Running minio-java tests ... done in 46 seconds

All tests ran successfully

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Cleanup/Maintenance (no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Unit tests added/updated
• Internal documentation updated
• Create a documentation update request here
• No internode changes / version interoperability introduced

[Copilot]

Pull request overview

This PR introduces SDK-level automatic retry for S3 requests with full-jitter exponential backoff, aligning the Java SDK’s behavior with minio-go’s retry policy and making transient S3/HTTP/IO failures less likely to leak to callers.

Changes:

• Added a new package-private Retry helper for retryability classification (S3 codes, HTTP status codes, IO exceptions) and jittered exponential backoff.
• Wired a retry loop into BaseS3Client.executeAsync() with delayed retries via a daemon ScheduledExecutorService, plus a seekability-based single-attempt gate for non-replayable bodies.
• Exposed configuration via maxRetries(...) on builders and setMaxRetries(...), and added an InvalidResponseException.responseCode() accessor plus a comprehensive new RetryTest suite.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
api/src/main/java/io/minio/BaseS3Client.java	Adds execute-with-retry loop, backoff scheduling, and max-retry configuration.
api/src/main/java/io/minio/Retry.java	Defines retryable error sets and backoff computation.
api/src/main/java/io/minio/MinioAsyncClient.java	Adds maxRetries(...) builder option and applies it to the built client.
api/src/main/java/io/minio/MinioClient.java	Exposes setMaxRetries(...) and maxRetries(...) on the synchronous client/builder.
api/src/main/java/io/minio/Http.java	Adds Http.S3Request.body() accessor used for retry seekability gating.
api/src/main/java/io/minio/errors/InvalidResponseException.java	Stores/returns the triggering HTTP response code for retry classification.
api/src/test/java/io/minio/RetryTest.java	Adds unit + MockWebServer integration tests covering retry classification and behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[balamurugana]

The feature can be simplified by

diff --git a/api/src/main/java/io/minio/Http.java b/api/src/main/java/io/minio/Http.java
index 5fa9bec2..23ef9378 100644
--- a/api/src/main/java/io/minio/Http.java
+++ b/api/src/main/java/io/minio/Http.java
@@ -52,6 +52,7 @@ import java.util.Locale;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
 import java.util.concurrent.TimeUnit;
 import java.util.regex.Matcher;
 import java.util.stream.Collectors;
@@ -65,9 +66,12 @@ import javax.net.ssl.SSLSocketFactory;
 import javax.net.ssl.TrustManager;
 import javax.net.ssl.TrustManagerFactory;
 import javax.net.ssl.X509TrustManager;
+import okhttp3.Interceptor;
 import okhttp3.MediaType;
 import okhttp3.OkHttpClient;
 import okhttp3.Protocol;
+import okhttp3.Request;
+import okhttp3.Response;
 import okio.BufferedSink;
 import okio.Okio;
 
@@ -608,6 +612,54 @@ public class Http {
         client, System.getenv("SSL_CERT_FILE"), System.getenv("SSL_CERT_DIR"));
   }
 
+  public static class StatusRetryInterceptor implements Interceptor {
+    private final int maxRetries;
+    private final Set<Integer> retryStatusCodes;
+    private final long delayMs; // This acts as our "base" delay (e.g., 100ms)
+
+    public StatusRetryInterceptor(int maxRetries, Set<Integer> retryStatusCodes, long delayMs) {
+      this.maxRetries = maxRetries;
+      this.retryStatusCodes = retryStatusCodes;
+      this.delayMs = delayMs;
+    }
+
+    @Override
+    public Response intercept(Chain chain) throws IOException {
+      okhttp3.Request request = chain.request();
+      okhttp3.Response response = chain.proceed(request);
+
+      int tryCount = 0;
+
+      while (retryStatusCodes.contains(response.code()) && tryCount < maxRetries) {
+        tryCount++;
+
+        // Critical step: close the response body to avoid socket leaks
+        response.close();
+
+        if (delayMs > 0) {
+          // 1. Calculate exponential backoff limit: base * 2^tryCount
+          // This doubles the maximum potential wait time with each retry
+          long maxBackoffLimit = delayMs * (1L << tryCount); // Using bit-shift for 2^tryCount
+
+          // 2. Apply random jitter: pick a random number between 0 and maxBackoffLimit
+          long jitteredDelay = ThreadLocalRandom.current().nextLong(0, maxBackoffLimit);
+
+          try {
+            Thread.sleep(jitteredDelay);
+          } catch (InterruptedException e) {
+            Thread.currentThread().interrupt();
+            throw new IOException("Retry interrupted", e);
+          }
+        }
+
+        // Execute the request again
+        response = chain.proceed(request);
+      }
+
+      return response;
+    }
+  }
+
   /**
    * Creates new HTTP client with default timeout with additional TLS certificates from
    * SSL_CERT_FILE and SSL_CERT_DIR environment variables if present.
@@ -620,6 +672,19 @@ public class Http {
             .writeTimeout(DEFAULT_TIMEOUT, TimeUnit.MILLISECONDS)
             .readTimeout(DEFAULT_TIMEOUT, TimeUnit.MILLISECONDS)
             .protocols(Arrays.asList(Protocol.HTTP_1_1))
+            .addInterceptor(
+                new StatusRetryInterceptor(
+                    5,
+                    ImmutableSet.of(
+                        408, // Request Timeout
+                        429, // Too Many Requests
+                        499, // Client Closed Request (nginx)
+                        500, // Internal Server Error
+                        502, // Bad Gateway
+                        503, // Service Unavailable
+                        504, // Gateway Timeout
+                        520), // Cloudflare unknown error
+                    1000))
             .build();
     try {
       return enableExternalCertificatesFromEnv(client);

[allanrogerr]

Thank you for the suggestion — an OkHttp interceptor is an appealing simplification for the HTTP-status case. The reason the current implementation uses a CompletableFuture-based wrapper instead is that the interceptor approach has three gaps that would be regressions for this PR:

1. S3 XML error codes are not visible at the interceptor layer.
S3 errors like SlowDown, ThrottlingException, InternalError, ExpiredToken etc. arrive as HTTP 200 responses with an XML body — the status code is always 200. An interceptor keyed on response.code() never sees these; they must be classified after parsing the body in throwMinioException(), which is downstream of OkHttp.

2. Thread.sleep() inside an interceptor blocks an OkHttp dispatcher thread.
OkHttp's async dispatcher has a bounded thread pool (default 64). Sleeping inside Callback.onResponse / an interceptor for exponential backoff (up to ~31 s at 5 retries × 1 000 ms base) would hold a dispatcher thread for the full delay, starving other concurrent requests. The current design schedules the next attempt on RETRY_SCHEDULER and releases the dispatcher thread immediately.

3. Non-seekable request bodies cannot be replayed via chain.proceed(request).
RequestBody.writeTo() is called once; OkHttp does not buffer or reset it. For multipart uploads or streamed PUT bodies, a second chain.proceed would send an empty or corrupt body. The current wrapper gates retry on the body's seekability before re-issuing the request.

If you'd prefer to move the HTTP-status subset into an interceptor and keep only S3-code + IOException handling in the wrapper, I'm happy to split it that way — but I wanted to flag the above before making that change.

[balamurugana]

1. S3 XML error codes are not visible at the interceptor layer.

No S3 error is returned with 2xx HTTP status code. If so, it is a bug at server side. As @harshavardhana mentioned previously we do not need to use S3 error code instead we need to use HTTP status code.

2. Thread.sleep() inside an interceptor blocks an OkHttp dispatcher thread.

This is fine. If the server is busy, why send new requests and wait in another thread. New request to the same server is blocked if maxRequestsPerHost = 5 is hit due to retry sleep. Instead of 1s, we need to start the sleep with 100ms or lesser.

3. Non-seekable request bodies cannot be replayed via chain.proceed(request).

You don't need to. We have retry-able custom RequestBody. It works for ByteBuffer, RandomAccessFile and byte[].

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.