External Storage Integration: WorkflowWorker, replay, history by cconstable · Pull Request #3017 · temporalio/sdk-java

cconstable · 2026-08-18T20:40:18Z

What was changed

Integrated external storage into the replay handler and history iterator.
WorkflowWorker handle + send response now use external storage.

Why

Workflow workers should utilize external storage and fetch payloads on replay.

Checklist

Added tests.

… replay handler.

…nal payloads after fetching history.

…ng command.getAttributesCase() + switch for exhaustiveness checking.

…ails.

jmaeagle99 · 2026-09-01T15:25:25Z

+      Duration stickyTaskQueueScheduleToStartTimeout,
+      WorkflowServiceStubs service,
+      LocalActivityDispatcher localActivityDispatcher,
+      CancellationToken<CancellationException> storageCancellation) {


Instead of passing in a cancellation token to each handler type, can we update ExternalStorageRunner to have a shutdown method (or something similar) and have the owner of the runner shut it down from one place?

Hmm yea I think this would simplify this. Will update unless I can think of a reason why we would need the token

Thought about this a bit and I don't think we could add a shutdown method. A shutdown method would cancel everything and external storage is shared by across clients and workers (that is how we will eventually get our concurrency controls). One worker shutting down shouldn't cause external storage to shutdown in the others.

We could potentially put a token on the worker options that can be used to then cancel all inflight extstore operations and get rid of the threading.

We also still need to retain the ability to cancel individual extstore operations for things like heartbeats (they can be canceled)

jmaeagle99 · 2026-09-01T15:28:22Z

+        return current;
+      case ATTRIBUTES_NOT_SET:
+      case START_TIMER_COMMAND_ATTRIBUTES:
+      case COMPLETE_WORKFLOW_EXECUTION_COMMAND_ATTRIBUTES:


This one should be retargeting to the parent if (1) the parent is known, and (2) is not a CaN'd workflow. Just like the other SDKs already do.

This is going to require some plumbing. I don't think we have access to that information here

jmaeagle99 · 2026-09-01T15:29:42Z

+    CommandOrBuilder command = (CommandOrBuilder) message;
+    // Keep this exhaustive so new command attributes require an explicit target decision.
+    switch (command.getAttributesCase()) {
+      case SCHEDULE_ACTIVITY_TASK_COMMAND_ATTRIBUTES:


This should not be retargeted to the activity and should remain as the workflow. The external storage is tied to the lifetime of the workflow, not the activity.

jmaeagle99 · 2026-09-01T15:35:30Z

    String workflowType = workflowTask.getWorkflowType().getName();
    Scope metricsScope =
        options.getMetricsScope().tagged(ImmutableMap.of(MetricsTag.WORKFLOW_TYPE, workflowType));
+    ExternalStorageRunner externalStorageRunner = options.getExternalStorageRunner();


These are executed outside of the try-catch that would proactively fail the workflow task; otherwise, an external storage failure is going to just end task processing and wait for the server to timeout.

jmaeagle99 · 2026-09-01T15:36:48Z

+    }
+    try {
+      externalStorageRunner.store(builder, target, targetVisitor, storageCancellation);
+    } catch (Throwable e) {


Ignore the catch if it's cancelled?

Yes, should be checking if it's a cancellation here.

Ok I think we do need to check for cancellation here but we also need we also need to throw the cancellation and then check it somewhere up the chain (sendTaskCompleted) and explicitly handle cancellation there (e.g. if the worker is shutting down we want to ignore the error). This was not very easy to understand.

We can't just ignore it here because that would cause the completion event to be sent in sendTaskCompleted.

Another issue: when the task fails from this it just says Failed to send workflow task completion: External storage store failed even if it was cancellation. That seems misleading.

jmaeagle99 · 2026-09-01T15:59:09Z

          grpcRetryOptions);
    }

    private void sendDirectQueryCompletedResponse(


Should have test coverage for this.

jmaeagle99 · 2026-09-01T16:07:07Z

+                      workflowTypeScope,
+                      workflowStorageTarget(workflowExecution, workflowType));
+                } catch (ExternalStorageTaskFailure e) {
+                  if (currentTask.getAttempt() > 1) {


This is behaviorally different than the other SDKs. I think this came from duplicating the gRPC response path. However, they are different causes, where too large gRPC is not fixable but external storage failures likely are (missing replicated payloads, service temporarily faulted, etc).

That makes sense. Will remove this check.

jmaeagle99 · 2026-09-01T16:07:45Z

+      WorkflowServiceStubs service,
+      LocalActivityDispatcher localActivityDispatcher,
+      CancellationToken<CancellationException> storageCancellation) {
+    this.storageCancellation = storageCancellation;


I don't think there is test coverage for the shutdown / cancellation behavior.

jmaeagle99 · 2026-09-01T16:09:12Z

  }

+  @Test
+  public void resolvesExternalStorageReferencesInFetchedFullHistory() throws Throwable {


This test probably passed even if external storage just passed the payload through without modification. So the test isn't demonstrating any capability of external storage.

jmaeagle99 · 2026-09-01T16:10:50Z

    return mockFactory;
  }
+
+  private static final class InMemoryStorageDriver implements StorageDriver {


There are several test driver implementations that nearly do the same thing. Can we make one and alter the behavior via parameters / options?

…ellation gracefully during shutdown

…UTES

…retarget to parent for storage target.

…'t fail

cconstable force-pushed the extstore/workflow-worker branch from dff9b7f to 3a20dcf Compare August 18, 2026 21:04

cconstable mentioned this pull request Aug 19, 2026

External Storage Integration: Lazy resolving references, general refactoring #3016

Merged

cconstable changed the base branch from main to extstore/foundation August 19, 2026 15:58

cconstable force-pushed the extstore/foundation branch from 831c284 to 460bfbf Compare August 19, 2026 17:40

cconstable changed the base branch from extstore/foundation to main August 19, 2026 17:53

cconstable changed the base branch from main to extstore/foundation August 19, 2026 17:53

cconstable force-pushed the extstore/workflow-worker branch from 3a20dcf to 0e37f9d Compare August 19, 2026 18:12

cconstable changed the title ~~extstore/workflow worker~~ External Storage Integration: WorkflowWorker, replay, history Aug 19, 2026

cconstable force-pushed the extstore/workflow-worker branch from 0e37f9d to c67e0f0 Compare August 19, 2026 18:30

cconstable marked this pull request as ready for review August 19, 2026 18:57

cconstable requested a review from a team as a code owner August 19, 2026 18:57

cconstable force-pushed the extstore/foundation branch from 460bfbf to b3804da Compare August 21, 2026 20:35

cconstable force-pushed the extstore/workflow-worker branch from c67e0f0 to e292757 Compare August 21, 2026 21:57

cconstable force-pushed the extstore/foundation branch from b3804da to ca09b50 Compare August 24, 2026 01:23

cconstable force-pushed the extstore/workflow-worker branch from e292757 to cb05443 Compare August 24, 2026 01:25

cconstable commented Aug 24, 2026

View reviewed changes

Comment thread temporal-sdk/src/main/java/io/temporal/internal/worker/WorkflowWorker.java Outdated

cconstable force-pushed the extstore/workflow-worker branch 5 times, most recently from 6d90f7e to 1538817 Compare August 27, 2026 21:28

cconstable force-pushed the extstore/foundation branch from 2d2d90e to 504467c Compare August 28, 2026 16:10

Base automatically changed from extstore/foundation to main August 28, 2026 17:27

cconstable force-pushed the extstore/workflow-worker branch from 1538817 to d7b9a62 Compare August 28, 2026 21:36

cconstable added 6 commits August 31, 2026 11:47

feature(extstore): integrate into workflow worker pipeline, including…

09573ca

… replay handler.

feat(extstore): make sure sticky cache miss path also retrieves exter…

55b9f7b

…nal payloads after fetching history.

refactor(extstore): refactor the way we derive storage targets by usi…

03c16cd

…ng command.getAttributesCase() + switch for exhaustiveness checking.

Explicitly pass cancellation tokens for external storage methods.

0a91150

more cancellation token threading

4a7ae5e

externalStorage -> externalStorageRunner

8062db7

cconstable force-pushed the extstore/workflow-worker branch from d7b9a62 to 8062db7 Compare August 31, 2026 15:49

fix(extstore): return workflow task failures properly when extstore f…

7be2124

…ails.

cconstable mentioned this pull request Aug 31, 2026

External Storage Integration: NexusWorker #3018

Open

jmaeagle99 reviewed Sep 1, 2026

View reviewed changes

cconstable added 9 commits September 1, 2026 13:39

fix(extstore): don't fail tasks when extstore fails more than once.

7ccf9a0

fix(extstore): add error for worker shutdown and handle extstore canc…

7d02fc0

…ellation gracefully during shutdown

fix(extstore): correctly target SCHEDULE_ACTIVITY_TASK_COMMAND_ATTRIB…

268207f

…UTES

fix(extstore): COMPLETE_WORKFLOW_EXECUTION_COMMAND_ATTRIBUTES should …

f0f193f

…retarget to parent for storage target.

fix(extstore): run extstore inside the try of replay handler.

4f2550e

fix replay tests

2f3b425

add more tests

7ffd079

consolidate test storage drivers.

5a7e291

if the worker is shutting down and extstore retrieve is canceled, don…

c29f141

…'t fail

Uh oh!

Conversation

cconstable commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why

Checklist

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cconstable Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cconstable commented Aug 18, 2026 •

edited

Loading

cconstable Sep 1, 2026 •

edited

Loading