From 8d81cac14b5bd67aae2104ce0975d33f94ef0fd6 Mon Sep 17 00:00:00 2001 From: Garrett Beatty Date: Thu, 3 Sep 2026 21:04:08 -0400 Subject: [PATCH 1/5] feat(DurableExecution): per-operation serializer override (#2555) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(DurableExecution): add per-operation serializer override Add an optional `ILambdaSerializer? Serializer` to StepConfig, CallbackConfig, InvokeConfig, WaitForConditionConfig, and ChildContextConfig. DurableContext resolves `config?.Serializer ?? ILambdaContext.Serializer` per operation; the operation classes are otherwise unchanged. When Serializer is null (default), behavior is identical to today (the globally-registered serializer). Boundary serialization (workflow input, handler return, service envelopes) is untouched. Reuses the existing ILambdaSerializer contract (no new SerDes interface); AOT-safe because T stays concrete at each call site. Additive / non-breaking. - Unit tests (PerOperationSerializerTests) covering per-op serialize, default fallback, and replay-deserialize for Step/Callback/Invoke/ChildContext/WaitForCondition. - Integration test (TestFunctions/PerOperationSerializerFunction + PerOperationSerializerTest) verifying per-step routing end-to-end (camelCase override vs PascalCase default). - docs/core updates and an AutoVer change file (Minor). Separate feature from #2540 (FileSystem offload). Phase 1 (single-result ops); Map/Parallel two-level slots are a follow-up (Phase 2). * feat(DurableExecution): add per-item serializer for Map/Parallel Add an optional `ILambdaSerializer? ItemSerializer` to MapConfig and ParallelConfig. DurableContext.RunMap/RunParallel resolve `config.ItemSerializer ?? ILambdaContext.Serializer` and pass it as the serializer used for per-item/branch results (each unit's child-context checkpoint and the inline copy on the operation's summary). The aggregated batch envelope (per-unit statuses + completion reason) is a source-generated structure and is unchanged — there is no separate whole-result serializer (matches the Java SDK). Additive / non-breaking. - Unit tests: Map_WithItemSerializer, Map_NoItemSerializer, Parallel_WithItemSerializer. - docs/core (parallel.md, steps.md) + AutoVer change file. Phase 2 of the per-operation serializer feature (Phase 1 = single-result ops). * test(DurableExecution): cloud integration test for Map/Parallel ItemSerializer Deploys MapParallelItemSerializerFunction and asserts from event history that map items and a parallel branch configured with a camelCase ItemSerializer produce camelCase per-item child-context result payloads, while a control step (global serializer) stays PascalCase. Validates Phase 2 end-to-end on AWS. * address review: consistent serializer resolution, config headers, serializer unit coverage - DurableContext: Parallel now resolves the effective serializer via LambdaSerializerHelper.GetRequired(LambdaContext) like every other operation, instead of an inline 'LambdaContext.Serializer ?? throw' (consistent error path). - Add the standard copyright/SPDX header to MapConfig.cs and WaitForConditionConfig.cs. - Add unit tests: Invoke fresh (non-replay) path serializes the request payload via the per-op serializer; Map/Parallel ItemSerializer is used to deserialize cached per-item/per-branch results on replay. * test(DurableExecution): per-op serdes conformance + fresh-success serializer round-trip (#2556) * test(DurableExecution): conformance handlers for per-item/result serdes (map 9-14, parallel 8-15, invoke 5-16) Add conformance handlers exercising the new per-operation serializer slots: - map/MapCustomSerdes (9-14): MapConfig.ItemSerializer wraps each item result - parallel/ParallelCustomSerdes (8-15): ParallelConfig.ItemSerializer wraps each branch result - invoke/InvokeCustomResultSerdes (5-16): InvokeConfig.Serializer uppercases the result on deserialize Remove the corresponding NotImplemented declarations and wire the resources in template_map/parallel/invoke.yaml. All three pass against real AWS via the runner. Packaging fix: add Conformance/Directory.Build.props supplying the SDK's runtime NuGet dependencies (AWSSDK.Lambda, Microsoft.Extensions.Logging.Abstractions) to every handler, since those transitive package assets do not flow into a handler's net8.0 framework-dependent publish (WrapAsyncCore threw FileNotFoundException at runtime). Drop the now-redundant per-handler AWSSDK.Lambda references. * feat(DurableExecution): round-trip step/child results through the serializer on fresh success On a fresh (non-replay) success, StepOperation and ChildContextOperation now return the value deserialized from the just-written checkpoint instead of the original in-memory object, matching replay semantics. This makes a custom per-operation serializer's transform observable in the operation result on the first execution, not only on replay (a non-round-tripping serializer previously had no effect on the fresh result). Overflow (replay-children) child results are unaffected — the payload is stripped and the value is recovered by re-execution. Behavior change documented in the AutoVer change file. * test(DurableExecution): real per-op serdes conformance handlers for step 1-6 and child 3-14 Refactor StepCustomSerdes (1-6) to use a real StepConfig.Serializer (uppercase-on-serialize) instead of transforming inside the step body, and add child/ChildCustomSerdes (3-14) using ChildContextConfig.Serializer. Both rely on the fresh-success round-trip so the serialize-side transform reaches the result. Remove the 3-14 NotImplemented and wire the resource in template_child.yaml. Both pass against real AWS via the runner. * address review: harden conformance serde handlers - Custom serializers now guard typeof(T) == typeof(string) and throw a clear NotSupportedException for unsupported result types (instead of an opaque InvalidCastException from the (T)(object) cast). - Read with an explicit UTF-8 StreamReader for symmetry with the UTF-8 bytes written. - MapCustomSerdes: drop the async-without-await iteration lambda (CS1998); return Task.FromResult(...) so no per-item async state machine is allocated. - InvokeCustomResultSerdes: correct the header comment — the serializer serializes the outbound request payload normally on the initial execution and applies the uppercase transform on deserialize (which, for a chained invoke, happens on replay). The handlers still transform the raw serialized payload, which is the behavior the conformance requirements assert (e.g. invoke 5-16 ExpectedResult '"HELLO"'); the string path is byte-identical, so the suite results are unchanged. * fix(DurableExecution): mark fresh-success round-trip change Major The fresh-success serializer round-trip changes the observable return value (fresh deserialized instance instead of the object the body produced, and a non-round-tripping serializer's transform now visible on the first run). At the released 1.0.0 baseline this is a breaking change and must be Major, not Minor. * chore(DurableExecution): normalize 12a4a1f7 change file (strip trailing newline) to match canonical content --- .../12a4a1f7-d59f-4544-aa1f-6db30289485e.json | 11 + .../45498f15-84f7-47cc-a871-47cdb059a0c4.json | 11 + .../7ed69083-0a98-424b-8d30-3d70c916c2a1.json | 11 + .../CallbackConfig.cs | 11 + .../ChildContextConfig.cs | 9 + .../DurableContext.cs | 26 +- .../Internal/ChildContextOperation.cs | 13 + .../Internal/StepOperation.cs | 12 +- .../InvokeConfig.cs | 19 +- .../MapConfig.cs | 27 +- .../ParallelConfig.cs | 22 +- .../StepConfig.cs | 17 + .../WaitForConditionConfig.cs | 12 + .../docs/core/callbacks.md | 4 + .../docs/core/child-contexts.md | 4 + .../docs/core/parallel.md | 4 + .../docs/core/steps.md | 21 +- .../docs/core/wait-for-condition.md | 4 + .../Conformance/Directory.Build.props | 19 + .../ChildCustomSerdes.csproj | 19 + .../child/ChildCustomSerdes/Function.cs | 77 +++ .../invoke/InvokeBasic/InvokeBasic.csproj | 1 - .../InvokeComplexObject.csproj | 1 - .../InvokeCustomPayloadSerdes.csproj | 1 - .../InvokeCustomResultSerdes/Function.cs | 66 +++ .../InvokeCustomResultSerdes.csproj | 19 + .../InvokeInChildContext.csproj | 1 - .../InvokeLargePayload.csproj | 1 - .../invoke/InvokeNull/InvokeNull.csproj | 1 - .../InvokeReplayRethrows.csproj | 1 - .../InvokeReplaySkips.csproj | 1 - .../InvokeSequential/InvokeSequential.csproj | 1 - .../InvokeTargetFails.csproj | 1 - .../InvokeTargetFailsCaught.csproj | 1 - .../InvokeThenStep/InvokeThenStep.csproj | 1 - .../InvokeWithName/InvokeWithName.csproj | 1 - .../InvokeWithTenantId.csproj | 1 - .../StepThenInvoke/StepThenInvoke.csproj | 1 - .../map/MapCustomSerdes/Function.cs | 75 +++ .../MapCustomSerdes/MapCustomSerdes.csproj | 19 + .../parallel/ParallelCustomSerdes/Function.cs | 76 +++ .../ParallelCustomSerdes.csproj | 19 + .../step/StepCustomSerdes/Function.cs | 41 +- .../Conformance/template_child.yaml | 21 +- .../Conformance/template_invoke.yaml | 25 +- .../Conformance/template_map.yaml | 29 +- .../Conformance/template_parallel.yaml | 21 +- .../MapParallelItemSerializerTest.cs | 60 +++ .../PerOperationSerializerTest.cs | 62 +++ .../Function.cs | 68 +++ .../MapParallelItemSerializerFunction.csproj | 18 + .../Function.cs | 59 +++ .../PerOperationSerializerFunction.csproj | 18 + .../PerOperationSerializerTests.cs | 458 ++++++++++++++++++ 54 files changed, 1458 insertions(+), 64 deletions(-) create mode 100644 .autover/changes/12a4a1f7-d59f-4544-aa1f-6db30289485e.json create mode 100644 .autover/changes/45498f15-84f7-47cc-a871-47cdb059a0c4.json create mode 100644 .autover/changes/7ed69083-0a98-424b-8d30-3d70c916c2a1.json create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/Directory.Build.props create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/child/ChildCustomSerdes/ChildCustomSerdes.csproj create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/child/ChildCustomSerdes/Function.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomResultSerdes/Function.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomResultSerdes/InvokeCustomResultSerdes.csproj create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/map/MapCustomSerdes/Function.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/map/MapCustomSerdes/MapCustomSerdes.csproj create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/parallel/ParallelCustomSerdes/Function.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/parallel/ParallelCustomSerdes/ParallelCustomSerdes.csproj create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/MapParallelItemSerializerTest.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/PerOperationSerializerTest.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/MapParallelItemSerializerFunction/Function.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/MapParallelItemSerializerFunction/MapParallelItemSerializerFunction.csproj create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/PerOperationSerializerFunction/Function.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/PerOperationSerializerFunction/PerOperationSerializerFunction.csproj create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.Tests/PerOperationSerializerTests.cs diff --git a/.autover/changes/12a4a1f7-d59f-4544-aa1f-6db30289485e.json b/.autover/changes/12a4a1f7-d59f-4544-aa1f-6db30289485e.json new file mode 100644 index 000000000..1dca7e54c --- /dev/null +++ b/.autover/changes/12a4a1f7-d59f-4544-aa1f-6db30289485e.json @@ -0,0 +1,11 @@ +{ + "Projects": [ + { + "Name": "Amazon.Lambda.DurableExecution", + "Type": "Major", + "ChangelogMessages": [ + "Behavior change: steps and child contexts now round-trip their result through the configured serializer on a fresh (non-replay) success, deserializing the just-written checkpoint before returning \u2014 matching replay semantics. A custom per-operation serializer\u0027s transform is now reflected in the operation result on the first execution, not only on replay. The returned value is a fresh deserialized instance rather than the exact object the step/child body produced. Overflow (replay-children) results are unaffected." + ] + } + ] +} \ No newline at end of file diff --git a/.autover/changes/45498f15-84f7-47cc-a871-47cdb059a0c4.json b/.autover/changes/45498f15-84f7-47cc-a871-47cdb059a0c4.json new file mode 100644 index 000000000..a30b2ac38 --- /dev/null +++ b/.autover/changes/45498f15-84f7-47cc-a871-47cdb059a0c4.json @@ -0,0 +1,11 @@ +{ + "Projects": [ + { + "Name": "Amazon.Lambda.DurableExecution", + "Type": "Minor", + "ChangelogMessages": [ + "Add optional per-item serializer override (ItemSerializer) on MapConfig and ParallelConfig; when null the globally-registered ILambdaSerializer is used for item/branch results." + ] + } + ] +} \ No newline at end of file diff --git a/.autover/changes/7ed69083-0a98-424b-8d30-3d70c916c2a1.json b/.autover/changes/7ed69083-0a98-424b-8d30-3d70c916c2a1.json new file mode 100644 index 000000000..8fa4b02c6 --- /dev/null +++ b/.autover/changes/7ed69083-0a98-424b-8d30-3d70c916c2a1.json @@ -0,0 +1,11 @@ +{ + "Projects": [ + { + "Name": "Amazon.Lambda.DurableExecution", + "Type": "Minor", + "ChangelogMessages": [ + "Add optional per-operation serializer override (Serializer) on StepConfig, CallbackConfig, InvokeConfig, WaitForConditionConfig, and ChildContextConfig; when null the globally-registered ILambdaSerializer is used." + ] + } + ] +} \ No newline at end of file diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/CallbackConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/CallbackConfig.cs index e565ddb06..29d322578 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/CallbackConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/CallbackConfig.cs @@ -1,6 +1,8 @@ // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. // SPDX-License-Identifier: Apache-2.0 +using Amazon.Lambda.Core; + namespace Amazon.Lambda.DurableExecution; /// @@ -60,6 +62,15 @@ public TimeSpan HeartbeatTimeout } } + /// + /// Optional serializer used to deserialize the callback payload delivered by the + /// external system. When null (default), the globally-registered + /// on is used. + /// Only the deserialize path is used for callbacks — the SDK never serializes a callback + /// result (the external system provides the payload). + /// + public ILambdaSerializer? Serializer { get; set; } + private static void ValidateTimeout(TimeSpan value, string paramName) { // Allow Zero (means "not set"); reject negative; reject sub-second diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/ChildContextConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/ChildContextConfig.cs index 3a5bc589e..608f93a43 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/ChildContextConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/ChildContextConfig.cs @@ -1,6 +1,8 @@ // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. // SPDX-License-Identifier: Apache-2.0 +using Amazon.Lambda.Core; + namespace Amazon.Lambda.DurableExecution; /// @@ -54,4 +56,11 @@ public sealed class ChildContextConfig /// /// public NestingType NestingType { get; set; } = NestingType.Nested; + + /// + /// Optional serializer for this child context's result payload. When null + /// (default), the globally-registered on + /// is used. + /// + public ILambdaSerializer? Serializer { get; set; } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/DurableContext.cs b/Libraries/src/Amazon.Lambda.DurableExecution/DurableContext.cs index ca692f9d3..1144c6bf1 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/DurableContext.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/DurableContext.cs @@ -84,7 +84,7 @@ private Task RunStep( StepConfig? config, CancellationToken cancellationToken) { - var serializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = config?.Serializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); var operationId = _idGenerator.NextId(); var op = new StepOperation( @@ -147,7 +147,7 @@ public Task WaitForConditionAsync( ArgumentNullException.ThrowIfNull(config); ArgumentNullException.ThrowIfNull(config.WaitStrategy); - var serializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = config.Serializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); var operationId = _idGenerator.NextId(); var op = new WaitForConditionOperation( operationId, name, _idGenerator.ParentId, check, config, serializer, Logger, @@ -161,7 +161,7 @@ private Task RunChildContext( ChildContextConfig? config, CancellationToken cancellationToken) { - var serializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = config?.Serializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); var operationId = _idGenerator.NextId(); @@ -183,7 +183,7 @@ private Task> RunCallback( CallbackConfig? config, CancellationToken cancellationToken) { - var serializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = config?.Serializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); var operationId = _idGenerator.NextId(); var op = new CallbackOperation( @@ -242,11 +242,11 @@ private Task> RunParallel( var effectiveConfig = config ?? new ParallelConfig(); - var serializer = LambdaContext.Serializer - ?? throw new InvalidOperationException( - "No ILambdaSerializer is registered on ILambdaContext.Serializer. " + - "Register a serializer via LambdaBootstrapBuilder.Create(handler, serializer) " + - "(or in tests, set TestLambdaContext.Serializer)."); + // Per-branch result serialization: the config's ItemSerializer if set, else the + // globally-registered serializer. This is the only serializer ConcurrentOperation + // uses (per-unit child results + inline summary results); the aggregate batch + // envelope is a source-generated structure and is unaffected. + var serializer = effectiveConfig.ItemSerializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); var operationId = _idGenerator.NextId(); var op = new Internal.ParallelOperation( @@ -275,7 +275,11 @@ private Task> RunMap( var effectiveConfig = config ?? new MapConfig(); - var serializer = LambdaSerializerHelper.GetRequired(LambdaContext); + // Per-item result serialization: the config's ItemSerializer if set, else the + // globally-registered serializer. This is the only serializer ConcurrentOperation + // uses (per-unit child results + inline summary results); the aggregate batch + // envelope is a source-generated structure and is unaffected. + var serializer = effectiveConfig.ItemSerializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); var operationId = _idGenerator.NextId(); var op = new Internal.MapOperation( @@ -495,7 +499,7 @@ private Task RunInvoke( if (string.IsNullOrWhiteSpace(functionName)) throw new ArgumentException("Function name must not be empty or whitespace.", nameof(functionName)); - var serializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = config?.Serializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); cancellationToken.ThrowIfCancellationRequested(); diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs index 5dd4671b1..2cef36626 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs @@ -253,6 +253,19 @@ await EnqueueAsync(new SdkOperationUpdate ? new SdkContextOptions { ReplayChildren = true } : null }, cancellationToken); + + // Non-overflow: round-trip the just-written checkpoint so the value the + // workflow observes on this fresh execution matches replay (where the + // result is always deserialized from the checkpoint). This makes a + // custom (possibly non-round-tripping) ChildContextConfig.Serializer's + // transform visible in the child-context result on the first run. + // Behavior change: the returned object is no longer the same instance + // the child body produced. See the AutoVer change note. Overflow skips + // this (the payload was stripped; the value is recovered by replay). + if (!overflow) + { + return DeserializeResult(serialized); + } } return result; diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/StepOperation.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/StepOperation.cs index 76780957a..bb580fbfd 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/StepOperation.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/StepOperation.cs @@ -234,6 +234,7 @@ private async Task ExecuteFunc(int attemptNumber, CancellationToken cancellat result = await _func(stepContext, linked.Token); } + var serialized = SerializeResult(result); await EnqueueAsync(new SdkOperationUpdate { Id = OperationId, @@ -242,10 +243,17 @@ await EnqueueAsync(new SdkOperationUpdate Action = OperationAction.SUCCEED, SubType = OperationSubTypes.Step, Name = Name, - Payload = SerializeResult(result) + Payload = serialized }, cancellationToken); - return result; + // Round-trip the just-written checkpoint so the value the workflow + // observes on this fresh execution is the deserialized-from-checkpoint + // value, exactly as it would be on replay. This makes a custom + // (possibly non-round-tripping) StepConfig.Serializer's transform + // visible in the step result on the first run, not just on replay. + // Behavior change: the returned object is no longer the same instance + // the step body produced. See the AutoVer change note. + return DeserializeResult(serialized); } catch (OperationCanceledException) when (linked.IsCancellationRequested) { diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/InvokeConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/InvokeConfig.cs index b58f810a1..356219b6b 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/InvokeConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/InvokeConfig.cs @@ -1,6 +1,8 @@ // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. // SPDX-License-Identifier: Apache-2.0 +using Amazon.Lambda.Core; + namespace Amazon.Lambda.DurableExecution; /// @@ -11,9 +13,8 @@ namespace Amazon.Lambda.DurableExecution; /// to configure a single chained invocation. Payload/result serialization is /// performed by the registered on /// (typically configured via -/// LambdaBootstrapBuilder.Create(handler, serializer)); there are -/// intentionally no serializer fields here, matching the pattern established -/// by . +/// LambdaBootstrapBuilder.Create(handler, serializer)), unless overridden for this +/// operation via . /// public sealed class InvokeConfig { @@ -24,4 +25,16 @@ public sealed class InvokeConfig /// Python, JavaScript, and Java SDKs. /// public string? TenantId { get; set; } + + /// + /// Optional serializer for this invoke's payload and result. When null + /// (default), the globally-registered on + /// is used. + /// + /// + /// The chained (callee) function serializes and deserializes with its own registered + /// serializer, so an override here must produce a form the callee can read (and read a + /// form the callee produces). Prefer overriding only when both sides agree on the format. + /// + public ILambdaSerializer? Serializer { get; set; } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/MapConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/MapConfig.cs index 9a58ea489..9080e656c 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/MapConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/MapConfig.cs @@ -1,3 +1,8 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using Amazon.Lambda.Core; + namespace Amazon.Lambda.DurableExecution; /// @@ -9,11 +14,12 @@ namespace Amazon.Lambda.DurableExecution; /// receives the strongly-typed item rather than object. /// /// -/// Per-item checkpoint payloads are serialized via the -/// registered on -/// (typically -/// configured via LambdaBootstrapBuilder.Create(handler, serializer)); -/// this config does not expose a serializer slot. +/// Per-item result payloads are serialized via the +/// registered on +/// (typically configured via +/// LambdaBootstrapBuilder.Create(handler, serializer)), unless overridden per +/// operation via . The aggregated batch envelope (per-item +/// statuses and completion reason) is SDK-internal and is not user-serialized. /// public sealed class MapConfig { @@ -77,4 +83,15 @@ public int? MaxConcurrency /// named by index ("0", "1", ...). /// public Func? ItemNamer { get; set; } + + /// + /// Optional serializer for each item's result payload. When null + /// (default), item results are serialized with the + /// registered on . This controls only the + /// per-item result — both the inline copy on the map's checkpoint and each nested + /// item's own checkpoint. It does not change the aggregated batch envelope + /// (statuses / completion reason), and durable operations inside an item's body use + /// their own configuration. + /// + public ILambdaSerializer? ItemSerializer { get; set; } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/ParallelConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/ParallelConfig.cs index 69b82a3bc..1fec4216f 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/ParallelConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/ParallelConfig.cs @@ -1,6 +1,8 @@ // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. // SPDX-License-Identifier: Apache-2.0 +using Amazon.Lambda.Core; + namespace Amazon.Lambda.DurableExecution; /// @@ -8,11 +10,12 @@ namespace Amazon.Lambda.DurableExecution; /// . /// /// -/// Per-branch checkpoint payloads are serialized via the -/// registered on -/// (typically -/// configured via LambdaBootstrapBuilder.Create(handler, serializer)); -/// this config does not expose a serializer slot. +/// Per-branch result payloads are serialized via the +/// registered on +/// (typically configured via +/// LambdaBootstrapBuilder.Create(handler, serializer)), unless overridden per +/// operation via . The aggregated batch envelope (per-branch +/// statuses and completion reason) is SDK-internal and is not user-serialized. /// public sealed class ParallelConfig { @@ -65,4 +68,13 @@ public int? MaxConcurrency /// payload instead. /// public NestingType NestingType { get; set; } = NestingType.Nested; + + /// + /// Optional serializer for each branch's result payload. When null + /// (default), branch results are serialized with the + /// registered on . This controls only the + /// per-branch result — not the aggregated batch envelope (statuses / completion + /// reason) — and durable operations inside a branch use their own configuration. + /// + public ILambdaSerializer? ItemSerializer { get; set; } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/StepConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/StepConfig.cs index eea3dc791..6b6eeb604 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/StepConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/StepConfig.cs @@ -1,6 +1,8 @@ // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. // SPDX-License-Identifier: Apache-2.0 +using Amazon.Lambda.Core; + namespace Amazon.Lambda.DurableExecution; /// @@ -18,4 +20,19 @@ public sealed class StepConfig /// Default is . /// public StepSemantics Semantics { get; set; } = StepSemantics.AtLeastOncePerRetry; + + /// + /// Optional serializer for this step's result payload. When null (default), + /// the globally-registered on + /// is used. Set this to override how this + /// step's result is serialized to the checkpoint and deserialized on replay, + /// without affecting other operations or the handler's input/return value. + /// + /// + /// The serializer is part of the workflow's deterministic definition: it is + /// re-resolved on every replay, so a step must be able to deserialize a result it + /// previously serialized. Changing a step's serializer for an in-flight execution in + /// a way that cannot read the stored payload will break replay. + /// + public ILambdaSerializer? Serializer { get; set; } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/WaitForConditionConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/WaitForConditionConfig.cs index ea99a76ef..158b61c6d 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/WaitForConditionConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/WaitForConditionConfig.cs @@ -1,3 +1,8 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using Amazon.Lambda.Core; + namespace Amazon.Lambda.DurableExecution; /// @@ -26,4 +31,11 @@ public sealed class WaitForConditionConfig /// polling and how long to wait before the next attempt. /// public required IWaitStrategy WaitStrategy { get; set; } + + /// + /// Optional serializer for this operation's checkpointed state. When null + /// (default), the globally-registered on + /// is used. + /// + public ILambdaSerializer? Serializer { get; set; } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/callbacks.md b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/callbacks.md index 00aee3cd4..25fb9caaf 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/callbacks.md +++ b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/callbacks.md @@ -183,3 +183,7 @@ public class WaitForCallbackConfig : CallbackConfig public IRetryStrategy? RetryStrategy { get; set; } // applied to the submitter step only } ``` + +## Custom serializer + +Set `CallbackConfig.Serializer` to override the `ILambdaSerializer` used to **deserialize** the callback payload delivered by the external system. When `null` (default), the globally-registered serializer on `ILambdaContext.Serializer` is used. Only the deserialize path is used for callbacks — the SDK never serializes a callback result. See [Steps → Custom serializer](steps.md#custom-serializer) for details and replay/AOT notes. diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/child-contexts.md b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/child-contexts.md index 34904f290..723e87745 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/child-contexts.md +++ b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/child-contexts.md @@ -46,3 +46,7 @@ public sealed class ChildContextConfig ``` `ErrorMapping` lets you translate exceptions thrown inside the child context into a domain-specific exception type before they propagate to the parent. + +## Custom serializer + +Set `ChildContextConfig.Serializer` to override the `ILambdaSerializer` used to serialize and (on replay) deserialize the child context's result. When `null` (default), the globally-registered serializer on `ILambdaContext.Serializer` is used. See [Steps → Custom serializer](steps.md#custom-serializer) for details and replay/AOT notes. diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/parallel.md b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/parallel.md index 317a907fd..7a258717a 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/parallel.md +++ b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/parallel.md @@ -135,3 +135,7 @@ if (batch.HasFailure) batch.ThrowIfError(); // rethrow the first branch's DurableExecutionException, if desired } ``` + +## Custom serializer + +Set `ParallelConfig.ItemSerializer` (and, for maps, `MapConfig.ItemSerializer`) to serialize each branch/item **result** with a specific `ILambdaSerializer`. When `null` (default), the globally-registered serializer on `ILambdaContext.Serializer` is used. This controls only the per-branch/per-item result — both the inline copy on the operation's checkpoint and each nested unit's own checkpoint. It does **not** change the aggregated `IBatchResult` envelope (per-unit statuses and completion reason), which is an SDK-internal, source-generated structure; and durable operations inside a branch/item body use their own configuration. See [Steps → Custom serializer](steps.md#custom-serializer) for replay/AOT notes. diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/steps.md b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/steps.md index cb6fa48a2..0eb07c551 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/steps.md +++ b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/steps.md @@ -18,7 +18,7 @@ Task StepAsync( CancellationToken cancellationToken = default); ``` -The `IStepContext` parameter exposes the current `AttemptNumber`, the deterministic `OperationId`, and a scoped `Logger`. The `CancellationToken` parameter is a linked token combining the caller-supplied token with the SDK's workflow-shutdown signal — pass it to cancellation-aware APIs (`HttpClient.SendAsync`, `Task.Delay`, AWS SDK calls) so the step body unwinds cleanly when the workflow is being torn down. Returned values are serialized via the `ILambdaSerializer` registered on `ILambdaContext.Serializer`. +The `IStepContext` parameter exposes the current `AttemptNumber`, the deterministic `OperationId`, and a scoped `Logger`. The `CancellationToken` parameter is a linked token combining the caller-supplied token with the SDK's workflow-shutdown signal — pass it to cancellation-aware APIs (`HttpClient.SendAsync`, `Task.Delay`, AWS SDK calls) so the step body unwinds cleanly when the workflow is being torn down. Returned values are serialized via the `ILambdaSerializer` registered on `ILambdaContext.Serializer`, unless a per-step serializer is supplied on `StepConfig.Serializer` (see [Custom serializer](#custom-serializer)). ## Basic step @@ -45,6 +45,7 @@ public sealed class StepConfig { public IRetryStrategy? RetryStrategy { get; set; } // null = no retry public StepSemantics Semantics { get; set; } = StepSemantics.AtLeastOncePerRetry; + public ILambdaSerializer? Serializer { get; set; } // null = global serializer } ``` @@ -146,3 +147,21 @@ var result = await ctx.StepAsync( RetryStrategy = RetryStrategy.None }); ``` + +### Custom serializer + +By default a step's result is serialized to the checkpoint (and deserialized on replay) with the `ILambdaSerializer` registered on `ILambdaContext.Serializer`. Set `StepConfig.Serializer` to override this for a single step — useful when one step's result needs different `JsonSerializerOptions`, a different `JsonSerializerContext`, or a different naming policy than the rest of the workflow. Other steps, the workflow input, and the handler's return value are unaffected. + +```csharp +var report = await ctx.StepAsync( + async (_, ct) => await BuildReportAsync(ct), + name: "report", + config: new StepConfig + { + Serializer = new SourceGeneratorLambdaJsonSerializer() + }); +``` + +The serializer is part of the workflow's deterministic definition: it is re-resolved on every replay, so a step must be able to deserialize a result it previously serialized. Changing a step's serializer for an in-flight execution in a way that cannot read the stored payload will break replay. The override is Native-AOT safe as long as the serializer you supply is (for example `SourceGeneratorLambdaJsonSerializer`). + +> The same optional `Serializer` property is available on `CallbackConfig` (used to deserialize the callback payload), `InvokeConfig` (payload and result), `WaitForConditionConfig` (the checkpointed state), and `ChildContextConfig` (the child result). `MapConfig` and `ParallelConfig` expose `ItemSerializer` for each item/branch **result** (the aggregated batch envelope — per-item statuses and completion reason — is SDK-internal and not user-serialized). In every case, `null` means the globally-registered serializer is used. diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/wait-for-condition.md b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/wait-for-condition.md index b900b8ffe..c25e8e288 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/wait-for-condition.md +++ b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/wait-for-condition.md @@ -114,3 +114,7 @@ catch (WaitForConditionException ex) ctx.Logger.LogWarning("Gave up after {Attempts} polls; last status was {Status}", attempts, last); } ``` + +## Custom serializer + +Set `WaitForConditionConfig.Serializer` to override the `ILambdaSerializer` used to serialize and (on replay) deserialize the checkpointed `TState`. When `null` (default), the globally-registered serializer on `ILambdaContext.Serializer` is used. See [Steps → Custom serializer](steps.md#custom-serializer) for details and replay/AOT notes. diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/Directory.Build.props b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/Directory.Build.props new file mode 100644 index 000000000..09b9cb0db --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/Directory.Build.props @@ -0,0 +1,19 @@ + + + + + + + + + diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/child/ChildCustomSerdes/ChildCustomSerdes.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/child/ChildCustomSerdes/ChildCustomSerdes.csproj new file mode 100644 index 000000000..4b354e178 --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/child/ChildCustomSerdes/ChildCustomSerdes.csproj @@ -0,0 +1,19 @@ + + + + net8.0 + Exe + true + bootstrap + enable + enable + + + + + + + + + + diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/child/ChildCustomSerdes/Function.cs b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/child/ChildCustomSerdes/Function.cs new file mode 100644 index 000000000..d32c1248b --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/child/ChildCustomSerdes/Function.cs @@ -0,0 +1,77 @@ +// 3-14: Child context with custom serdes (succeed) +// A child context whose custom serializer uppercases the child result on serialize. +// The inner step returns the input via the global serializer (so its checkpoint stays +// "hello child"); the child context serializes its result with the custom serdes +// (raw uppercased "HELLO CHILD"). Thanks to the SDK's fresh-success round-trip, the +// child context returns the uppercased value, so the execution result is uppercased. +using System.IO; +using System.Text; +using Amazon.Lambda.Core; +using Amazon.Lambda.DurableExecution; +using Amazon.Lambda.RuntimeSupport; +using Amazon.Lambda.Serialization.SystemTextJson; + +namespace ChildCustomSerdes; + +public class Function +{ + public static async Task Main(string[] args) + { + var handler = new Function(); + var serializer = new DefaultLambdaJsonSerializer(); + using var handlerWrapper = HandlerWrapper.GetHandlerWrapper(handler.Handler, serializer); + using var bootstrap = new LambdaBootstrap(handlerWrapper); + await bootstrap.RunAsync(); + } + + public Task Handler( + DurableExecutionInvocationInput input, ILambdaContext context) + => DurableFunction.WrapAsync(Workflow, input, context); + + private async Task Workflow(string input, IDurableContext context) + { + var result = await context.RunInChildContextAsync( + async (childContext, _ct) => + { + var stepResult = await childContext.StepAsync( + async (_, _s) => + { + await Task.CompletedTask; + return input; + }); + + return stepResult; + }, + name: "child-serdes", + config: new ChildContextConfig + { + SubType = "RunInChildContext", + Serializer = new UppercaseSerializer(), + }); + + return result; + } +} + +/// +/// A custom serializer that uppercases the result on serialize (writing the raw +/// transformed text) and reads it back verbatim on deserialize. +/// +public sealed class UppercaseSerializer : ILambdaSerializer +{ + public T Deserialize(Stream requestStream) + { + if (typeof(T) != typeof(string)) + throw new NotSupportedException( + $"{nameof(UppercaseSerializer)} only supports string results; got {typeof(T)}."); + using var reader = new StreamReader(requestStream, Encoding.UTF8); + return (T)(object)reader.ReadToEnd(); + } + + public void Serialize(T response, Stream responseStream) + { + var value = (response as string ?? response?.ToString() ?? string.Empty).ToUpperInvariant(); + var bytes = Encoding.UTF8.GetBytes(value); + responseStream.Write(bytes, 0, bytes.Length); + } +} diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeBasic/InvokeBasic.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeBasic/InvokeBasic.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeBasic/InvokeBasic.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeBasic/InvokeBasic.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeComplexObject/InvokeComplexObject.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeComplexObject/InvokeComplexObject.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeComplexObject/InvokeComplexObject.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeComplexObject/InvokeComplexObject.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomPayloadSerdes/InvokeCustomPayloadSerdes.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomPayloadSerdes/InvokeCustomPayloadSerdes.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomPayloadSerdes/InvokeCustomPayloadSerdes.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomPayloadSerdes/InvokeCustomPayloadSerdes.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomResultSerdes/Function.cs b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomResultSerdes/Function.cs new file mode 100644 index 000000000..70fe3470a --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomResultSerdes/Function.cs @@ -0,0 +1,66 @@ +// 5-16: Invoke with custom result serdes (custom deserializer for the returned result) +// The custom serializer serializes the OUTBOUND request payload normally on the initial +// execution (its Serialize delegates to the default JSON serializer, so the checkpointed +// input and the ChainedInvokeSucceeded result both stay "hello"). It uppercases the value +// when the result is deserialized — which, for a chained invoke, happens on the replay +// that observes the completed invocation — so the workflow returns the uppercased "HELLO". +using System.IO; +using System.Text; +using Amazon.Lambda.Core; +using Amazon.Lambda.DurableExecution; +using Amazon.Lambda.RuntimeSupport; +using Amazon.Lambda.Serialization.SystemTextJson; + +namespace InvokeCustomResultSerdes; + +public class Function +{ + public static async Task Main(string[] args) + { + var handler = new Function(); + var serializer = new DefaultLambdaJsonSerializer(); + using var handlerWrapper = HandlerWrapper.GetHandlerWrapper(handler.Handler, serializer); + using var bootstrap = new LambdaBootstrap(handlerWrapper); + await bootstrap.RunAsync(); + } + + public Task Handler( + DurableExecutionInvocationInput input, ILambdaContext context) + => DurableFunction.WrapAsync(Workflow, input, context); + + private async Task Workflow(string input, IDurableContext context) + { + var targetFunctionName = Environment.GetEnvironmentVariable("TARGET_FUNCTION_NAME")!; + + var result = await context.InvokeAsync( + targetFunctionName, + input, + config: new InvokeConfig { Serializer = new UppercaseResultSerializer() }); + + return result; + } +} + +/// +/// Custom result serializer: serializes the outgoing payload normally, but uppercases +/// the raw serialized result on deserialize (the "custom result serdes" applied to the +/// chained invoke's serialized payload — e.g. "hello" becomes "HELLO", +/// so the JSON-decoded execution result is the string "HELLO"). +/// +public sealed class UppercaseResultSerializer : ILambdaSerializer +{ + private readonly DefaultLambdaJsonSerializer _inner = new(); + + public T Deserialize(Stream requestStream) + { + if (typeof(T) != typeof(string)) + throw new NotSupportedException( + $"{nameof(UppercaseResultSerializer)} only supports string results; got {typeof(T)}."); + using var reader = new StreamReader(requestStream, Encoding.UTF8); + var raw = reader.ReadToEnd(); + return (T)(object)raw.ToUpperInvariant(); + } + + public void Serialize(T response, Stream responseStream) + => _inner.Serialize(response, responseStream); +} diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomResultSerdes/InvokeCustomResultSerdes.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomResultSerdes/InvokeCustomResultSerdes.csproj new file mode 100644 index 000000000..4b354e178 --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeCustomResultSerdes/InvokeCustomResultSerdes.csproj @@ -0,0 +1,19 @@ + + + + net8.0 + Exe + true + bootstrap + enable + enable + + + + + + + + + + diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeInChildContext/InvokeInChildContext.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeInChildContext/InvokeInChildContext.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeInChildContext/InvokeInChildContext.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeInChildContext/InvokeInChildContext.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeLargePayload/InvokeLargePayload.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeLargePayload/InvokeLargePayload.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeLargePayload/InvokeLargePayload.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeLargePayload/InvokeLargePayload.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeNull/InvokeNull.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeNull/InvokeNull.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeNull/InvokeNull.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeNull/InvokeNull.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeReplayRethrows/InvokeReplayRethrows.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeReplayRethrows/InvokeReplayRethrows.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeReplayRethrows/InvokeReplayRethrows.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeReplayRethrows/InvokeReplayRethrows.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeReplaySkips/InvokeReplaySkips.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeReplaySkips/InvokeReplaySkips.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeReplaySkips/InvokeReplaySkips.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeReplaySkips/InvokeReplaySkips.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeSequential/InvokeSequential.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeSequential/InvokeSequential.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeSequential/InvokeSequential.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeSequential/InvokeSequential.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeTargetFails/InvokeTargetFails.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeTargetFails/InvokeTargetFails.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeTargetFails/InvokeTargetFails.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeTargetFails/InvokeTargetFails.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeTargetFailsCaught/InvokeTargetFailsCaught.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeTargetFailsCaught/InvokeTargetFailsCaught.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeTargetFailsCaught/InvokeTargetFailsCaught.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeTargetFailsCaught/InvokeTargetFailsCaught.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeThenStep/InvokeThenStep.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeThenStep/InvokeThenStep.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeThenStep/InvokeThenStep.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeThenStep/InvokeThenStep.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeWithName/InvokeWithName.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeWithName/InvokeWithName.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeWithName/InvokeWithName.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeWithName/InvokeWithName.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeWithTenantId/InvokeWithTenantId.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeWithTenantId/InvokeWithTenantId.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeWithTenantId/InvokeWithTenantId.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/InvokeWithTenantId/InvokeWithTenantId.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/StepThenInvoke/StepThenInvoke.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/StepThenInvoke/StepThenInvoke.csproj index fede088c4..4b354e178 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/StepThenInvoke/StepThenInvoke.csproj +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/invoke/StepThenInvoke/StepThenInvoke.csproj @@ -14,7 +14,6 @@ - diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/map/MapCustomSerdes/Function.cs b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/map/MapCustomSerdes/Function.cs new file mode 100644 index 000000000..80d77927e --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/map/MapCustomSerdes/Function.cs @@ -0,0 +1,75 @@ +// 9-14: Map with a custom per-item serdes (ItemSerializer) +// Each iteration returns the uppercased item directly; the custom per-item +// serializer wraps the value as `wrapped:` on serialize (the checkpointed +// iteration payload) and unwraps it on deserialize, so the ordered results +// survive the round-trip. MaxConcurrency=1 gives a deterministic history. +using System.IO; +using System.Text; +using Amazon.Lambda.Core; +using Amazon.Lambda.DurableExecution; +using Amazon.Lambda.RuntimeSupport; +using Amazon.Lambda.Serialization.SystemTextJson; + +namespace MapCustomSerdes; + +public class Function +{ + public static async Task Main(string[] args) + { + var handler = new Function(); + var serializer = new DefaultLambdaJsonSerializer(); + using var handlerWrapper = HandlerWrapper.GetHandlerWrapper(handler.Handler, serializer); + using var bootstrap = new LambdaBootstrap(handlerWrapper); + await bootstrap.RunAsync(); + } + + public Task Handler( + DurableExecutionInvocationInput input, ILambdaContext context) + => DurableFunction.WrapAsync>(Workflow, input, context); + + private async Task> Workflow(object? input, IDurableContext context) + { + var result = await context.MapAsync( + new List { "x", "y" }, + (ctx, item, index, all, ct) => Task.FromResult(item.ToUpperInvariant()), + name: "serdes", + config: new MapConfig + { + MaxConcurrency = 1, + ItemSerializer = new WrapSerializer(), + }); + + return result.GetResults().ToList(); + } +} + +/// +/// A real (non-identity) per-item serializer: serializes a string value v +/// as the raw text wrapped:v and deserializes wrapped:v back to +/// v. Used only for per-item results, so it only handles strings. +/// +public sealed class WrapSerializer : ILambdaSerializer +{ + private const string Prefix = "wrapped:"; + + public T Deserialize(Stream requestStream) + { + if (typeof(T) != typeof(string)) + throw new NotSupportedException( + $"{nameof(WrapSerializer)} only supports string results; got {typeof(T)}."); + using var reader = new StreamReader(requestStream, Encoding.UTF8); + var text = reader.ReadToEnd(); + if (text.StartsWith(Prefix, StringComparison.Ordinal)) + { + text = text.Substring(Prefix.Length); + } + return (T)(object)text; + } + + public void Serialize(T response, Stream responseStream) + { + var text = response as string ?? response?.ToString() ?? string.Empty; + var bytes = Encoding.UTF8.GetBytes(Prefix + text); + responseStream.Write(bytes, 0, bytes.Length); + } +} diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/map/MapCustomSerdes/MapCustomSerdes.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/map/MapCustomSerdes/MapCustomSerdes.csproj new file mode 100644 index 000000000..4b354e178 --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/map/MapCustomSerdes/MapCustomSerdes.csproj @@ -0,0 +1,19 @@ + + + + net8.0 + Exe + true + bootstrap + enable + enable + + + + + + + + + + diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/parallel/ParallelCustomSerdes/Function.cs b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/parallel/ParallelCustomSerdes/Function.cs new file mode 100644 index 000000000..4a7e455d6 --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/parallel/ParallelCustomSerdes/Function.cs @@ -0,0 +1,76 @@ +// 8-15: Parallel with a custom per-branch serde (ItemSerializer) +// Each branch returns a string directly ("x", "y"); the custom per-branch serializer +// wraps each value as the envelope {"wrapped": v} on serialize (the checkpointed branch +// payload) and unwraps it on deserialize, so the ordered results round-trip back to +// ["x", "y"]. MaxConcurrency=1 gives a deterministic history. +using System.IO; +using System.Text; +using System.Text.Json; +using Amazon.Lambda.Core; +using Amazon.Lambda.DurableExecution; +using Amazon.Lambda.RuntimeSupport; +using Amazon.Lambda.Serialization.SystemTextJson; + +namespace ParallelCustomSerdes; + +public class Function +{ + public static async Task Main(string[] args) + { + var handler = new Function(); + var serializer = new DefaultLambdaJsonSerializer(); + using var handlerWrapper = HandlerWrapper.GetHandlerWrapper(handler.Handler, serializer); + using var bootstrap = new LambdaBootstrap(handlerWrapper); + await bootstrap.RunAsync(); + } + + public Task Handler( + DurableExecutionInvocationInput input, ILambdaContext context) + => DurableFunction.WrapAsync>(Workflow, input, context); + + private async Task> Workflow(object? input, IDurableContext context) + { + var result = await context.ParallelAsync( + new Func>[] + { + async (_, _) => { await Task.CompletedTask; return "x"; }, + async (_, _) => { await Task.CompletedTask; return "y"; }, + }, + name: "serde", + config: new ParallelConfig + { + MaxConcurrency = 1, + ItemSerializer = new WrapJsonSerializer(), + }); + + return result.GetResults().ToList(); + } +} + +/// +/// A symmetric per-branch serializer: serializes a string value v as the JSON +/// envelope {"wrapped":"v"} and deserializes that envelope back to v. +/// Used only for per-branch results, so it only handles strings. +/// +public sealed class WrapJsonSerializer : ILambdaSerializer +{ + public T Deserialize(Stream requestStream) + { + if (typeof(T) != typeof(string)) + throw new NotSupportedException( + $"{nameof(WrapJsonSerializer)} only supports string results; got {typeof(T)}."); + using var reader = new StreamReader(requestStream, Encoding.UTF8); + var text = reader.ReadToEnd(); + using var doc = JsonDocument.Parse(text); + var value = doc.RootElement.GetProperty("wrapped").GetString() ?? string.Empty; + return (T)(object)value; + } + + public void Serialize(T response, Stream responseStream) + { + var value = response as string ?? response?.ToString() ?? string.Empty; + var json = "{\"wrapped\":" + JsonSerializer.Serialize(value) + "}"; + var bytes = Encoding.UTF8.GetBytes(json); + responseStream.Write(bytes, 0, bytes.Length); + } +} diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/parallel/ParallelCustomSerdes/ParallelCustomSerdes.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/parallel/ParallelCustomSerdes/ParallelCustomSerdes.csproj new file mode 100644 index 000000000..4b354e178 --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/parallel/ParallelCustomSerdes/ParallelCustomSerdes.csproj @@ -0,0 +1,19 @@ + + + + net8.0 + Exe + true + bootstrap + enable + enable + + + + + + + + + + diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/step/StepCustomSerdes/Function.cs b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/step/StepCustomSerdes/Function.cs index 7f043ef18..6fa4233f6 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/step/StepCustomSerdes/Function.cs +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/step/StepCustomSerdes/Function.cs @@ -1,7 +1,11 @@ -// 1-6: Custom serdes (per-step) - transforms string to uppercase -// Note: The .NET SDK does not have a per-step serdes API like the JS SDK. -// Instead, we achieve the same effect by transforming the value within the step -// function itself, since the step result is what gets checkpointed. +// 1-6: Custom serdes (per-step) via StepConfig.Serializer +// The step returns the input unchanged; a custom per-step serializer transforms the +// result to uppercase on serialize. Thanks to the SDK's fresh-success round-trip +// (the result is deserialized from the just-written checkpoint before being returned), +// the uppercased value is what the workflow observes and returns — so the execution +// result is the uppercased input. +using System.IO; +using System.Text; using Amazon.Lambda.Core; using Amazon.Lambda.DurableExecution; using Amazon.Lambda.RuntimeSupport; @@ -30,10 +34,33 @@ private async Task Workflow(string input, IDurableContext context) async (_, _ct) => { await Task.CompletedTask; - // Simulate custom serdes by transforming to uppercase - return input.ToUpperInvariant(); - }); + return input; + }, + config: new StepConfig { Serializer = new UppercaseSerializer() }); return result; } } + +/// +/// A custom per-step serializer that uppercases the result on serialize (writing the +/// raw transformed text) and reads it back verbatim on deserialize. +/// +public sealed class UppercaseSerializer : ILambdaSerializer +{ + public T Deserialize(Stream requestStream) + { + if (typeof(T) != typeof(string)) + throw new NotSupportedException( + $"{nameof(UppercaseSerializer)} only supports string results; got {typeof(T)}."); + using var reader = new StreamReader(requestStream, Encoding.UTF8); + return (T)(object)reader.ReadToEnd(); + } + + public void Serialize(T response, Stream responseStream) + { + var value = (response as string ?? response?.ToString() ?? string.Empty).ToUpperInvariant(); + var bytes = Encoding.UTF8.GetBytes(value); + responseStream.Write(bytes, 0, bytes.Length); + } +} diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_child.yaml b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_child.yaml index 4bb91c2db..f985be4d8 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_child.yaml +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_child.yaml @@ -60,9 +60,6 @@ Resources: Type: AWS::Serverless::Function TestingMetadata: TestDescription: ["3-1"] - NotImplemented: - - id: "3-14" - reason: "Child context with custom serdes: .NET has no per-operation serdes slot; all payloads use the one registered ILambdaSerializer (matches the Java approach)." Metadata: BuildMethod: makefile Properties: @@ -372,3 +369,21 @@ Resources: DurableConfig: RetentionPeriodInDays: 7 ExecutionTimeout: 300 + + ChildCustomSerdes: + Type: AWS::Serverless::Function + TestingMetadata: + TestDescription: ["3-14"] + Metadata: + BuildMethod: makefile + Properties: + CodeUri: publish/ChildCustomSerdes/ + Handler: bootstrap + Description: Child context with a custom serdes (uppercases the child result on serialize) + Role: + Fn::GetAtt: + - DurableFunctionRole + - Arn + DurableConfig: + RetentionPeriodInDays: 7 + ExecutionTimeout: 300 diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_invoke.yaml b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_invoke.yaml index 747dbe078..074d2a342 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_invoke.yaml +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_invoke.yaml @@ -69,9 +69,6 @@ Resources: Type: AWS::Serverless::Function TestingMetadata: TestDescription: ["5-1"] - NotImplemented: - - id: "5-16" - reason: "Invoke with custom result serdes: .NET has no per-operation serdes slot; the invoke result is deserialized via the one registered ILambdaSerializer (matches the Java approach)." Metadata: BuildMethod: makefile Properties: @@ -415,3 +412,25 @@ Resources: Variables: TARGET_FUNCTION_NAME: !Sub "${InvokeEchoTarget.Arn}:$LATEST" + + InvokeCustomResultSerdes: + Type: AWS::Serverless::Function + TestingMetadata: + TestDescription: ["5-16"] + Metadata: + BuildMethod: makefile + Properties: + CodeUri: publish/InvokeCustomResultSerdes/ + Handler: bootstrap + Description: Invoke with a custom result serdes (uppercases the deserialized result) + Role: + Fn::GetAtt: + - DurableFunctionRole + - Arn + DurableConfig: + RetentionPeriodInDays: 7 + ExecutionTimeout: 300 + Environment: + Variables: + TARGET_FUNCTION_NAME: + !Sub "${InvokeEchoTarget.Arn}:$LATEST" diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_map.yaml b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_map.yaml index 2c74104f3..550f2edb9 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_map.yaml +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_map.yaml @@ -8,10 +8,11 @@ Globals: MemorySize: 512 # Coverage note: .NET's MapConfig exposes MaxConcurrency, CompletionConfig, -# NestingType, and ItemNamer, but no item-level or whole-result serdes slots -# (per-item checkpoint payloads use the registered ILambdaSerializer). Tests -# 9-14 (per-item serdes), 9-19 and 9-20 (operation-level serdes) therefore have -# no .NET example and are left uncovered by design, matching the Java approach. +# NestingType, ItemNamer, and ItemSerializer (per-item result serializer). The +# per-item serdes test (9-14) uses ItemSerializer. The whole-result serdes tests +# 9-19 and 9-20 have no .NET example — MapConfig has no operation-level (aggregate) +# serializer slot; the aggregate BatchResult envelope is SDK-internal — so they are +# left uncovered by design, matching the Java approach. Resources: DurableFunctionRole: @@ -42,8 +43,6 @@ Resources: TestingMetadata: TestDescription: ["9-1"] NotImplemented: - - id: "9-14" - reason: "Per-item serdes: .NET MapConfig has no item-level serializer slot; all payloads use the one registered ILambdaSerializer (matches the Java approach)." - id: "9-19" reason: "Operation-level serdes: .NET MapConfig has no whole-result serializer slot; all payloads use the one registered ILambdaSerializer." - id: "9-20" @@ -349,3 +348,21 @@ Resources: DurableConfig: RetentionPeriodInDays: 7 ExecutionTimeout: 300 + + MapCustomSerdes: + Type: AWS::Serverless::Function + TestingMetadata: + TestDescription: ["9-14"] + Metadata: + BuildMethod: makefile + Properties: + CodeUri: publish/MapCustomSerdes/ + Handler: bootstrap + Description: Map with a custom per-item serializer (ItemSerializer round-trips each result) + Role: + Fn::GetAtt: + - DurableFunctionRole + - Arn + DurableConfig: + RetentionPeriodInDays: 7 + ExecutionTimeout: 300 diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_parallel.yaml b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_parallel.yaml index 589382ca8..3313371e0 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_parallel.yaml +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/Conformance/template_parallel.yaml @@ -36,9 +36,6 @@ Resources: Type: AWS::Serverless::Function TestingMetadata: TestDescription: ["8-1"] - NotImplemented: - - id: "8-15" - reason: "Parallel with custom per-item serdes: .NET ParallelConfig has no item serializer slot; all branch payloads use the one registered ILambdaSerializer (matches the Java approach)." Metadata: BuildMethod: makefile Properties: @@ -412,3 +409,21 @@ Resources: DurableConfig: RetentionPeriodInDays: 7 ExecutionTimeout: 300 + + ParallelCustomSerdes: + Type: AWS::Serverless::Function + TestingMetadata: + TestDescription: ["8-15"] + Metadata: + BuildMethod: makefile + Properties: + CodeUri: publish/ParallelCustomSerdes/ + Handler: bootstrap + Description: Parallel with a custom per-branch serde (ItemSerializer wraps/unwraps each branch result) + Role: + Fn::GetAtt: + - DurableFunctionRole + - Arn + DurableConfig: + RetentionPeriodInDays: 7 + ExecutionTimeout: 300 diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/MapParallelItemSerializerTest.cs b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/MapParallelItemSerializerTest.cs new file mode 100644 index 000000000..001b3b4a0 --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/MapParallelItemSerializerTest.cs @@ -0,0 +1,60 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using System.Linq; +using Amazon.Lambda.Model; +using Xunit; +using Xunit.Abstractions; + +namespace Amazon.Lambda.DurableExecution.IntegrationTests; + +/// +/// Cloud integration test for the Map/Parallel per-item serializer (ItemSerializer). +/// Deploys MapParallelItemSerializerFunction and asserts from event history that the +/// map items (m-0, m-1) and parallel branch (p-0) — each configured with a camelCase +/// ItemSerializer — produce camelCase per-item result payloads, while the control +/// step (global serializer) stays PascalCase. Proves ItemSerializer routes end-to-end. +/// +public class MapParallelItemSerializerTest +{ + private readonly ITestOutputHelper _output; + public MapParallelItemSerializerTest(ITestOutputHelper output) => _output = output; + + [Fact] + public async Task ItemSerializer_AppliesToMapItemsAndParallelBranches() + { + await using var deployment = await DurableFunctionDeployment.CreateAsync( + DurableFunctionDeployment.FindTestFunctionDir("MapParallelItemSerializerFunction"), + "mpser", _output); + + var (_, executionName) = await deployment.InvokeAsync("{}"); + + var arn = await deployment.FindDurableExecutionArnByNameAsync(executionName, TimeSpan.FromSeconds(60)); + Assert.NotNull(arn); + + var status = await deployment.PollForCompletionAsync(arn!, TimeSpan.FromSeconds(90)); + Assert.Equal("SUCCEEDED", status, ignoreCase: true); + + var itemNames = new[] { "m-0", "m-1", "p-0" }; + var history = await deployment.WaitForHistoryAsync( + arn!, + h => (h.Events?.Any(e => e.EventType == EventType.StepSucceeded && e.Name == "control_step") ?? false) + && itemNames.All(n => h.Events?.Any(e => e.EventType == EventType.ContextSucceeded && e.Name == n) ?? false), + TimeSpan.FromSeconds(90)); + var events = history.Events ?? new List(); + + // Control step used the global serializer → PascalCase. + var control = events.First(e => e.EventType == EventType.StepSucceeded && e.Name == "control_step"); + Assert.Contains("\"Message\"", control.StepSucceededDetails.Result.Payload); + + // Each map item + parallel branch used its camelCase ItemSerializer for the result. + foreach (var name in itemNames) + { + var ev = events.First(e => e.EventType == EventType.ContextSucceeded && e.Name == name); + var payload = ev.ContextSucceededDetails.Result?.Payload ?? string.Empty; + _output.WriteLine($"{name} payload: {payload}"); + Assert.Contains("\"message\"", payload); + Assert.DoesNotContain("\"Message\"", payload); + } + } +} diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/PerOperationSerializerTest.cs b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/PerOperationSerializerTest.cs new file mode 100644 index 000000000..8687bfcde --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/PerOperationSerializerTest.cs @@ -0,0 +1,62 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using System.Linq; +using Amazon.Lambda.Model; +using Xunit; +using Xunit.Abstractions; + +namespace Amazon.Lambda.DurableExecution.IntegrationTests; + +/// +/// Cloud integration test for the per-operation serializer override. Deploys +/// PerOperationSerializerFunction, which runs two steps returning the same record: +/// default_step (global serializer, PascalCase) and camel_step +/// (StepConfig.Serializer = CamelCaseLambdaJsonSerializer, camelCase). Asserts the +/// checkpointed payloads differ accordingly — proving the per-step serializer is applied to +/// exactly the configured step, end-to-end through the durable execution service. +/// +public class PerOperationSerializerTest +{ + private readonly ITestOutputHelper _output; + public PerOperationSerializerTest(ITestOutputHelper output) => _output = output; + + [Fact] + public async Task PerStepSerializer_AppliesToConfiguredStepOnly() + { + await using var deployment = await DurableFunctionDeployment.CreateAsync( + DurableFunctionDeployment.FindTestFunctionDir("PerOperationSerializerFunction"), + "perser", _output); + + var (_, executionName) = await deployment.InvokeAsync("{}"); + + var arn = await deployment.FindDurableExecutionArnByNameAsync(executionName, TimeSpan.FromSeconds(60)); + Assert.NotNull(arn); + + var status = await deployment.PollForCompletionAsync(arn!, TimeSpan.FromSeconds(60)); + Assert.Equal("SUCCEEDED", status, ignoreCase: true); + + // History is eventually consistent — wait until both step-succeeded events are indexed. + var history = await deployment.WaitForHistoryAsync( + arn!, + h => (h.Events?.Count(e => e.StepSucceededDetails != null) ?? 0) >= 2, + TimeSpan.FromSeconds(60)); + var events = history.Events ?? new List(); + + string PayloadFor(string name) => events + .First(e => e.StepSucceededDetails != null && e.Name == name) + .StepSucceededDetails.Result.Payload; + + var defaultPayload = PayloadFor("default_step"); + var camelPayload = PayloadFor("camel_step"); + _output.WriteLine($"default_step payload: {defaultPayload}"); + _output.WriteLine($"camel_step payload: {camelPayload}"); + + // default_step used the global serializer (AWS naming policy → PascalCase). + Assert.Contains("\"Message\"", defaultPayload); + + // camel_step overrode the serializer (camelCase) — applied to this step only. + Assert.Contains("\"message\"", camelPayload); + Assert.DoesNotContain("\"Message\"", camelPayload); + } +} diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/MapParallelItemSerializerFunction/Function.cs b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/MapParallelItemSerializerFunction/Function.cs new file mode 100644 index 000000000..04982155e --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/MapParallelItemSerializerFunction/Function.cs @@ -0,0 +1,68 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using Amazon.Lambda.Core; +using Amazon.Lambda.DurableExecution; +using Amazon.Lambda.RuntimeSupport; +using Amazon.Lambda.Serialization.SystemTextJson; + +namespace DurableExecutionTestFunction; + +/// +/// Deployed entry point exercising the per-item serializer on Map and Parallel. +/// +/// Global serializer is (PascalCase). A control +/// step (no config) serializes its result with the global serializer +/// (payload contains "Message"). The map items (m-0, m-1) and the parallel branch +/// (p-0) each set ItemSerializer = CamelCaseLambdaJsonSerializer, so their per-item +/// child-context result payloads are camelCase ("message"). Each item/branch returns +/// a directly (no inner step) so the only per-item serialization is the +/// item result itself. The paired integration test asserts this from event history. +/// +public class Function +{ + public static async Task Main(string[] args) + { + var handler = new Function(); + var serializer = new DefaultLambdaJsonSerializer(); + using var handlerWrapper = HandlerWrapper.GetHandlerWrapper(handler.Handler, serializer); + using var bootstrap = new LambdaBootstrap(handlerWrapper); + await bootstrap.RunAsync(); + } + + public Task Handler( + DurableExecutionInvocationInput input, ILambdaContext context) + => DurableFunction.WrapAsync>(RunAsync, input, context); + + private static async Task> RunAsync(object input, IDurableContext ctx) + { + // Control: global serializer → PascalCase "Message". + await ctx.StepAsync( + async (_, _) => { await Task.CompletedTask; return new Doc("ctrl"); }, + name: "control_step"); + + // Map: per-item camelCase serializer → each item result "message". + var map = await ctx.MapAsync( + new[] { "a", "b" }, + async (_, item, _, _, _) => { await Task.CompletedTask; return new Doc(item); }, + name: "map", + config: new MapConfig + { + ItemSerializer = new CamelCaseLambdaJsonSerializer(), + ItemNamer = (item, idx) => $"m-{idx}", + }); + + // Parallel: per-branch camelCase serializer. + await ctx.ParallelAsync( + new[] + { + new DurableBranch("p-0", async (_, _) => { await Task.CompletedTask; return new Doc("px"); }), + }, + name: "par", + config: new ParallelConfig { ItemSerializer = new CamelCaseLambdaJsonSerializer() }); + + return map.GetResults(); + } +} + +public record Doc(string Message); diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/MapParallelItemSerializerFunction/MapParallelItemSerializerFunction.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/MapParallelItemSerializerFunction/MapParallelItemSerializerFunction.csproj new file mode 100644 index 000000000..f8bf7fd0c --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/MapParallelItemSerializerFunction/MapParallelItemSerializerFunction.csproj @@ -0,0 +1,18 @@ + + + + net10.0 + Exe + true + bootstrap + enable + enable + + + + + + + + + diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/PerOperationSerializerFunction/Function.cs b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/PerOperationSerializerFunction/Function.cs new file mode 100644 index 000000000..47abdf46a --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/PerOperationSerializerFunction/Function.cs @@ -0,0 +1,59 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using Amazon.Lambda.Core; +using Amazon.Lambda.DurableExecution; +using Amazon.Lambda.RuntimeSupport; +using Amazon.Lambda.Serialization.SystemTextJson; + +namespace DurableExecutionTestFunction; + +/// +/// Deployed entry point exercising the per-operation serializer override. +/// +/// The global serializer is (AWS naming policy, +/// which preserves PascalCase property names). The workflow runs two steps that return the +/// same record: +/// +/// default_step uses no config, so it is serialized with the global +/// serializer → checkpoint payload contains "Message" (PascalCase). +/// camel_step overrides with +/// → checkpoint payload contains +/// "message" (camelCase). +/// +/// The paired integration test asserts both payloads from the event history, proving the +/// per-step serializer is applied to exactly the configured step and nothing else. +/// +public class Function +{ + public static async Task Main(string[] args) + { + var handler = new Function(); + var serializer = new DefaultLambdaJsonSerializer(); + using var handlerWrapper = HandlerWrapper.GetHandlerWrapper(handler.Handler, serializer); + using var bootstrap = new LambdaBootstrap(handlerWrapper); + await bootstrap.RunAsync(); + } + + public Task Handler( + DurableExecutionInvocationInput input, ILambdaContext context) + => DurableFunction.WrapAsync(RunAsync, input, context); + + private static async Task RunAsync(object input, IDurableContext ctx) + { + // Serialized with the global serializer (PascalCase "Message"). + _ = await ctx.StepAsync( + async (_, _) => { await Task.CompletedTask; return new Doc("hi"); }, + name: "default_step"); + + // Same result type, but this step overrides the serializer (camelCase "message"). + var viaCustom = await ctx.StepAsync( + async (_, _) => { await Task.CompletedTask; return new Doc("hi"); }, + name: "camel_step", + config: new StepConfig { Serializer = new CamelCaseLambdaJsonSerializer() }); + + return viaCustom; + } +} + +public record Doc(string Message); diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/PerOperationSerializerFunction/PerOperationSerializerFunction.csproj b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/PerOperationSerializerFunction/PerOperationSerializerFunction.csproj new file mode 100644 index 000000000..f8bf7fd0c --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.IntegrationTests/TestFunctions/PerOperationSerializerFunction/PerOperationSerializerFunction.csproj @@ -0,0 +1,18 @@ + + + + net10.0 + Exe + true + bootstrap + enable + enable + + + + + + + + + diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/PerOperationSerializerTests.cs b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/PerOperationSerializerTests.cs new file mode 100644 index 000000000..48b342377 --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/PerOperationSerializerTests.cs @@ -0,0 +1,458 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using Amazon.Lambda.Core; +using Amazon.Lambda.DurableExecution.Internal; +using Amazon.Lambda.Serialization.SystemTextJson; +using Amazon.Lambda.TestUtilities; +using Xunit; + +namespace Amazon.Lambda.DurableExecution.Tests; + +/// +/// Verifies the per-operation serializer override (Phase 1): each single-result +/// operation config exposes an optional that, when set, +/// is used for that operation's payload (serialize and/or replay-deserialize) instead of +/// the globally-registered serializer on . When +/// unset, the global serializer is used. +/// +public class PerOperationSerializerTests +{ + private const string TestArn = "arn:aws:lambda:us-east-1:123:durable-execution:test"; + + private static string IdAt(int position) => OperationIdGenerator.HashOperationId(position.ToString()); + + private static string ChildIdAt(string parentOpId, int position) => + OperationIdGenerator.HashOperationId($"{parentOpId}-{position}"); + + private static DurableContext CreateContext(ILambdaSerializer globalSerializer, InitialExecutionState? initialState = null) + { + var state = new ExecutionState(); + state.LoadFromCheckpoint(initialState); + var tm = new TerminationManager(); + var idGen = new OperationIdGenerator(); + var lambdaContext = new TestLambdaContext { Serializer = globalSerializer }; + return new DurableContext(state, tm, new WorkflowCancellation(tm), idGen, TestArn, lambdaContext); + } + + /// + /// Spy that counts calls and delegates to a real + /// System.Text.Json serializer, so tests can assert which serializer an operation used. + /// + private sealed class SpySerializer : ILambdaSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + public int SerializeCount { get; private set; } + public int DeserializeCount { get; private set; } + + public T Deserialize(Stream requestStream) + { + DeserializeCount++; + return _inner.Deserialize(requestStream); + } + + public void Serialize(T response, Stream responseStream) + { + SerializeCount++; + _inner.Serialize(response, responseStream); + } + } + + // ---------------------------------------------------------------- Step + + [Fact] + public async Task Step_WithConfigSerializer_UsesPerOpSerializer_NotGlobal() + { + var global = new SpySerializer(); + var perOp = new SpySerializer(); + var ctx = CreateContext(global); + + var result = await ctx.StepAsync( + async (_, _) => { await Task.CompletedTask; return 42; }, + name: "s", + config: new StepConfig { Serializer = perOp }); + + Assert.Equal(42, result); + Assert.Equal(1, perOp.SerializeCount); + Assert.Equal(0, global.SerializeCount); + } + + [Fact] + public async Task Step_NoConfigSerializer_UsesGlobal() + { + var global = new SpySerializer(); + var ctx = CreateContext(global); + + var result = await ctx.StepAsync( + async (_, _) => { await Task.CompletedTask; return 7; }, + name: "s"); + + Assert.Equal(7, result); + Assert.Equal(1, global.SerializeCount); + } + + [Fact] + public async Task Step_Replay_UsesPerOpSerializerToDeserialize() + { + var global = new SpySerializer(); + var perOp = new SpySerializer(); + var ctx = CreateContext(global, new InitialExecutionState + { + Operations = new List + { + new() + { + Id = IdAt(1), + Type = OperationTypes.Step, + Status = OperationStatuses.Succeeded, + StepDetails = new StepDetails { Result = "\"cached\"" } + } + } + }); + + var result = await ctx.StepAsync( + async (_, _) => { await Task.CompletedTask; return "fresh"; }, + name: "s", + config: new StepConfig { Serializer = perOp }); + + Assert.Equal("cached", result); + Assert.Equal(1, perOp.DeserializeCount); + Assert.Equal(0, global.DeserializeCount); + } + + // ---------------------------------------------------------------- Callback (deserialize side) + + [Fact] + public async Task Callback_Replay_UsesPerOpSerializerToDeserialize() + { + var global = new SpySerializer(); + var perOp = new SpySerializer(); + var ctx = CreateContext(global, new InitialExecutionState + { + Operations = new List + { + new() + { + Id = IdAt(1), + Type = OperationTypes.Callback, + Status = OperationStatuses.Succeeded, + CallbackDetails = new CallbackDetails { CallbackId = "cb-1", Result = "\"cbval\"" } + } + } + }); + + var callback = await ctx.CreateCallbackAsync(name: "cb", config: new CallbackConfig { Serializer = perOp }); + var result = await callback.GetResultAsync(); + + Assert.Equal("cbval", result); + Assert.Equal(1, perOp.DeserializeCount); + Assert.Equal(0, global.DeserializeCount); + } + + // ---------------------------------------------------------------- Invoke + + [Fact] + public async Task Invoke_Replay_UsesPerOpSerializerToDeserialize() + { + var global = new SpySerializer(); + var perOp = new SpySerializer(); + var ctx = CreateContext(global, new InitialExecutionState + { + Operations = new List + { + new() + { + Id = IdAt(1), + Type = OperationTypes.ChainedInvoke, + Status = OperationStatuses.Succeeded, + ChainedInvokeDetails = new ChainedInvokeDetails { Result = "\"invval\"" } + } + } + }); + + var result = await ctx.InvokeAsync( + "arn:aws:lambda:us-east-1:123:function:callee:1", + "payload", + name: "inv", + config: new InvokeConfig { Serializer = perOp }); + + Assert.Equal("invval", result); + Assert.Equal(1, perOp.DeserializeCount); + Assert.Equal(0, global.DeserializeCount); + } + + [Fact] + public async Task Invoke_FreshExecution_UsesPerOpSerializerToSerializeRequestPayload() + { + // Comment (Copilot): InvokeConfig.Serializer applies to the outbound request + // payload on the initial (non-replay) execution as well as to the replay result. + // This covers the outbound-serialize path (the replay-deserialize path is above). + var global = new SpySerializer(); + var perOp = new SpySerializer(); + + var state = new ExecutionState(); + state.LoadFromCheckpoint(null); + var tm = new TerminationManager(); + var idGen = new OperationIdGenerator(); + var lambdaContext = new TestLambdaContext { Serializer = global }; + var recorder = new RecordingBatcher(); + var ctx = new DurableContext( + state, tm, new WorkflowCancellation(tm), idGen, TestArn, lambdaContext, recorder.Batcher); + + // Fresh chained invoke: the request payload is serialized (via the per-op + // serializer), the CHAINED_INVOKE START is flushed, then the workflow suspends. + // The returned task never completes on the fresh path. + var task = ctx.InvokeAsync( + "arn:aws:lambda:us-east-1:123:function:callee:1", + "payload", + name: "inv", + config: new InvokeConfig { Serializer = perOp }); + + await tm.WaitForTerminationAsync(); + Assert.False(task.IsCompleted); + + // The outbound request payload used the per-op serializer, not the global one. + Assert.Equal(1, perOp.SerializeCount); + Assert.Equal(0, global.SerializeCount); + } + + // ---------------------------------------------------------------- ChildContext + + [Fact] + public async Task ChildContext_WithConfigSerializer_UsesPerOpSerializer() + { + var global = new SpySerializer(); + var perOp = new SpySerializer(); + var ctx = CreateContext(global); + + var result = await ctx.RunInChildContextAsync( + async (_, _) => { await Task.CompletedTask; return 99; }, + name: "child", + config: new ChildContextConfig { Serializer = perOp }); + + Assert.Equal(99, result); + Assert.True(perOp.SerializeCount >= 1); + Assert.Equal(0, global.SerializeCount); + } + + // ---------------------------------------------------------------- WaitForCondition + + private sealed class StopImmediatelyStrategy : IWaitStrategy + { + public WaitDecision Decide(int state, int attemptNumber) => WaitDecision.Stop(); + } + + [Fact] + public async Task WaitForCondition_WithConfigSerializer_UsesPerOpSerializer() + { + var global = new SpySerializer(); + var perOp = new SpySerializer(); + var ctx = CreateContext(global); + + var result = await ctx.WaitForConditionAsync( + async (state, _, _) => { await Task.CompletedTask; return state + 1; }, + new WaitForConditionConfig + { + InitialState = 0, + WaitStrategy = new StopImmediatelyStrategy(), + Serializer = perOp, + }, + name: "wfc"); + + Assert.Equal(1, result); + Assert.True(perOp.SerializeCount >= 1); + Assert.Equal(0, global.SerializeCount); + } + + // ---------------------------------------------------------------- Map / Parallel (ItemSerializer) + + [Fact] + public async Task Map_WithItemSerializer_UsesItForItemResults() + { + var global = new SpySerializer(); + var itemSer = new SpySerializer(); + var ctx = CreateContext(global); + + var result = await ctx.MapAsync( + new[] { "a", "b" }, + async (_, item, _, _, _) => { await Task.CompletedTask; return item.ToUpperInvariant(); }, + name: "map", + config: new MapConfig { ItemSerializer = itemSer }); + + Assert.Equal(2, result.SuccessCount); + Assert.Equal(new[] { "A", "B" }, result.GetResults()); + Assert.True(itemSer.SerializeCount >= 2); // each item result serialized via the item serializer + Assert.Equal(0, global.SerializeCount); + } + + [Fact] + public async Task Map_NoItemSerializer_UsesGlobal() + { + var global = new SpySerializer(); + var ctx = CreateContext(global); + + var result = await ctx.MapAsync( + new[] { "a" }, + async (_, item, _, _, _) => { await Task.CompletedTask; return item; }, + name: "map"); + + Assert.Equal(1, result.SuccessCount); + Assert.True(global.SerializeCount >= 1); + } + + [Fact] + public async Task Parallel_WithItemSerializer_UsesItForBranchResults() + { + var global = new SpySerializer(); + var itemSer = new SpySerializer(); + var ctx = CreateContext(global); + + var branches = new Func>[] + { + async (_, _) => { await Task.CompletedTask; return "x"; }, + async (_, _) => { await Task.CompletedTask; return "y"; }, + }; + + var result = await ctx.ParallelAsync( + branches, + name: "par", + config: new ParallelConfig { ItemSerializer = itemSer }); + + Assert.Equal(2, result.SuccessCount); + Assert.True(itemSer.SerializeCount >= 2); // each branch result serialized via the item serializer + Assert.Equal(0, global.SerializeCount); + } + + // ---------------------------------------------------------------- Map / Parallel replay (deserialize) + + [Fact] + public async Task Map_Replay_UsesItemSerializerToDeserializeItemResults() + { + // Comment (Copilot): also cover that ItemSerializer is used for replay + // deserialization of cached per-item results (not just fresh serialization). + var global = new SpySerializer(); + var itemSer = new SpySerializer(); + + var parentOpId = IdAt(1); + var i0 = ChildIdAt(parentOpId, 1); + var i1 = ChildIdAt(parentOpId, 2); + + var summaryJson = + "{\"CompletionReason\":\"ALL_COMPLETED\",\"Units\":[" + + "{\"Index\":0,\"Name\":\"0\",\"Status\":\"SUCCEEDED\"}," + + "{\"Index\":1,\"Name\":\"1\",\"Status\":\"SUCCEEDED\"}]}"; + + var ctx = CreateContext(global, new InitialExecutionState + { + Operations = new List + { + new() + { + Id = parentOpId, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.Map, + Name = "map", + ContextDetails = new ContextDetails { Result = summaryJson } + }, + new() + { + Id = i0, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.MapIteration, + Name = "0", + ContextDetails = new ContextDetails { Result = "\"A\"" } + }, + new() + { + Id = i1, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.MapIteration, + Name = "1", + ContextDetails = new ContextDetails { Result = "\"B\"" } + } + } + }); + + var calls = 0; + var result = await ctx.MapAsync( + new[] { "a", "b" }, + async (_, item, _, _, _) => { calls++; await Task.CompletedTask; return item.ToUpperInvariant(); }, + name: "map", + config: new MapConfig { ItemSerializer = itemSer }); + + Assert.Equal(0, calls); // cached — callback not re-run + Assert.Equal(new[] { "A", "B" }, result.GetResults()); + Assert.True(itemSer.DeserializeCount >= 2); // each cached item result deserialized via the item serializer + Assert.Equal(0, global.DeserializeCount); // aggregate summary uses source-gen, not the ILambdaSerializer + } + + [Fact] + public async Task Parallel_Replay_UsesItemSerializerToDeserializeBranchResults() + { + var global = new SpySerializer(); + var itemSer = new SpySerializer(); + + var parentOpId = IdAt(1); + var b0 = ChildIdAt(parentOpId, 1); + var b1 = ChildIdAt(parentOpId, 2); + + var summaryJson = + "{\"CompletionReason\":\"ALL_COMPLETED\",\"Units\":[" + + "{\"Index\":0,\"Name\":\"0\",\"Status\":\"SUCCEEDED\"}," + + "{\"Index\":1,\"Name\":\"1\",\"Status\":\"SUCCEEDED\"}]}"; + + var ctx = CreateContext(global, new InitialExecutionState + { + Operations = new List + { + new() + { + Id = parentOpId, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.Parallel, + Name = "par", + ContextDetails = new ContextDetails { Result = summaryJson } + }, + new() + { + Id = b0, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.ParallelBranch, + Name = "0", + ContextDetails = new ContextDetails { Result = "\"x\"" } + }, + new() + { + Id = b1, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.ParallelBranch, + Name = "1", + ContextDetails = new ContextDetails { Result = "\"y\"" } + } + } + }); + + var executed = false; + var branches = new Func>[] + { + async (_, _) => { executed = true; await Task.CompletedTask; return "x"; }, + async (_, _) => { executed = true; await Task.CompletedTask; return "y"; }, + }; + + var result = await ctx.ParallelAsync( + branches, + name: "par", + config: new ParallelConfig { ItemSerializer = itemSer }); + + Assert.False(executed); // cached — branches not re-run + Assert.Equal(new[] { "x", "y" }, result.GetResults()); + Assert.True(itemSer.DeserializeCount >= 2); // each cached branch result deserialized via the item serializer + Assert.Equal(0, global.DeserializeCount); + } +} From f792e00d0865bc8807c6836ead497971a693339d Mon Sep 17 00:00:00 2001 From: Garrett Beatty Date: Thu, 3 Sep 2026 21:04:08 -0400 Subject: [PATCH 2/5] feat(DurableExecution): context-aware serializer engine + Map/Parallel determinism (#2559) Introduces the serializer engine that the FileSystemSerializer builds on, split out of the original filesystem PR (#2558) for reviewability: - Optional IDurableResultSerializer + DurableSerializationContext (EntityId + DurableExecutionArn) and LambdaSerializerHelper dispatch; plain ILambdaSerializer serializers are unaffected (byte-identical fallback). - Extends the fresh-success serializer round-trip (established for Step/Child in the conformance work) to Map/Parallel Flat per-item results, eliminating a fresh-vs-replay divergence for non-round-tripping ItemSerializers. - Map/Parallel Nested parent now inlines each child's ORIGINAL SUCCEED payload verbatim instead of re-serializing the round-tripped value. - Terminal handling: a fresh-success round-trip deserialize failure (Step / Child / Flat) is terminal (no retry, body never re-run); Step post-SUCCEED enqueue failure routed through the same terminal path. These changes are triggered by the per-operation serializer feature (#2555), not by FileSystemSerializer specifically; FileSystemSerializer is the leaf that exercises them. Note: two doc links are rendered as ... here and restored to cref links in the FileSystemSerializer PR that introduces the type. --- .../3691fc04-ad66-49c5-a383-f9592a6da0ce.json | 11 + .../c1cb8e3a-cca9-415e-b9ca-08289658df08.json | 11 + .../ca5a9844-031a-4442-899a-913d689449ec.json | 11 + .../d12219eb-e44b-4e90-8473-0ca63319ad53.json | 11 + .../f3a1520c-8eca-4ac8-a7c6-27168feaff25.json | 11 + .../ChildContextConfig.cs | 14 + .../DurableSerializationContext.cs | 39 +++ .../IDurableResultSerializer.cs | 44 +++ .../Internal/ChildContextOperation.cs | 92 +++++- .../Internal/ConcurrentOperation.cs | 286 +++++++++++++++--- .../Internal/LambdaSerializerHelper.cs | 29 ++ .../Internal/StepOperation.cs | 118 ++++++-- .../Internal/WaitForConditionOperation.cs | 12 +- .../MapConfig.cs | 16 + .../ParallelConfig.cs | 16 + .../ChildContextOperationTests.cs | 128 ++++++++ .../DurableContextTests.cs | 122 ++++++++ .../DurableResultSerializerContextTests.cs | 273 +++++++++++++++++ .../MapOperationTests.cs | 230 +++++++++++++- 19 files changed, 1399 insertions(+), 75 deletions(-) create mode 100644 .autover/changes/3691fc04-ad66-49c5-a383-f9592a6da0ce.json create mode 100644 .autover/changes/c1cb8e3a-cca9-415e-b9ca-08289658df08.json create mode 100644 .autover/changes/ca5a9844-031a-4442-899a-913d689449ec.json create mode 100644 .autover/changes/d12219eb-e44b-4e90-8473-0ca63319ad53.json create mode 100644 .autover/changes/f3a1520c-8eca-4ac8-a7c6-27168feaff25.json create mode 100644 Libraries/src/Amazon.Lambda.DurableExecution/DurableSerializationContext.cs create mode 100644 Libraries/src/Amazon.Lambda.DurableExecution/IDurableResultSerializer.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableResultSerializerContextTests.cs diff --git a/.autover/changes/3691fc04-ad66-49c5-a383-f9592a6da0ce.json b/.autover/changes/3691fc04-ad66-49c5-a383-f9592a6da0ce.json new file mode 100644 index 000000000..6facd2c4f --- /dev/null +++ b/.autover/changes/3691fc04-ad66-49c5-a383-f9592a6da0ce.json @@ -0,0 +1,11 @@ +{ + "Projects": [ + { + "Name": "Amazon.Lambda.DurableExecution", + "Type": "Major", + "ChangelogMessages": [ + "Behavior change: when a step\u0027s fresh-success serializer round-trip fails (e.g. an asymmetric/broken StepConfig.Serializer that cannot deserialize its own output), the step now fails TERMINALLY (FAIL checkpoint \u002B StepException) instead of being routed through the configured RetryStrategy. Previously a non-null RetryStrategy would re-invoke the already-succeeded, side-effecting step body on each attempt (duplicating side effects) until attempts exhausted. This mirrors ChildContextOperation\u0027s terminal handling of a round-trip failure." + ] + } + ] +} \ No newline at end of file diff --git a/.autover/changes/c1cb8e3a-cca9-415e-b9ca-08289658df08.json b/.autover/changes/c1cb8e3a-cca9-415e-b9ca-08289658df08.json new file mode 100644 index 000000000..f9db457d8 --- /dev/null +++ b/.autover/changes/c1cb8e3a-cca9-415e-b9ca-08289658df08.json @@ -0,0 +1,11 @@ +{ + "Projects": [ + { + "Name": "Amazon.Lambda.DurableExecution", + "Type": "Patch", + "ChangelogMessages": [ + "Behavior change: when a step post-round-trip SUCCEED-checkpoint enqueue fails (e.g. a broken CheckpointBatcher), the failure is now routed through the terminal path (FAIL checkpoint plus StepException) rather than propagating raw, consistent with the fresh-success round-trip failure handling. The side-effecting body is never re-invoked, and if emitting the terminal FAIL itself throws, the error propagates without recursion." + ] + } + ] +} \ No newline at end of file diff --git a/.autover/changes/ca5a9844-031a-4442-899a-913d689449ec.json b/.autover/changes/ca5a9844-031a-4442-899a-913d689449ec.json new file mode 100644 index 000000000..493339088 --- /dev/null +++ b/.autover/changes/ca5a9844-031a-4442-899a-913d689449ec.json @@ -0,0 +1,11 @@ +{ + "Projects": [ + { + "Name": "Amazon.Lambda.DurableExecution", + "Type": "Major", + "ChangelogMessages": [ + "Behavior change: under NestingType.Flat, Map and Parallel per-item/branch results are now round-tripped through the configured ItemSerializer on a fresh (non-replay) success -- deserializing the just-serialized inline payload before it enters the result -- matching what replay reconstructs from that payload. A non-round-tripping ItemSerializer transform is now reflected in the item/branch result on the first execution, not only on replay, eliminating a fresh-vs-replay divergence. As with steps and child contexts, the returned value is a fresh deserialized instance, and overflow (replay-children) results are unaffected (recovered by re-running the body)." + ] + } + ] +} \ No newline at end of file diff --git a/.autover/changes/d12219eb-e44b-4e90-8473-0ca63319ad53.json b/.autover/changes/d12219eb-e44b-4e90-8473-0ca63319ad53.json new file mode 100644 index 000000000..67dc64bdb --- /dev/null +++ b/.autover/changes/d12219eb-e44b-4e90-8473-0ca63319ad53.json @@ -0,0 +1,11 @@ +{ + "Projects": [ + { + "Name": "Amazon.Lambda.DurableExecution", + "Type": "Major", + "ChangelogMessages": [ + "Behavior change: a fresh-success serializer round-trip DESERIALIZE failure is now terminal on the child-context and Flat Map/Parallel paths, matching StepOperation. (Child) ChildContextOperation now emits a CONTEXT FAIL for a non-virtual, non-suppressed child whose round-trip deserialize throws, instead of throwing with no terminal checkpoint (which left the child STARTED and re-ran its side-effecting body on replay). (Flat Map/Parallel) a per-item round-trip deserialize failure now records that unit\u0027s terminal Error inline and still emits the parent SUCCEED, instead of propagating raw with no parent checkpoint (which re-ran every virtual unit body on replay, a poison loop, and bypassed the record-failure contract). Replay reconstructs the terminal failure without re-running or re-deserializing." + ] + } + ] +} \ No newline at end of file diff --git a/.autover/changes/f3a1520c-8eca-4ac8-a7c6-27168feaff25.json b/.autover/changes/f3a1520c-8eca-4ac8-a7c6-27168feaff25.json new file mode 100644 index 000000000..468421911 --- /dev/null +++ b/.autover/changes/f3a1520c-8eca-4ac8-a7c6-27168feaff25.json @@ -0,0 +1,11 @@ +{ + "Projects": [ + { + "Name": "Amazon.Lambda.DurableExecution", + "Type": "Major", + "ChangelogMessages": [ + "Behavior change (Map/Parallel, NestingType.Nested default): the parent now inlines each Nested unit\u0027s ORIGINAL child SUCCEED-checkpoint payload (S(body)) verbatim on the parent summary, and reconstructs it on replay with the child\u0027s own operation id, instead of re-serializing the already-round-tripped return value at the parent\u0027s per-unit id. This removes a fresh-vs-replay divergence with a non-round-tripping ItemSerializer (fresh vs replay values now match) and, for a context-aware serializer such as FileSystemSerializer, stops the parent writing a second file that orphaned the child\u0027s file. Round-tripping (symmetric) serializers and the default JSON serializer are unaffected; the Flat path is unchanged." + ] + } + ] +} \ No newline at end of file diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/ChildContextConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/ChildContextConfig.cs index 608f93a43..7619d54eb 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/ChildContextConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/ChildContextConfig.cs @@ -62,5 +62,19 @@ public sealed class ChildContextConfig /// (default), the globally-registered on /// is used. /// + /// + /// Size-dependent transform on overflow. On a fresh (non-replay) success the + /// child context result is round-tripped through this serializer before it is + /// returned, so a serializer that transforms the value (e.g. redaction, canonicalization) + /// has that transform reflected in the returned result — but only while the + /// serialized payload fits inline in a single checkpoint (≈256 KB). If the result + /// is large enough to overflow, the payload is not stored inline; the value is instead + /// recovered by re-running the child body on replay, which does not apply the + /// round-trip transform. A serializer whose Deserialize is not a pure inverse of + /// its Serialize will therefore observe a size-dependent difference in the returned + /// value (transform applied for small results, skipped for overflowed ones). Prefer a + /// round-tripping serializer, or do not depend on the transform being applied to results + /// large enough to overflow the checkpoint. + /// public ILambdaSerializer? Serializer { get; set; } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/DurableSerializationContext.cs b/Libraries/src/Amazon.Lambda.DurableExecution/DurableSerializationContext.cs new file mode 100644 index 000000000..bedd0bc7a --- /dev/null +++ b/Libraries/src/Amazon.Lambda.DurableExecution/DurableSerializationContext.cs @@ -0,0 +1,39 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +namespace Amazon.Lambda.DurableExecution; + +/// +/// Identifying information passed to an when a +/// durable operation's result is (de)serialized. It lets a context-aware serializer +/// derive a stable, collision-free external location (for example a file path) for the +/// value being stored. +/// +/// +/// The two values are stable for the lifetime of a durable execution — they do not +/// change across the multiple Lambda invocations (replays) of that execution. +/// +public readonly struct DurableSerializationContext +{ + /// + /// Stable identifier of the operation whose result is being (de)serialized (the + /// durable operation id; for Map/Parallel units, a per-unit id derived from it). + /// Unique within a single durable execution. + /// + public string EntityId { get; } + + /// + /// ARN of the durable execution. Used to avoid collisions between the stored + /// results of different executions. + /// + public string DurableExecutionArn { get; } + + /// Creates a new . + /// The per-operation entity id. + /// The durable execution ARN. + public DurableSerializationContext(string entityId, string durableExecutionArn) + { + EntityId = entityId; + DurableExecutionArn = durableExecutionArn; + } +} diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/IDurableResultSerializer.cs b/Libraries/src/Amazon.Lambda.DurableExecution/IDurableResultSerializer.cs new file mode 100644 index 000000000..86e90b59c --- /dev/null +++ b/Libraries/src/Amazon.Lambda.DurableExecution/IDurableResultSerializer.cs @@ -0,0 +1,44 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using System.IO; + +namespace Amazon.Lambda.DurableExecution; + +/// +/// Optional capability interface a per-operation serializer may implement to receive a +/// when a durable operation result is +/// (de)serialized. When the serializer configured on an operation (for example +/// StepConfig.Serializer) implements this interface, the durable runtime invokes +/// these context-aware overloads; otherwise it falls back to the plain +/// methods. +/// +/// +/// +/// conveys only the value and a stream — +/// it has no way to tell the serializer which operation or execution a value belongs to. +/// Serializers that offload results to external storage (for example +/// FileSystemSerializer) need that identity to build a stable, unique +/// location and to avoid different operations clobbering one another. Serializers that +/// do not need it simply implement and +/// are used unchanged. +/// +/// +/// A type typically implements both +/// and this interface. The durable runtime prefers this interface when present. +/// +/// +public interface IDurableResultSerializer +{ + /// + /// Serializes to , using + /// to identify the operation/execution. + /// + void Serialize(T value, Stream stream, DurableSerializationContext context); + + /// + /// Deserializes a value of type from , + /// using to identify the operation/execution. + /// + T Deserialize(Stream stream, DurableSerializationContext context); +} diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs index 2cef36626..628256eda 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs @@ -53,6 +53,20 @@ internal sealed class ChildContextOperation : DurableOperation // Set once on overflow-replay re-execution; never reset. private bool _suppressTerminalCheckpoint; + /// + /// The exact SUCCEED-checkpoint payload this (non-virtual) child wrote on a fresh, + /// non-overflow success — i.e. S(body), serialized with the child's own + /// OperationId as the entity id. A Nested Map/Parallel + /// parent captures this so it can inline the child's ORIGINAL checkpoint payload + /// verbatim, rather than re-serializing the (already round-tripped) return value. + /// Re-serializing the return value double-transforms a non-round-tripping serializer + /// (fresh vs replay diverge) and, for a context-aware serializer such as + /// FileSystemSerializer, writes a SECOND file at the parent's per-unit id + /// that orphans the child's file. null for virtual (Flat) children, for + /// overflow-replay re-execution, and before a fresh success has been serialized. + /// + internal string? SerializedResultPayload { get; private set; } + public ChildContextOperation( string operationId, string? name, @@ -235,11 +249,73 @@ await EnqueueAsync(new SdkOperationUpdate if (!_isVirtual && !_suppressTerminalCheckpoint) { var serialized = SerializeResult(result); + + // Capture the child's own SUCCEED-checkpoint payload so a Nested Map/Parallel + // parent can inline it verbatim (serialized here with THIS child's OperationId + // as the entity id). Captured before the overflow decision so the parent sees + // the same S(body) it would otherwise re-derive: on overflow the parent inlines + // this (large) payload, overflows in turn, and recovers via ReplayChildren. + SerializedResultPayload = serialized; + // Overflow: result too large to checkpoint inline. Emit an empty // payload + ReplayChildren so replay re-executes this body to recover // the value (mirrors the concurrent-operation overflow strategy). var overflow = Encoding.UTF8.GetByteCount(serialized) > DurableConstants.MaxOperationCheckpointBytes; + // Non-overflow: round-trip the just-written checkpoint payload so the + // value the workflow observes on this fresh execution matches replay + // (where the result is always deserialized from the checkpoint). This + // makes a custom (possibly non-round-tripping) ChildContextConfig.Serializer's + // transform visible in the child-context result on the first run. + // Behavior change: the returned object is no longer the same instance + // the child body produced. See the AutoVer change note. Overflow skips + // this (the payload was stripped; the value is recovered by replay). + // + // Deserialize BEFORE emitting the SUCCEED: a serializer whose deserialize + // fails is then surfaced as a normal child failure (wrapped in + // ChildContextException, with no SUCCEED recorded) instead of throwing + // against an already-SUCCEEDED operation — which would diverge from + // replay and, for a Map/Parallel Nested unit, record the unit Failed + // against its own SUCCEEDED checkpoint. + T roundTripped = result; + if (!overflow) + { + try + { + roundTripped = DeserializeResult(serialized); + } + catch (Exception ex) + { + // The body SUCCEEDED (side effects ran) but its result could not be + // deserialized back. This is TERMINAL — emit a CONTEXT FAIL so the op + // is terminal in the store and its side-effecting body is NOT re-run on + // replay. Previously this threw with NO terminal checkpoint, leaving a + // top-level child STARTED (its body re-runs on replay) and a Nested unit + // recorded Failed against its own — now correctly absent — SUCCEED. This + // mirrors the body-failure catch above and StepOperation.FailStepTerminallyAsync. + // We are inside the non-virtual, non-suppressed block, so FAIL is the correct + // terminal record. If this FAIL enqueue itself throws, it propagates straight + // out (no retry loop, no recursion) — a broken batcher surfaces its own error. + await EnqueueAsync(new SdkOperationUpdate + { + Id = OperationId, + ParentId = ParentId, + Type = OperationTypes.Context, + Action = OperationAction.FAIL, + SubType = _config?.SubType, + Name = Name, + Error = ToSdkError(ex) + }, cancellationToken); + + throw MapFailureException(new ChildContextException(ex.Message, ex) + { + SubType = _config?.SubType, + ErrorType = ex.GetType().FullName, + OriginalStackTrace = ex.StackTrace?.Split(new[] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries).ToList() + }); + } + } + await EnqueueAsync(new SdkOperationUpdate { Id = OperationId, @@ -254,17 +330,9 @@ await EnqueueAsync(new SdkOperationUpdate : null }, cancellationToken); - // Non-overflow: round-trip the just-written checkpoint so the value the - // workflow observes on this fresh execution matches replay (where the - // result is always deserialized from the checkpoint). This makes a - // custom (possibly non-round-tripping) ChildContextConfig.Serializer's - // transform visible in the child-context result on the first run. - // Behavior change: the returned object is no longer the same instance - // the child body produced. See the AutoVer change note. Overflow skips - // this (the payload was stripped; the value is recovered by replay). if (!overflow) { - return DeserializeResult(serialized); + return roundTripped; } } @@ -297,13 +365,15 @@ private T DeserializeResult(string? serialized) if (serialized == null) return default!; var bytes = Encoding.UTF8.GetBytes(serialized); using var ms = new MemoryStream(bytes); - return _serializer.Deserialize(ms); + return LambdaSerializerHelper.Deserialize( + _serializer, ms, new DurableSerializationContext(OperationId, DurableExecutionArn)); } private string SerializeResult(T value) { using var ms = new MemoryStream(); - _serializer.Serialize(value, ms); + LambdaSerializerHelper.Serialize( + _serializer, value, ms, new DurableSerializationContext(OperationId, DurableExecutionArn)); return Encoding.UTF8.GetString(ms.ToArray()); } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ConcurrentOperation.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ConcurrentOperation.cs index 44658262f..01f8c9547 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ConcurrentOperation.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ConcurrentOperation.cs @@ -369,12 +369,30 @@ void SignalShortCircuit() var completionReason = ComputeCompletionReason(items, unitCount); var result = new BatchResult(items, completionReason); - await CheckpointParentResultAsync(result, completionReason, cancellationToken); + // For Nested (non-virtual) units, capture each succeeded child's OWN serialized + // checkpoint payload (S(body)) so the parent inlines it verbatim rather than + // re-serializing the round-tripped return value. Indexed by unit index. Empty/absent + // for Flat, which the parent serializes inline itself. + Dictionary? nestedInlinePayloads = null; + if (!_isVirtual) + { + nestedInlinePayloads = new Dictionary(unitCount); + for (var i = 0; i < unitCount; i++) + { + if (dispatched[i] && slots[i].Status == BatchItemStatus.Succeeded) + nestedInlinePayloads[i] = slots[i].SerializedPayload; + } + } + + // Returns the result to OBSERVE. For Flat (virtual) units on a non-overflow + // fresh success it is round-tripped through the item serializer so the value + // matches replay; otherwise it is the same result instance. + var observed = await CheckpointParentResultAsync(result, completionReason, nestedInlinePayloads, cancellationToken); // Never throw on failure — always return the aggregate result. The caller // inspects CompletionReason / HasFailure or calls ThrowIfError. Matches // the JS/Python/Java SDKs. - return result; + return observed; } /// @@ -500,7 +518,16 @@ private async Task RunUnitAsync( try { var result = await childOp.ExecuteAsync(cancellationToken).ConfigureAwait(false); - slots[index] = new UnitOutcome { Status = BatchItemStatus.Succeeded, Result = result }; + slots[index] = new UnitOutcome + { + Status = BatchItemStatus.Succeeded, + Result = result, + // Nested (non-virtual) children write their own SUCCEED checkpoint and + // expose its exact payload here; capture it so the parent inlines the + // child's ORIGINAL S(body) rather than re-serializing the round-tripped + // return value. Null for Flat children and for child-level overflow. + SerializedPayload = childOp.SerializedResultPayload + }; } catch (ChildContextException ex) { @@ -684,21 +711,27 @@ private CompletionReason ComputeCompletionReason(IReadOnlyList> it return _policy.Evaluate(succeeded, failed, started, totalCount); } - private async Task CheckpointParentResultAsync( + private async Task> CheckpointParentResultAsync( BatchResult result, CompletionReason completionReason, + IReadOnlyDictionary? nestedInlinePayloads, CancellationToken cancellationToken) { - // Local builder: includeInline=true writes per-unit Result/Error inline - // (Flat only); includeInline=false writes the minimal index/name/status - // map (the shape Nested always uses, and the Flat overflow fallback). - // The persisted summary keeps EVERY declared unit — including - // never-dispatched ones tagged STARTED — even though result.All now omits - // them from the user-facing view. Replay/reconstruct and the unit-name - // drift check both loop the declared UnitCount and read per-unit - // status/name from this summary, so it must stay complete. Dispatched - // units are looked up by Index in result.All; the gaps are the - // never-dispatched branches. + // The completion reason to persist, and any Flat units whose fresh-success + // round-trip deserialize failed. Both may be updated after the round-trip below; + // BuildSummary closes over them so a rebuild reflects the terminal failures. + var reasonToPersist = completionReason; + Dictionary? flatRoundTripFailures = null; + + // Local builder: includeInline=true writes per-unit Result/Error inline; false writes + // the minimal index/name/status map (the Flat overflow fallback). Serialization always + // reads from `result` (the ORIGINAL per-unit values) so a Nested unit inlines the + // child's captured S(body) and a Flat unit inlines S(raw) — never the round-tripped + // value, which would double-transform an asymmetric serializer. The persisted summary + // keeps EVERY declared unit — including never-dispatched ones tagged STARTED — even + // though result.All now omits them from the user-facing view. Replay/reconstruct and + // the unit-name drift check both loop the declared UnitCount and read per-unit + // status/name from this summary, so it must stay complete. BatchSummary BuildSummary(bool includeInline) { var byIndex = new Dictionary>(result.All.Count); @@ -707,33 +740,66 @@ BatchSummary BuildSummary(bool includeInline) var s = new BatchSummary { - CompletionReason = SerializeCompletionReason(completionReason), + CompletionReason = SerializeCompletionReason(reasonToPersist), Units = new List(UnitCount) }; for (var i = 0; i < UnitCount; i++) { var (unitName, _) = GetUnit(i); byIndex.TryGetValue(i, out var item); + + // A Flat unit whose fresh-success round-trip deserialize failed is recorded + // Failed (terminal) even though its body succeeded, so replay reads the Error + // and never re-deserializes the poison payload (which would fail identically). + DurableExecutionException? forcedError = null; + if (flatRoundTripFailures != null) + flatRoundTripFailures.TryGetValue(i, out forcedError); + + var status = forcedError != null + ? BatchItemStatus.Failed + : (item?.Status ?? BatchItemStatus.Started); + var unit = new BatchUnitSummary { Index = i, Name = item?.Name ?? unitName, - Status = SerializeStatus(item?.Status ?? BatchItemStatus.Started) + Status = SerializeStatus(status) }; - // Persist each unit's result/error inline on the parent summary — - // for BOTH Nested and Flat units. The service collapses completed - // per-unit child contexts out of the state returned on a later - // (post-operation) resume, so replay cannot recover a Nested unit's - // value from its child checkpoint; the inline copy is the only - // durable source. This mirrors the JS SDK, whose default - // BatchResult serdes serializes the whole `all` array (results - // included) into the parent payload. - if (includeInline && item != null) + + // Persist each unit's result/error inline on the parent summary — for BOTH + // Nested and Flat units. The service collapses completed per-unit child + // contexts out of the state returned on a later (post-operation) resume, so + // replay cannot recover a Nested unit's value from its child checkpoint; the + // inline copy is the only durable source. This mirrors the JS SDK, whose + // default BatchResult serdes serializes the whole `all` array into the parent + // payload. + if (includeInline) { - if (item.Status == BatchItemStatus.Succeeded) - unit.Result = SerializeResult(item.Result); - else if (item.Status == BatchItemStatus.Failed && item.Error != null) + if (forcedError != null) + { + unit.Error = ErrorObject.FromException(forcedError); + } + else if (item != null && item.Status == BatchItemStatus.Succeeded) + { + // Nested: inline the child's ORIGINAL SUCCEED payload (serialized by + // the child at its own OperationId), captured on this fresh run. + // Flat: the virtual child emitted no checkpoint, so the parent is the + // sole writer — serialize the raw item result at "{OperationId}#{i}". + if (_isVirtual) + { + unit.Result = SerializeResult(item.Result, i); + } + else + { + unit.Result = nestedInlinePayloads != null && nestedInlinePayloads.TryGetValue(i, out var p) + ? p + : null; + } + } + else if (item != null && item.Status == BatchItemStatus.Failed && item.Error != null) + { unit.Error = ErrorObject.FromException(item.Error); + } } s.Units.Add(unit); } @@ -749,6 +815,50 @@ BatchSummary BuildSummary(bool includeInline) // Applies to both Nested and Flat now that both inline their results. var overflow = Encoding.UTF8.GetByteCount(payload) > DurableConstants.MaxOperationCheckpointBytes; + + // Result to OBSERVE. Starts as the raw aggregate; the Flat path below replaces it + // with the round-tripped values (and any terminal round-trip failures applied). + var observed = result; + + // Fresh-success round-trip for Flat (virtual) units. A Flat unit's body result is + // returned RAW from ChildContextOperation (a virtual child suppresses its own SUCCEED + // and the round-trip a Nested child performs), whereas ReconstructFromCheckpoints + // rebuilds each value by DESERIALIZING the inline payload. Mirror Step/Child: + // deserialize the just-serialized per-unit payload and observe THAT value, so a + // non-round-tripping ItemSerializer yields identical values on a fresh run and on + // replay. Runs BEFORE the parent SUCCEED is emitted so a deserialize failure is + // handled terminally (below) instead of after a terminal checkpoint exists. + // + // Nested units already round-tripped inside ChildContextOperation and are never + // re-processed here (guarded by _isVirtual). On overflow the inline results are + // stripped and ReplayChildrenAsync recovers each value by re-running the unit body + // (also raw), so leaving the fresh values raw keeps fresh==replay in that case too. + if (_isVirtual && !overflow) + { + observed = RoundTripFlatResults(result, summary, reasonToPersist, out var failures); + + if (failures.Count > 0) + { + // A unit's inline result could not be deserialized on this fresh run. This is + // TERMINAL for that unit (mirrors Step/Child): rather than let it propagate raw + // — which would leave the parent with NO terminal checkpoint and re-run every + // virtual unit body on replay (a deterministic poison loop) — record each such + // unit Failed, recompute the completion reason, and STILL emit the parent + // SUCCEED. Replay then reconstructs the same Failed unit from the rebuilt inline + // summary (Error, not Result) without re-deserializing or re-running it. + flatRoundTripFailures = failures; + reasonToPersist = ComputeCompletionReason(observed.All, UnitCount); + observed = new BatchResult(observed.All, reasonToPersist); + + // Rebuild from the ORIGINAL result (raw survivors) with the failed units forced + // to Error, so surviving Succeeded units still inline S(raw) — deserialized to + // the round-tripped value on replay, matching `observed`. + summary = BuildSummary(includeInline: true); + payload = JsonSerializer.Serialize(summary, BatchJsonContext.Default.BatchSummary); + overflow = Encoding.UTF8.GetByteCount(payload) > DurableConstants.MaxOperationCheckpointBytes; + } + } + if (overflow) { summary = BuildSummary(includeInline: false); @@ -772,6 +882,91 @@ await EnqueueAsync(new SdkOperationUpdate ? new SdkContextOptions { ReplayChildren = true } : null }, cancellationToken); + + return observed; + } + + /// + /// Rebuilds with each SUCCEEDED unit's value replaced by the + /// round-trip of its just-serialized inline payload, so the observed per-unit value on a + /// Flat (virtual) fresh success matches what + /// rebuilds from the same inline payload on replay. Only used on the Flat, non-overflow + /// path — mirrors the fresh-success round-trip in and + /// . Failed / never-dispatched units are left + /// untouched. + /// + /// A per-unit deserialize failure is TERMINAL for that unit: it is added to + /// and materialized as a Failed , + /// so the caller can record it inline and still emit the parent SUCCEED (avoiding a raw + /// propagation that would leave no terminal checkpoint and re-run every unit on replay). + /// + /// + private BatchResult RoundTripFlatResults( + BatchResult result, + BatchSummary inlineSummary, + CompletionReason completionReason, + out Dictionary failures) + { + failures = new Dictionary(); + + var serializedByIndex = new Dictionary(inlineSummary.Units.Count); + foreach (var u in inlineSummary.Units) + serializedByIndex[u.Index] = u.Result; + + var items = new List>(result.All.Count); + foreach (var item in result.All) + { + if (item.Status == BatchItemStatus.Succeeded + && serializedByIndex.TryGetValue(item.Index, out var serialized) + && serialized != null) + { + T value; + try + { + // Symmetric with SerializeResult(item.Result, i) and with + // ReconstructFromCheckpoints' DeserializeResult(..., "{OperationId}#{i}"). + value = DeserializeResult(serialized, $"{OperationId}#{item.Index}"); + } + catch (Exception ex) + { + // Terminal for this unit — record it Failed. The caller rebuilds the inline + // summary with this Error (not a Result) and still emits the parent SUCCEED, + // so replay reads a terminal Error instead of re-deserializing the same + // poison payload or re-running the body. + var wrapped = new ChildContextException(ex.Message, ex) + { + SubType = ChildSubType, + ErrorType = ex.GetType().FullName, + OriginalStackTrace = ex.StackTrace?.Split(new[] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries).ToList() + }; + failures[item.Index] = wrapped; + items.Add(new BatchItem + { + Index = item.Index, + Name = item.Name, + Status = BatchItemStatus.Failed, + Result = default, + Error = wrapped + }); + continue; + } + + items.Add(new BatchItem + { + Index = item.Index, + Name = item.Name, + Status = item.Status, + Result = value, + Error = item.Error + }); + } + else + { + items.Add(item); + } + } + + return new BatchResult(items, completionReason); } private IBatchResult ReconstructFromCheckpoints(Operation parent) @@ -826,7 +1021,15 @@ private IBatchResult ReconstructFromCheckpoints(Operation parent) // whose child op is still present in state). if (status == BatchItemStatus.Succeeded && summaryEntry?.Result != null) { - unitResult = DeserializeResult(summaryEntry.Result); + // The inline payload's entity id depends on who wrote it: a Flat unit was + // serialized by THIS parent at "{OperationId}#{i}", whereas a Nested unit's + // inline payload IS the child's own checkpoint payload, serialized by the + // child at childOpId. Deserialize each with the entity id it was written with + // so a context-aware serializer (e.g. FileSystemSerializer) resolves the same + // external location — and, for Nested, the SAME file the child wrote (no + // orphan, no second file). + var inlineEntityId = _isVirtual ? $"{OperationId}#{i}" : childOpId; + unitResult = DeserializeResult(summaryEntry.Result, inlineEntityId); } else if (status == BatchItemStatus.Failed && summaryEntry?.Error != null) { @@ -841,7 +1044,10 @@ private IBatchResult ReconstructFromCheckpoints(Operation parent) } else if (status == BatchItemStatus.Succeeded && childOp?.ContextDetails?.Result != null) { - unitResult = DeserializeResult(childOp.ContextDetails.Result); + // This payload is the child's OWN context checkpoint, written by + // ChildContextOperation using the child op's id as EntityId — so + // deserialize with childOpId, not the parent's per-unit id. + unitResult = DeserializeResult(childOp.ContextDetails.Result, childOpId); } else if (status == BatchItemStatus.Failed && childOp?.ContextDetails?.Error != null) { @@ -930,11 +1136,12 @@ private static BatchItemStatus InferStatusFromChildOp(Operation? childOp) _ => CompletionReason.AllCompleted }; - private T DeserializeResult(string serialized) + private T DeserializeResult(string serialized, string entityId) { var bytes = Encoding.UTF8.GetBytes(serialized); using var ms = new MemoryStream(bytes); - return Serializer.Deserialize(ms); + return LambdaSerializerHelper.Deserialize( + Serializer, ms, new DurableSerializationContext(entityId, DurableExecutionArn)); } /// @@ -943,10 +1150,12 @@ private T DeserializeResult(string serialized) /// serialization a Nested unit's would /// have written to its own checkpoint. /// - private string SerializeResult(T? value) + private string SerializeResult(T? value, int index) { using var ms = new MemoryStream(); - Serializer.Serialize(value!, ms); + LambdaSerializerHelper.Serialize( + Serializer, value!, ms, + new DurableSerializationContext($"{OperationId}#{index}", DurableExecutionArn)); return Encoding.UTF8.GetString(ms.ToArray()); } @@ -960,5 +1169,14 @@ private struct UnitOutcome public BatchItemStatus Status; public T? Result; public DurableExecutionException? Error; + + /// + /// For a Nested (non-virtual) succeeded unit: the child's OWN serialized + /// SUCCEED-checkpoint payload (S(body)), captured from + /// so the parent can + /// inline it verbatim instead of re-serializing the round-tripped return value. + /// null for Flat units, failures, and overflow. + /// + public string? SerializedPayload; } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/LambdaSerializerHelper.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/LambdaSerializerHelper.cs index dfebe820e..261d2a33a 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/LambdaSerializerHelper.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/LambdaSerializerHelper.cs @@ -1,6 +1,7 @@ // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. // SPDX-License-Identifier: Apache-2.0 +using System.IO; using Amazon.Lambda.Core; namespace Amazon.Lambda.DurableExecution.Internal; @@ -16,4 +17,32 @@ internal static class LambdaSerializerHelper public static ILambdaSerializer GetRequired(ILambdaContext lambdaContext) => lambdaContext.Serializer ?? throw new InvalidOperationException(MissingSerializerMessage); + + /// + /// Serializes a durable operation result. If implements + /// , the context-aware overload is used so the + /// serializer can key external storage by operation/execution; otherwise the plain + /// path is used (behavior identical to before). + /// + public static void Serialize( + ILambdaSerializer serializer, T value, Stream stream, in DurableSerializationContext context) + { + if (serializer is IDurableResultSerializer durable) + durable.Serialize(value, stream, context); + else + serializer.Serialize(value, stream); + } + + /// + /// Deserializes a durable operation result. Mirrors : + /// uses the overload when available, else the + /// plain path. + /// + public static T Deserialize( + ILambdaSerializer serializer, Stream stream, in DurableSerializationContext context) + { + if (serializer is IDurableResultSerializer durable) + return durable.Deserialize(stream, context); + return serializer.Deserialize(stream); + } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/StepOperation.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/StepOperation.cs index bb580fbfd..11c486c5d 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/StepOperation.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/StepOperation.cs @@ -215,6 +215,7 @@ private async Task ExecuteFunc(int attemptNumber, CancellationToken cancellat using var linked = CancellationTokenSource.CreateLinkedTokenSource( cancellationToken, _workflowCancellation.Token); + T result; try { var stepContext = new StepContext(OperationId, attemptNumber, _logger); @@ -223,7 +224,6 @@ private async Task ExecuteFunc(int attemptNumber, CancellationToken cancellat // lines with the operation id, name, and current attempt. Wrap // only the user-func call — checkpoint emission shouldn't carry // step metadata into any side-channel logging. - T result; using (_logger.BeginScope(new Dictionary { ["operationId"] = OperationId, @@ -233,8 +233,74 @@ private async Task ExecuteFunc(int attemptNumber, CancellationToken cancellat { result = await _func(stepContext, linked.Token); } + } + catch (OperationCanceledException) when (linked.IsCancellationRequested) + { + // Cancellation owned by the linked source (caller-cancel or workflow + // shutdown). Do NOT checkpoint FAIL and do NOT consult the retry + // strategy — the termination signal that fired (if any) owns the + // suspend/abort decision; an upstream caller-cancel propagates up + // as a fault on the workflow user task. + throw; + } + catch (Exception ex) + { + // The user step body itself threw. Funnel into the retry/fail decision + // tree: may checkpoint RETRY and suspend (Pending), or checkpoint FAIL + // and rethrow to user. A user-thrown OperationCanceledException unrelated + // to our linked token falls through here and is treated as a normal + // step failure. Retry is safe here precisely because the body has NOT + // recorded a success — re-running it is the intended behavior. + return await HandleStepFailureAsync(ex, attemptNumber, cancellationToken); + } - var serialized = SerializeResult(result); + // The user step body succeeded. Serialize the result and round-trip it + // through the (possibly custom) serializer so the value the workflow + // observes on this fresh execution is the deserialized-from-checkpoint + // value, exactly as it would be on replay. This makes a custom (possibly + // non-round-tripping) StepConfig.Serializer's transform visible in the step + // result on the first run, not just on replay. Behavior change: the returned + // object is no longer the same instance the step body produced. See the + // AutoVer change note. + // + // CRITICAL: a failure from here on is TERMINAL and must NOT enter the retry + // decision. The side-effecting body has already run to completion; routing a + // serialize/round-trip failure through HandleStepFailureAsync (with a non-null + // RetryStrategy) would re-invoke the already-succeeded body on the next + // attempt — duplicating side effects and looping until attempts exhaust. We + // mirror ChildContextOperation, which catches its round-trip deserialize + // distinctly and fails terminally (no retry). Deserialize BEFORE emitting the + // SUCCEED so a broken serializer surfaces as a terminal failure with no + // terminal checkpoint recorded, rather than throwing against an already- + // SUCCEEDED operation (which would emit a second terminal checkpoint and + // diverge from replay, which returns the cached success). + string serialized; + T roundTripped; + try + { + serialized = SerializeResult(result); + roundTripped = DeserializeResult(serialized); + } + catch (Exception ex) + { + return await FailStepTerminallyAsync(ex, cancellationToken); + } + + // The post-round-trip SUCCEED enqueue. This sits AFTER the retryable + // try/catch, so a failure here must NOT re-enter the retry strategy — the + // side-effecting body has already run to completion. Route a non-OCE + // failure through the SAME terminal path as the round-trip failure just + // above (FailStepTerminallyAsync: FAIL checkpoint + StepException) so a + // post-success enqueue failure is handled consistently rather than + // propagating raw with no terminal record. + // + // Recursion guard: FailStepTerminallyAsync emits the terminal FAIL via this + // same EnqueueAsync. If the batcher is broken and that FAIL emit ALSO throws, + // its exception propagates straight out of FailStepTerminallyAsync — it never + // re-enters this SUCCEED path — so a broken batcher surfaces its error + // instead of looping. + try + { await EnqueueAsync(new SdkOperationUpdate { Id = OperationId, @@ -245,33 +311,21 @@ await EnqueueAsync(new SdkOperationUpdate Name = Name, Payload = serialized }, cancellationToken); - - // Round-trip the just-written checkpoint so the value the workflow - // observes on this fresh execution is the deserialized-from-checkpoint - // value, exactly as it would be on replay. This makes a custom - // (possibly non-round-tripping) StepConfig.Serializer's transform - // visible in the step result on the first run, not just on replay. - // Behavior change: the returned object is no longer the same instance - // the step body produced. See the AutoVer change note. - return DeserializeResult(serialized); } - catch (OperationCanceledException) when (linked.IsCancellationRequested) + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { - // Cancellation owned by the linked source (caller-cancel or workflow - // shutdown). Do NOT checkpoint FAIL and do NOT consult the retry - // strategy — the termination signal that fired (if any) owns the - // suspend/abort decision; an upstream caller-cancel propagates up - // as a fault on the workflow user task. + // Caller-cancel during the SUCCEED flush owns the outcome — propagate + // untouched; do NOT synthesize a terminal FAIL. This enqueue observes only + // the caller token (not the linked workflow-shutdown token), so a + // workflow-shutdown OCE cannot originate here. throw; } catch (Exception ex) { - // Funnel into the retry/fail decision tree. May checkpoint RETRY and - // suspend (Pending), or checkpoint FAIL and rethrow to user. A user- - // thrown OperationCanceledException unrelated to our linked token - // falls through here and is treated as a normal step failure. - return await HandleStepFailureAsync(ex, attemptNumber, cancellationToken); + return await FailStepTerminallyAsync(ex, cancellationToken); } + + return roundTripped; } /// @@ -313,6 +367,20 @@ await EnqueueAsync(new SdkOperationUpdate } } + return await FailStepTerminallyAsync(ex, cancellationToken); + } + + /// + /// Fails the step terminally: emits a FAIL checkpoint and throws + /// WITHOUT consulting the retry strategy. Used both + /// as the terminal tail of (retries + /// exhausted / no strategy) and directly for a post-success failure (an + /// asymmetric serializer whose serialize/round-trip throws). Retry must not be + /// consulted for the latter: the side-effecting step body has already run, so + /// re-invoking it would duplicate side effects. + /// + private async Task FailStepTerminallyAsync(Exception ex, CancellationToken cancellationToken) + { await EnqueueAsync(new SdkOperationUpdate { Id = OperationId, @@ -335,13 +403,15 @@ private T DeserializeResult(string? serialized) if (serialized == null) return default!; var bytes = Encoding.UTF8.GetBytes(serialized); using var ms = new MemoryStream(bytes); - return _serializer.Deserialize(ms); + return LambdaSerializerHelper.Deserialize( + _serializer, ms, new DurableSerializationContext(OperationId, DurableExecutionArn)); } private string SerializeResult(T value) { using var ms = new MemoryStream(); - _serializer.Serialize(value, ms); + LambdaSerializerHelper.Serialize( + _serializer, value, ms, new DurableSerializationContext(OperationId, DurableExecutionArn)); return Encoding.UTF8.GetString(ms.ToArray()); } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/WaitForConditionOperation.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/WaitForConditionOperation.cs index 508ee2091..2345eda9d 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/WaitForConditionOperation.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/WaitForConditionOperation.cs @@ -308,7 +308,8 @@ private TState DeserializeState(string? serialized) if (serialized == null) return default!; var bytes = Encoding.UTF8.GetBytes(serialized); using var ms = new MemoryStream(bytes); - return _serializer.Deserialize(ms); + return LambdaSerializerHelper.Deserialize( + _serializer, ms, new DurableSerializationContext(OperationId, DurableExecutionArn)); } private TState DeserializeStateOrInitial(string? serialized) @@ -318,7 +319,8 @@ private TState DeserializeStateOrInitial(string? serialized) { var bytes = Encoding.UTF8.GetBytes(serialized); using var ms = new MemoryStream(bytes); - return _serializer.Deserialize(ms); + return LambdaSerializerHelper.Deserialize( + _serializer, ms, new DurableSerializationContext(OperationId, DurableExecutionArn)); } catch (Exception ex) { @@ -336,7 +338,8 @@ private TState DeserializeStateOrInitial(string? serialized) private string SerializeState(TState value) { using var ms = new MemoryStream(); - _serializer.Serialize(value, ms); + LambdaSerializerHelper.Serialize( + _serializer, value, ms, new DurableSerializationContext(OperationId, DurableExecutionArn)); return Encoding.UTF8.GetString(ms.ToArray()); } @@ -361,7 +364,8 @@ private Exception BuildFailureException(Operation failedOp) { var bytes = Encoding.UTF8.GetBytes(lastStatePayload); using var ms = new MemoryStream(bytes); - lastState = _serializer.Deserialize(ms); + lastState = LambdaSerializerHelper.Deserialize( + _serializer, ms, new DurableSerializationContext(OperationId, DurableExecutionArn)); } catch (Exception deserEx) { diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/MapConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/MapConfig.cs index 9080e656c..809580d2d 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/MapConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/MapConfig.cs @@ -93,5 +93,21 @@ public int? MaxConcurrency /// (statuses / completion reason), and durable operations inside an item's body use /// their own configuration. /// + /// + /// Size-dependent transform on overflow (Flat only). Under + /// each item's result is round-tripped through this + /// serializer on a fresh (non-replay) success before it enters the result, so a + /// serializer that transforms the value (e.g. redaction, canonicalization) has that + /// transform reflected in the returned item — but only while the aggregated + /// per-item payloads fit inline in a single checkpoint (≈256 KB). If they are + /// large enough to overflow, the inline results are stripped and each value is instead + /// recovered by re-running the item body on replay, which does not apply the + /// round-trip transform. A serializer whose Deserialize is not a pure inverse of + /// its Serialize will therefore observe a size-dependent difference (transform + /// applied for small results, skipped for overflowed ones). Prefer a round-tripping + /// serializer, or do not depend on the transform being applied to results large enough + /// to overflow the checkpoint. (Under the same + /// size-dependent caveat is documented on each item's child-context result.) + /// public ILambdaSerializer? ItemSerializer { get; set; } } diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/ParallelConfig.cs b/Libraries/src/Amazon.Lambda.DurableExecution/ParallelConfig.cs index 1fec4216f..80fe393a9 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/ParallelConfig.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/ParallelConfig.cs @@ -76,5 +76,21 @@ public int? MaxConcurrency /// per-branch result — not the aggregated batch envelope (statuses / completion /// reason) — and durable operations inside a branch use their own configuration. /// + /// + /// Size-dependent transform on overflow (Flat only). Under + /// each branch's result is round-tripped through this + /// serializer on a fresh (non-replay) success before it enters the result, so a + /// serializer that transforms the value (e.g. redaction, canonicalization) has that + /// transform reflected in the returned branch — but only while the aggregated + /// per-branch payloads fit inline in a single checkpoint (≈256 KB). If they are + /// large enough to overflow, the inline results are stripped and each value is instead + /// recovered by re-running the branch body on replay, which does not apply the + /// round-trip transform. A serializer whose Deserialize is not a pure inverse of + /// its Serialize will therefore observe a size-dependent difference (transform + /// applied for small results, skipped for overflowed ones). Prefer a round-tripping + /// serializer, or do not depend on the transform being applied to results large enough + /// to overflow the checkpoint. (Under the same + /// size-dependent caveat is documented on each branch's child-context result.) + /// public ILambdaSerializer? ItemSerializer { get; set; } } diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/ChildContextOperationTests.cs b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/ChildContextOperationTests.cs index 04fbeba29..0a7e9154d 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/ChildContextOperationTests.cs +++ b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/ChildContextOperationTests.cs @@ -1,6 +1,7 @@ // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. // SPDX-License-Identifier: Apache-2.0 +using Amazon.Lambda.Core; using Amazon.Lambda.DurableExecution; using Amazon.Lambda.DurableExecution.Internal; using Amazon.Lambda.Serialization.SystemTextJson; @@ -31,6 +32,133 @@ private static (DurableContext context, RecordingBatcher recorder, TerminationMa return (context, recorder, tm, state); } + /// + /// A non-round-tripping serializer: serialize is plain JSON, but deserialize marks a + /// string result with a "[rt]" prefix. Lets a test observe whether the fresh-success + /// round-trip transform was applied. + /// + private sealed class MarkingSerializer : ILambdaSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + public void Serialize(T response, Stream responseStream) => _inner.Serialize(response, responseStream); + public T Deserialize(Stream requestStream) + { + var value = _inner.Deserialize(requestStream); + if (value is string s) return (T)(object)("[rt]" + s); + return value; + } + } + + [Fact] + public async Task RunInChildContextAsync_CustomSerializerTransform_AppliedInline_SkippedOnOverflow() + { + // Pins the documented size-dependent limitation on ChildContextConfig.Serializer: + // the fresh-success round-trip transform is reflected in the returned result ONLY + // when the payload fits inline. On overflow the result is recovered by re-running + // the body (round-trip skipped), so a non-round-tripping serializer diverges by size. + var marking = new MarkingSerializer(); + + // Small result: fits inline -> round-trip transform is applied. + var (smallCtx, _, _, _) = CreateContext(); + var small = await smallCtx.RunInChildContextAsync( + async (_, _) => { await Task.CompletedTask; return "x"; }, + name: "small", + config: new ChildContextConfig { Serializer = marking }); + Assert.Equal("[rt]x", small); + + // Large result: overflows the checkpoint -> transform is NOT applied (value raw). + var big = new string('a', DurableConstants.MaxOperationCheckpointBytes + 1024); + var (bigCtx, bigRecorder, _, _) = CreateContext(); + var large = await bigCtx.RunInChildContextAsync( + async (_, _) => { await Task.CompletedTask; return big; }, + name: "big", + config: new ChildContextConfig { Serializer = marking }); + Assert.Equal(big, large); // NOT "[rt]" + big + Assert.DoesNotContain("[rt]", large); + + // Confirm it really took the overflow path: empty payload + ReplayChildren. + await bigRecorder.Batcher.DrainAsync(); + var succeed = bigRecorder.Flushed.Single(o => o.Type == "CONTEXT" && o.Action == "SUCCEED"); + Assert.Equal(string.Empty, succeed.Payload); + } + + /// + /// A serializer that serializes normally (plain JSON) but ALWAYS throws on deserialize — + /// simulates an asymmetric/broken ChildContextConfig.Serializer that cannot read back its + /// own output. + /// + private sealed class ThrowOnDeserializeSerializer : ILambdaSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + public void Serialize(T response, Stream responseStream) => _inner.Serialize(response, responseStream); + public T Deserialize(Stream requestStream) => + throw new InvalidOperationException("ThrowOnDeserializeSerializer: cannot deserialize"); + } + + [Fact] + public async Task RunInChildContextAsync_RoundTripDeserializeFails_CheckpointsFailAndBodyNotReRunOnReplay() + { + // Comment 2(b): when a non-virtual child's fresh-success round-trip DESERIALIZE fails, + // the child fails TERMINALLY — emitting CONTEXT FAIL — so the op is terminal in the + // store and its side-effecting body is NOT re-run on replay. Previously this threw + // with NO FAIL checkpoint, leaving the child STARTED (body re-runs on replay). + var (context, recorder, _, _) = CreateContext(); + + var executed = 0; + var ex = await Assert.ThrowsAsync(() => + context.RunInChildContextAsync( + async (_, _) => { executed++; await Task.CompletedTask; return "x"; }, + name: "phase", + config: new ChildContextConfig { Serializer = new ThrowOnDeserializeSerializer() })); + + Assert.Equal(1, executed); // body ran exactly once + Assert.Equal("System.InvalidOperationException", ex.ErrorType); + Assert.NotNull(ex.OriginalStackTrace); + Assert.NotEmpty(ex.OriginalStackTrace!); + + await recorder.Batcher.DrainAsync(); + var contextActions = recorder.Flushed + .Where(o => o.Type == "CONTEXT") + .Select(o => o.Action.ToString()) + .ToArray(); + // START then FAIL — never SUCCEED. The FAIL is the new terminal record. + Assert.Equal(new[] { "START", "FAIL" }, contextActions); + + // ---- Replay from that FAILED checkpoint: the body is NOT re-run. ---- + var (replayCtx, replayRec, _, _) = CreateContext(new InitialExecutionState + { + Operations = new List + { + new() + { + Id = IdAt(1), + Type = OperationTypes.Context, + Status = OperationStatuses.Failed, + Name = "phase", + ContextDetails = new ContextDetails + { + Error = new ErrorObject + { + ErrorType = "System.InvalidOperationException", + ErrorMessage = "ThrowOnDeserializeSerializer: cannot deserialize" + } + } + } + } + }); + + var reran = false; + await Assert.ThrowsAsync(() => + replayCtx.RunInChildContextAsync( + async (_, _) => { reran = true; await Task.CompletedTask; return "x"; }, + name: "phase", + config: new ChildContextConfig { Serializer = new ThrowOnDeserializeSerializer() })); + + Assert.False(reran); // poison body never re-runs + await replayRec.Batcher.DrainAsync(); + Assert.Empty(replayRec.Flushed); // already terminal — no new checkpoint + } + [Fact] public async Task RunInChildContextAsync_FreshExecution_RunsFuncAndCheckpoints() { diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableContextTests.cs b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableContextTests.cs index 5c9117b8e..bd253bc90 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableContextTests.cs +++ b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableContextTests.cs @@ -740,6 +740,128 @@ public async Task StepAsync_FailsNoRetryStrategy_CheckpointsFail() Assert.Equal("permanent", ex.Message); } + /// + /// A serializer that serializes fine but throws on deserialize — models an + /// asymmetric / broken whose fresh-success + /// round-trip fails. Counts calls so the test can assert the step body ran once. + /// + private sealed class DeserializeThrowsSerializer : ILambdaSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + public int SerializeCount { get; private set; } + public int DeserializeCount { get; private set; } + + public void Serialize(T response, Stream responseStream) + { + SerializeCount++; + _inner.Serialize(response, responseStream); + } + + public T Deserialize(Stream requestStream) + { + DeserializeCount++; + throw new InvalidOperationException("asymmetric serializer: cannot read back its own output"); + } + } + + [Fact] + public async Task StepAsync_FreshSuccessRoundTripThrows_FailsTerminally_DoesNotRetryOrRerunBody() + { + // Regression: the fresh-success round-trip DeserializeResult must NOT be routed + // through the retry strategy. The side-effecting body has already succeeded, so + // retrying would re-invoke it (duplicate side effects) and loop until attempts + // exhaust. It must fail TERMINALLY (FAIL checkpoint + StepException, no RETRY). + var tm = new TerminationManager(); + var state = new ExecutionState(); + state.LoadFromCheckpoint(null); + var idGen = new OperationIdGenerator(); + var lambdaContext = CreateLambdaContext(); + var recorder = new RecordingBatcher(); + var context = new DurableContext(state, tm, new WorkflowCancellation(tm), idGen, "arn:test", lambdaContext, recorder.Batcher); + + var brokenSerializer = new DeserializeThrowsSerializer(); + var bodyRuns = 0; + + var ex = await Assert.ThrowsAsync(() => + context.StepAsync( + async (_, _) => { bodyRuns++; await Task.CompletedTask; return "ok"; }, + name: "roundtrip_step", + config: new StepConfig + { + Serializer = brokenSerializer, + // A strategy that WOULD retry if consulted. The whole point of the + // fix is that this is NOT consulted for a post-success failure. + RetryStrategy = RetryStrategy.Exponential( + maxAttempts: 3, + initialDelay: TimeSpan.FromSeconds(5), + jitter: JitterStrategy.None) + })); + + // Body ran exactly once — no retry re-invoked the succeeded, side-effecting body. + Assert.Equal(1, bodyRuns); + // The round-trip actually happened once (serialize + the throwing deserialize). + Assert.Equal(1, brokenSerializer.SerializeCount); + Assert.Equal(1, brokenSerializer.DeserializeCount); + // Terminal, not suspended-for-retry. + Assert.False(tm.IsTerminated); + Assert.Equal("asymmetric serializer: cannot read back its own output", ex.Message); + + // A single terminal FAIL checkpoint was recorded — never a RETRY. + await recorder.Batcher.DrainAsync(); + Assert.DoesNotContain(recorder.Flushed, o => o.Action == "RETRY"); + Assert.Contains(recorder.Flushed, o => o.Action == "FAIL"); + } + + [Fact] + public async Task StepAsync_SuccessCheckpointEnqueueFails_NotRetried_PropagatesWithoutRecursing() + { + // Comment 2: the post-round-trip SUCCEED enqueue sits AFTER the retryable + // try/catch. A failure there must be routed through the terminal path + // (FailStepTerminallyAsync), NOT the retry strategy — the side-effecting body + // has already succeeded, so it must not be re-invoked. And because + // CheckpointBatcher is terminal-on-failure, the follow-on terminal FAIL emit + // fast-fails with the same error; that failure must PROPAGATE rather than + // re-entering the SUCCEED path (the recursion guard), and must not loop. + var tm = new TerminationManager(); + var state = new ExecutionState(); + state.LoadFromCheckpoint(null); + var idGen = new OperationIdGenerator(); + var lambdaContext = CreateLambdaContext(); + + // Flush fails only on the STEP SUCCEED; the fire-and-forget START (AtLeastOnce + // semantics) flushes fine first. + var batcher = new CheckpointBatcher("token", (token, ops, ct) => + ops.Any(o => o.Type == "STEP" && o.Action == "SUCCEED") + ? Task.FromException(new InvalidOperationException("succeed-enqueue boom")) + : Task.FromResult(token)); + var context = new DurableContext(state, tm, new WorkflowCancellation(tm), idGen, "arn:test", lambdaContext, batcher); + + var bodyRuns = 0; + var ex = await Assert.ThrowsAsync(() => + context.StepAsync( + async (_, _) => { bodyRuns++; await Task.CompletedTask; return "ok"; }, + name: "succeed_enqueue_fails", + config: new StepConfig + { + // A strategy that WOULD retry if (wrongly) consulted for this + // post-success enqueue failure. + RetryStrategy = RetryStrategy.Exponential( + maxAttempts: 3, + initialDelay: TimeSpan.FromSeconds(5), + jitter: JitterStrategy.None) + })); + + // The broken terminal-FAIL emit's error propagated (recursion guard's + // "propagate" branch) rather than looping. + Assert.Equal("succeed-enqueue boom", ex.Message); + // Body ran exactly once — not routed through retry, no recursive re-run. + Assert.Equal(1, bodyRuns); + // Not suspended for a retry. + Assert.False(tm.IsTerminated); + + await batcher.DisposeAsync(); + } + [Fact] public async Task StepAsync_RetryExhausted_CheckpointsFail() { diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableResultSerializerContextTests.cs b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableResultSerializerContextTests.cs new file mode 100644 index 000000000..1f8211347 --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableResultSerializerContextTests.cs @@ -0,0 +1,273 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using Amazon.Lambda.Core; +using Amazon.Lambda.DurableExecution.Internal; +using Amazon.Lambda.Serialization.SystemTextJson; +using Amazon.Lambda.TestUtilities; +using Xunit; + +namespace Amazon.Lambda.DurableExecution.Tests; + +/// +/// Verifies that when a per-operation serializer implements +/// , the durable runtime invokes the context-aware +/// overloads and supplies the operation's identity (EntityId = operation id, and +/// the durable execution ARN). Also verifies that a plain +/// still works via the fallback path. +/// +public class DurableResultSerializerContextTests +{ + private const string TestArn = "arn:aws:lambda:us-east-1:123:durable-execution:test"; + + private static string IdAt(int position) => OperationIdGenerator.HashOperationId(position.ToString()); + + private static DurableContext CreateContext(ILambdaSerializer global, InitialExecutionState? initial = null) + { + var state = new ExecutionState(); + state.LoadFromCheckpoint(initial); + var tm = new TerminationManager(); + var idGen = new OperationIdGenerator(); + var lambdaContext = new TestLambdaContext { Serializer = global }; + return new DurableContext(state, tm, new WorkflowCancellation(tm), idGen, TestArn, lambdaContext); + } + + /// + /// A serializer that captures the it is + /// handed on each context-aware call, delegating the actual bytes to JSON. + /// + private sealed class CapturingSerializer : ILambdaSerializer, IDurableResultSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + public List SerializeContexts { get; } = new(); + public List DeserializeContexts { get; } = new(); + + public void Serialize(T value, Stream stream, DurableSerializationContext context) + { + SerializeContexts.Add(context); + _inner.Serialize(value, stream); + } + + public T Deserialize(Stream stream, DurableSerializationContext context) + { + DeserializeContexts.Add(context); + return _inner.Deserialize(stream); + } + + // Plain path — should not be exercised while dispatch prefers the context overload. + void ILambdaSerializer.Serialize(T response, Stream responseStream) => _inner.Serialize(response, responseStream); + T ILambdaSerializer.Deserialize(Stream requestStream) => _inner.Deserialize(requestStream); + } + + /// Plain serializer (no ) that just counts. + private sealed class PlainSpy : ILambdaSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + public int SerializeCount { get; private set; } + public void Serialize(T response, Stream responseStream) { SerializeCount++; _inner.Serialize(response, responseStream); } + public T Deserialize(Stream requestStream) => _inner.Deserialize(requestStream); + } + + [Fact] + public async Task Step_PassesOperationIdAndArn_AsContext() + { + var ser = new CapturingSerializer(); + var ctx = CreateContext(new DefaultLambdaJsonSerializer()); + + var result = await ctx.StepAsync( + async (_, _) => { await Task.CompletedTask; return 42; }, + name: "s", + config: new StepConfig { Serializer = ser }); + + Assert.Equal(42, result); + // Fresh success round-trips (serialize + deserialize), both carry the same context. + Assert.All(ser.SerializeContexts, c => { Assert.Equal(IdAt(1), c.EntityId); Assert.Equal(TestArn, c.DurableExecutionArn); }); + Assert.All(ser.DeserializeContexts, c => { Assert.Equal(IdAt(1), c.EntityId); Assert.Equal(TestArn, c.DurableExecutionArn); }); + Assert.NotEmpty(ser.SerializeContexts); + // NotEmpty guards the Assert.All above from passing vacuously: a regression that + // dropped the fresh-success deserialize round-trip would leave DeserializeContexts + // empty and must fail here. + Assert.NotEmpty(ser.DeserializeContexts); + } + + [Fact] + public async Task ChildContext_PassesOperationIdAndArn_AsContext() + { + var ser = new CapturingSerializer(); + var ctx = CreateContext(new DefaultLambdaJsonSerializer()); + + var result = await ctx.RunInChildContextAsync( + async (_, _) => { await Task.CompletedTask; return 99; }, + name: "child", + config: new ChildContextConfig { Serializer = ser }); + + Assert.Equal(99, result); + Assert.NotEmpty(ser.SerializeContexts); + Assert.All(ser.SerializeContexts, c => { Assert.Equal(IdAt(1), c.EntityId); Assert.Equal(TestArn, c.DurableExecutionArn); }); + // The child context also round-trips on fresh success; assert the deserialize + // side both ran (NotEmpty) and carried the operation identity, so a regression + // dropping the round-trip is caught here too. + Assert.NotEmpty(ser.DeserializeContexts); + Assert.All(ser.DeserializeContexts, c => { Assert.Equal(IdAt(1), c.EntityId); Assert.Equal(TestArn, c.DurableExecutionArn); }); + } + + [Fact] + public async Task Map_Nested_PassesChildOperationEntityIds() + { + // Nested (the DEFAULT NestingType): each unit is a NON-virtual child that serializes + // its OWN result at its OWN operation id (childOpId), and the parent inlines that + // child payload verbatim. So the per-item serialize contexts carry the child op ids, + // NOT the parent's per-unit "{parentOpId}#{i}" ids. (Before the comment-1 fix the + // parent re-serialized each Nested unit at "{parentOpId}#{i}" — a DIFFERENT entity id + // than the child wrote at, which orphaned a context-aware serializer's file and + // double-transformed a non-round-tripping serializer.) + var ser = new CapturingSerializer(); + var ctx = CreateContext(new DefaultLambdaJsonSerializer()); + + var result = await ctx.MapAsync( + new[] { "a", "b" }, + async (_, item, _, _, _) => { await Task.CompletedTask; return item.ToUpperInvariant(); }, + name: "map", + config: new MapConfig { ItemSerializer = ser }); + + Assert.Equal(2, result.SuccessCount); + var parentOpId = IdAt(1); + var child0 = OperationIdGenerator.HashOperationId($"{parentOpId}-1"); + var child1 = OperationIdGenerator.HashOperationId($"{parentOpId}-2"); + var ids = ser.SerializeContexts.Select(c => c.EntityId).Distinct().ToList(); + Assert.Contains(child0, ids); + Assert.Contains(child1, ids); + // The stale parent per-unit ids must NOT appear (the double-serialize is gone). + Assert.DoesNotContain($"{parentOpId}#0", ids); + Assert.DoesNotContain($"{parentOpId}#1", ids); + Assert.All(ser.SerializeContexts, c => Assert.Equal(TestArn, c.DurableExecutionArn)); + } + + [Fact] + public async Task Map_Flat_PassesDistinctPerItemEntityIds() + { + // Flat (virtual) units emit no child checkpoint, so the PARENT serializes each unit + // result inline at its per-unit id "{parentOpId}#{i}". + var ser = new CapturingSerializer(); + var ctx = CreateContext(new DefaultLambdaJsonSerializer()); + + var result = await ctx.MapAsync( + new[] { "a", "b" }, + async (_, item, _, _, _) => { await Task.CompletedTask; return item.ToUpperInvariant(); }, + name: "map", + config: new MapConfig { NestingType = NestingType.Flat, ItemSerializer = ser }); + + Assert.Equal(2, result.SuccessCount); + var ids = ser.SerializeContexts.Select(c => c.EntityId).Distinct().ToList(); + Assert.Contains($"{IdAt(1)}#0", ids); + Assert.Contains($"{IdAt(1)}#1", ids); + Assert.All(ser.SerializeContexts, c => Assert.Equal(TestArn, c.DurableExecutionArn)); + } + + [Fact] + public async Task PlainSerializer_StillRoundTrips_ViaFallback() + { + var plain = new PlainSpy(); + var ctx = CreateContext(new DefaultLambdaJsonSerializer()); + + var result = await ctx.StepAsync( + async (_, _) => { await Task.CompletedTask; return "hello"; }, + name: "s", + config: new StepConfig { Serializer = plain }); + + Assert.Equal("hello", result); + Assert.True(plain.SerializeCount >= 1); + } + + /// + /// A serializer whose deserialize throws must fail the step (the round-trip runs + /// BEFORE the SUCCEED checkpoint), not leave it checkpointed SUCCEEDED-but-thrown. + /// Regression guard for the fresh-success round-trip being inside the fault path. + /// + [Fact] + public async Task Step_FreshSuccessRoundTripThatThrows_SurfacesAsStepFailure() + { + var ser = new ThrowingDeserializeSerializer(); + var ctx = CreateContext(new DefaultLambdaJsonSerializer()); + var runs = 0; + + await Assert.ThrowsAsync(() => ctx.StepAsync( + async (_, _) => { runs++; await Task.CompletedTask; return 42; }, + name: "s", + config: new StepConfig { Serializer = ser })); + + // Body ran exactly once (no retry loop), and it was serialized before the + // failing deserialize — i.e. the round-trip failed cleanly, not after a + // second terminal checkpoint. + Assert.Equal(1, runs); + Assert.Equal(1, ser.SerializeCount); + } + + /// + /// On replay, a Map reconstructed from inline per-unit payloads must deserialize + /// each unit with the SAME per-unit EntityId ({OperationId}#{index}) that + /// the serialize side used — otherwise a context-aware serializer keying storage + /// by EntityId cannot find the value. Regression guard for the serialize/deserialize + /// context asymmetry in ConcurrentOperation. + /// + [Fact] + public async Task Map_Replay_DeserializesEachUnitWithPerItemEntityId() + { + var parentOpId = IdAt(1); + var summaryJson = + "{\"CompletionReason\":\"ALL_COMPLETED\",\"Units\":[" + + "{\"Index\":0,\"Name\":\"0\",\"Status\":\"SUCCEEDED\",\"Result\":\"\\\"A\\\"\"}," + + "{\"Index\":1,\"Name\":\"1\",\"Status\":\"SUCCEEDED\",\"Result\":\"\\\"B\\\"\"}]}"; + + var ser = new CapturingSerializer(); + var ctx = CreateContext(new DefaultLambdaJsonSerializer(), new InitialExecutionState + { + Operations = new List + { + new() + { + Id = parentOpId, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.Map, + Name = "map", + ContextDetails = new ContextDetails { Result = summaryJson } + } + } + }); + + var executed = false; + var result = await ctx.MapAsync( + new[] { "a", "b" }, + async (_, item, _, _, _) => { executed = true; await Task.Yield(); return item.ToUpperInvariant(); }, + name: "map", + config: new MapConfig { NestingType = NestingType.Flat, ItemSerializer = ser }); + + Assert.False(executed); // replay: bodies not re-run + Assert.Equal(new[] { "A", "B" }, result.GetResults()); + + var ids = ser.DeserializeContexts.Select(c => c.EntityId).ToList(); + Assert.Contains($"{parentOpId}#0", ids); + Assert.Contains($"{parentOpId}#1", ids); + Assert.All(ser.DeserializeContexts, c => Assert.Equal(TestArn, c.DurableExecutionArn)); + } + + /// Serializes via JSON but always throws on the context-aware deserialize. + private sealed class ThrowingDeserializeSerializer : ILambdaSerializer, IDurableResultSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + public int SerializeCount { get; private set; } + + public void Serialize(T value, Stream stream, DurableSerializationContext context) + { + SerializeCount++; + _inner.Serialize(value, stream); + } + + public T Deserialize(Stream stream, DurableSerializationContext context) => + throw new InvalidOperationException("boom on deserialize"); + + void ILambdaSerializer.Serialize(T response, Stream responseStream) => _inner.Serialize(response, responseStream); + T ILambdaSerializer.Deserialize(Stream requestStream) => _inner.Deserialize(requestStream); + } +} diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/MapOperationTests.cs b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/MapOperationTests.cs index 645ccba3b..bf702ec62 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/MapOperationTests.cs +++ b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/MapOperationTests.cs @@ -1,3 +1,4 @@ +using Amazon.Lambda.Core; using Amazon.Lambda.DurableExecution; using Amazon.Lambda.DurableExecution.Internal; using Amazon.Lambda.Serialization.SystemTextJson; @@ -486,8 +487,233 @@ public async Task MapAsync_NestingTypeFlat_ReplaySucceeded_RebuildsFromInlinePay Assert.Empty(recorder.Flushed); } - // ────────────────────────────────────────────────────────────────────── - // Argument validation + /// + /// A Flat unit's body result is returned RAW from the child context (a virtual child + /// suppresses its own SUCCEED and the round-trip a Nested child performs), whereas on + /// replay each value is reconstructed by DESERIALIZING the inline payload. With a + /// non-round-tripping (asymmetric) ItemSerializer those two paths diverged before the + /// fix. This pins fresh == replay: the fresh run must observe the round-tripped value, + /// exactly what replay rebuilds from the same inline payload. + /// + [Fact] + public async Task MapAsync_Flat_NonRoundTrippingItemSerializer_FreshMatchesReplay() + { + var ser = new AppendOnDeserializeSerializer(); + + // ---- Fresh run ---- + var (freshCtx, freshRec, _, _) = CreateContext(); + var fresh = await freshCtx.MapAsync( + new[] { "a", "b" }, + async (ctx, item, index, all, _) => { await Task.Yield(); return item; }, + name: "m", + config: new MapConfig { NestingType = NestingType.Flat, ItemSerializer = ser }); + + await freshRec.Batcher.DrainAsync(); + + // Fresh observed values already reflect the deserialize transform (round-tripped), + // which is what replay reconstructs from the inline payload. + Assert.Equal(new[] { "a-rt", "b-rt" }, fresh.GetResults()); + + // Capture the exact Map SUCCEED payload the fresh run checkpointed. + var parentPayload = freshRec.Flushed + .Single(o => o.Type == "CONTEXT" && o.SubType == "Map" && $"{o.Action}" == "SUCCEED") + .Payload; + + // ---- Replay from that checkpoint ---- + var parentOpId = IdAt(1); + var (replayCtx, replayRec, _, _) = CreateContext(new InitialExecutionState + { + Operations = new List + { + new() + { + Id = parentOpId, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.Map, + Name = "m", + ContextDetails = new ContextDetails { Result = parentPayload } + } + } + }); + + var executed = false; + var replay = await replayCtx.MapAsync( + new[] { "a", "b" }, + async (ctx, item, index, all, _) => { executed = true; await Task.Yield(); return item; }, + name: "m", + config: new MapConfig { NestingType = NestingType.Flat, ItemSerializer = ser }); + + Assert.False(executed); // replay does not re-run bodies + Assert.Equal(fresh.GetResults(), replay.GetResults()); + Assert.Equal(new[] { "a-rt", "b-rt" }, replay.GetResults()); + } + + /// + /// Nested (the DEFAULT NestingType) counterpart to the Flat fresh==replay test. A Nested + /// unit is a NON-virtual child: it round-trips its result inside ChildContextOperation, + /// and the parent now inlines the child's ORIGINAL serialized checkpoint payload + /// (S(body)) rather than re-serializing the round-tripped return value. With a + /// non-round-tripping serializer the pre-fix parent re-serialized D(S(body)) and + /// replay deserialized it AGAIN, double-transforming (fresh "a-rt" vs replay "a-rt-rt"). + /// This pins fresh == replay for Nested on the default path. + /// + [Fact] + public async Task MapAsync_Nested_NonRoundTrippingItemSerializer_FreshMatchesReplay() + { + var ser = new AppendOnDeserializeSerializer(); + + // ---- Fresh run (Nested = default NestingType) ---- + var (freshCtx, freshRec, _, _) = CreateContext(); + var fresh = await freshCtx.MapAsync( + new[] { "a", "b" }, + async (ctx, item, index, all, _) => { await Task.Yield(); return item; }, + name: "m", + config: new MapConfig { ItemSerializer = ser }); + + await freshRec.Batcher.DrainAsync(); + + // The Nested child round-trips internally, so the fresh observed value already + // reflects the deserialize transform — exactly what replay rebuilds from the inline + // payload (which is now the child's own S(body), deserialized once). + Assert.Equal(new[] { "a-rt", "b-rt" }, fresh.GetResults()); + + var parentPayload = freshRec.Flushed + .Single(o => o.Type == "CONTEXT" && o.SubType == "Map" && $"{o.Action}" == "SUCCEED") + .Payload; + + // ---- Replay from that checkpoint ---- + var parentOpId = IdAt(1); + var (replayCtx, _, _, _) = CreateContext(new InitialExecutionState + { + Operations = new List + { + new() + { + Id = parentOpId, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.Map, + Name = "m", + ContextDetails = new ContextDetails { Result = parentPayload } + } + } + }); + + var executed = false; + var replay = await replayCtx.MapAsync( + new[] { "a", "b" }, + async (ctx, item, index, all, _) => { executed = true; await Task.Yield(); return item; }, + name: "m", + config: new MapConfig { ItemSerializer = ser }); + + Assert.False(executed); // replay does not re-run bodies + Assert.Equal(fresh.GetResults(), replay.GetResults()); + Assert.Equal(new[] { "a-rt", "b-rt" }, replay.GetResults()); + } + + /// + /// Comment 2(a): a Flat unit whose fresh-success round-trip DESERIALIZE fails must be + /// handled terminally — the unit is recorded Failed inline and the parent STILL emits a + /// terminal SUCCEED — instead of letting the exception propagate raw with no terminal + /// checkpoint (which would re-run every virtual unit body on replay: a deterministic + /// poison loop). On replay the body is NOT re-run and reconstruct reads the terminal + /// Error inline without re-deserializing the poison payload. + /// + [Fact] + public async Task MapAsync_Flat_RoundTripDeserializeFails_RecordsUnitFailedAndStillCheckpointsSucceed() + { + var ser = new ThrowOnDeserializeSerializer(); + + var (freshCtx, freshRec, _, _) = CreateContext(); + var fresh = await freshCtx.MapAsync( + new[] { "a" }, + async (ctx, item, index, all, _) => { await Task.Yield(); return item; }, + name: "m", + config: new MapConfig { NestingType = NestingType.Flat, ItemSerializer = ser }); + + await freshRec.Batcher.DrainAsync(); + + // Body succeeded but its result could not be deserialized back -> unit Failed, and + // the operation never throws. + Assert.True(fresh.HasFailure); + Assert.Equal(BatchItemStatus.Failed, fresh.All[0].Status); + Assert.IsType(fresh.All[0].Error); + + // A terminal parent SUCCEED checkpoint WAS written despite the failure. + var parentSucceed = freshRec.Flushed + .Single(o => o.Type == "CONTEXT" && o.SubType == "Map" && $"{o.Action}" == "SUCCEED"); + Assert.NotNull(parentSucceed.Payload); + + // ---- Replay from that checkpoint: poison body never re-runs, unit stays Failed. ---- + var parentOpId = IdAt(1); + var (replayCtx, _, _, _) = CreateContext(new InitialExecutionState + { + Operations = new List + { + new() + { + Id = parentOpId, + Type = OperationTypes.Context, + Status = OperationStatuses.Succeeded, + SubType = OperationSubTypes.Map, + Name = "m", + ContextDetails = new ContextDetails { Result = parentSucceed.Payload } + } + } + }); + + var executed = false; + var replay = await replayCtx.MapAsync( + new[] { "a" }, + async (ctx, item, index, all, _) => { executed = true; await Task.Yield(); return item; }, + name: "m", + config: new MapConfig { NestingType = NestingType.Flat, ItemSerializer = ser }); + + Assert.False(executed); + Assert.True(replay.HasFailure); + Assert.Equal(BatchItemStatus.Failed, replay.All[0].Status); + } + + /// + /// A serializer that serializes normally (plain JSON) but ALWAYS throws on deserialize — + /// simulates an asymmetric/broken ItemSerializer that cannot read back its own output. + /// + private sealed class ThrowOnDeserializeSerializer : ILambdaSerializer, IDurableResultSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + + public void Serialize(T value, Stream stream, DurableSerializationContext context) => _inner.Serialize(value, stream); + public T Deserialize(Stream stream, DurableSerializationContext context) => + throw new InvalidOperationException("ThrowOnDeserializeSerializer: cannot deserialize"); + + void ILambdaSerializer.Serialize(T response, Stream responseStream) => _inner.Serialize(response, responseStream); + T ILambdaSerializer.Deserialize(Stream requestStream) => + throw new InvalidOperationException("ThrowOnDeserializeSerializer: cannot deserialize"); + } + + /// + /// Deliberately NON-round-tripping serializer: Serialize writes the value as + /// JSON, but Deserialize appends a marker to a string result. Lets a test + /// observe whether a value went through a deserialize (round-trip) or is the raw body + /// result. + /// + private sealed class AppendOnDeserializeSerializer : ILambdaSerializer, IDurableResultSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + + public void Serialize(T value, Stream stream, DurableSerializationContext context) => _inner.Serialize(value, stream); + + public T Deserialize(Stream stream, DurableSerializationContext context) + { + var value = _inner.Deserialize(stream); + if (value is string s) return (T)(object)(s + "-rt"); + return value; + } + + void ILambdaSerializer.Serialize(T response, Stream responseStream) => _inner.Serialize(response, responseStream); + T ILambdaSerializer.Deserialize(Stream requestStream) => _inner.Deserialize(requestStream); + } // ────────────────────────────────────────────────────────────────────── [Fact] From e5924ac9b3f37012c16e0a5edb9fb83fefad5c58 Mon Sep 17 00:00:00 2001 From: Garrett Beatty Date: Thu, 3 Sep 2026 21:04:09 -0400 Subject: [PATCH 3/5] =?UTF-8?q?feat(DurableExecution):=20FileSystemSeriali?= =?UTF-8?q?zer=20=E2=80=94=20offload=20large=20results=20to=20a=20filesyst?= =?UTF-8?q?em=20(#2540)=20(#2558)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(DurableExecution): FileSystemSerializer for offloading large results Adds FileSystemSerializer (implements ILambdaSerializer + IDurableResultSerializer): stores a durable operation's serialized result on a filesystem (EFS / S3 Files, NOT Lambda /tmp) and keeps only a small {file}|{data} envelope in the checkpoint. Wraps an inner ILambdaSerializer so callers control the on-the-wire format; supports Always/Overflow storage modes and Uri/Hash path encoding; envelope is source-generated (AOT-safe). The plain ILambdaSerializer path throws (offload needs the durable context). Stacked on the serializer-engine PR; restores the two doc links now that the type exists. Includes the FileSystemSerializer AutoVer (Minor) change file and unit tests. * address review: inner-less FileSystemSerializer ctor uses global serializer normj asked for a constructor that omits the inner ILambdaSerializer and falls back to the durable execution's globally-registered serializer. Adds FileSystemSerializer(basePath, storageMode, pathEncoding) which leaves _inner null. The durable runtime binds the global serializer as the inner when it resolves the effective per-operation serializer, via a new internal IDefaultInnerSerializer capability (DurableContext.WithDefaultInner at the Step/ChildContext/WaitForCondition/Parallel/Map resolution sites). An explicitly-supplied inner always wins; used without a bound inner it throws a clear InvalidOperationException. --- .../3d3d307e-cf33-421d-b9c4-c88670b5c5ef.json | 11 + .../DurableContext.cs | 15 +- .../FileSystemSerializer.cs | 483 ++++++++++++++++++ .../IDurableResultSerializer.cs | 2 +- .../Internal/ChildContextOperation.cs | 2 +- .../Internal/IDefaultInnerSerializer.cs | 22 + .../Internal/LambdaSerializerHelper.cs | 10 + .../FileSystemSerializerTests.cs | 431 ++++++++++++++++ 8 files changed, 969 insertions(+), 7 deletions(-) create mode 100644 .autover/changes/3d3d307e-cf33-421d-b9c4-c88670b5c5ef.json create mode 100644 Libraries/src/Amazon.Lambda.DurableExecution/FileSystemSerializer.cs create mode 100644 Libraries/src/Amazon.Lambda.DurableExecution/Internal/IDefaultInnerSerializer.cs create mode 100644 Libraries/test/Amazon.Lambda.DurableExecution.Tests/FileSystemSerializerTests.cs diff --git a/.autover/changes/3d3d307e-cf33-421d-b9c4-c88670b5c5ef.json b/.autover/changes/3d3d307e-cf33-421d-b9c4-c88670b5c5ef.json new file mode 100644 index 000000000..414ae4af9 --- /dev/null +++ b/.autover/changes/3d3d307e-cf33-421d-b9c4-c88670b5c5ef.json @@ -0,0 +1,11 @@ +{ + "Projects": [ + { + "Name": "Amazon.Lambda.DurableExecution", + "Type": "Minor", + "ChangelogMessages": [ + "Add FileSystemSerializer for offloading large durable operation results to a filesystem (e.g. EFS/S3 Files), keeping only a small pointer in the checkpoint. It wraps an inner ILambdaSerializer, so callers control the on-the-wire format (JSON, compressed, etc.). Introduces the optional IDurableResultSerializer interface and DurableSerializationContext (EntityId \u002B DurableExecutionArn); when an operation\u0027s configured serializer implements it, the durable runtime supplies operation/execution identity so external storage can be keyed safely. Applied to Step, ChildContext, WaitForCondition, and Map/Parallel per-item results. Plain ILambdaSerializer serializers are unaffected." + ] + } + ] +} \ No newline at end of file diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/DurableContext.cs b/Libraries/src/Amazon.Lambda.DurableExecution/DurableContext.cs index 1144c6bf1..d74f09a7e 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/DurableContext.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/DurableContext.cs @@ -84,7 +84,8 @@ private Task RunStep( StepConfig? config, CancellationToken cancellationToken) { - var serializer = config?.Serializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); + var defaultSerializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = LambdaSerializerHelper.WithDefaultInner(config?.Serializer ?? defaultSerializer, defaultSerializer); var operationId = _idGenerator.NextId(); var op = new StepOperation( @@ -147,7 +148,8 @@ public Task WaitForConditionAsync( ArgumentNullException.ThrowIfNull(config); ArgumentNullException.ThrowIfNull(config.WaitStrategy); - var serializer = config.Serializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); + var defaultSerializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = LambdaSerializerHelper.WithDefaultInner(config.Serializer ?? defaultSerializer, defaultSerializer); var operationId = _idGenerator.NextId(); var op = new WaitForConditionOperation( operationId, name, _idGenerator.ParentId, check, config, serializer, Logger, @@ -161,7 +163,8 @@ private Task RunChildContext( ChildContextConfig? config, CancellationToken cancellationToken) { - var serializer = config?.Serializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); + var defaultSerializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = LambdaSerializerHelper.WithDefaultInner(config?.Serializer ?? defaultSerializer, defaultSerializer); var operationId = _idGenerator.NextId(); @@ -246,7 +249,8 @@ private Task> RunParallel( // globally-registered serializer. This is the only serializer ConcurrentOperation // uses (per-unit child results + inline summary results); the aggregate batch // envelope is a source-generated structure and is unaffected. - var serializer = effectiveConfig.ItemSerializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); + var defaultSerializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = LambdaSerializerHelper.WithDefaultInner(effectiveConfig.ItemSerializer ?? defaultSerializer, defaultSerializer); var operationId = _idGenerator.NextId(); var op = new Internal.ParallelOperation( @@ -279,7 +283,8 @@ private Task> RunMap( // globally-registered serializer. This is the only serializer ConcurrentOperation // uses (per-unit child results + inline summary results); the aggregate batch // envelope is a source-generated structure and is unaffected. - var serializer = effectiveConfig.ItemSerializer ?? LambdaSerializerHelper.GetRequired(LambdaContext); + var defaultSerializer = LambdaSerializerHelper.GetRequired(LambdaContext); + var serializer = LambdaSerializerHelper.WithDefaultInner(effectiveConfig.ItemSerializer ?? defaultSerializer, defaultSerializer); var operationId = _idGenerator.NextId(); var op = new Internal.MapOperation( diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/FileSystemSerializer.cs b/Libraries/src/Amazon.Lambda.DurableExecution/FileSystemSerializer.cs new file mode 100644 index 000000000..eb007e698 --- /dev/null +++ b/Libraries/src/Amazon.Lambda.DurableExecution/FileSystemSerializer.cs @@ -0,0 +1,483 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using System.IO; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; +using System.Text.RegularExpressions; +using System.Security.Cryptography; +using Amazon.Lambda.Core; + +namespace Amazon.Lambda.DurableExecution; + +/// +/// Controls when writes a value to the filesystem. +/// +public enum FileSystemStorageMode +{ + /// + /// Every value is written to a file; the checkpoint stores only a file pointer. + /// Best for consistently large payloads or predictable checkpoint sizes. + /// + Always, + + /// + /// The value is stored inline in the checkpoint unless it would exceed the durable + /// execution checkpoint size limit (~256 KB), in which case it overflows to a + /// file. Best for mixed workloads where most payloads are small. + /// + Overflow, +} + +/// +/// Controls how the durable execution ARN (directory) and entity id (file name) are +/// turned into filesystem path segments. +/// +public enum FileSystemPathEncoding +{ + /// + /// Human-navigable paths. The per-execution directory is built from the ARN's + /// function name, execution name and invocation id; the file name is the entity id, + /// URL-encoded. If the ARN does not match the expected durable-execution shape, the + /// whole ARN is URL-encoded into a single directory segment. + /// + Uri, + + /// + /// The ARN (directory) and entity id (file name) are each replaced by their SHA-256 + /// hex digest — fixed length and always filesystem-safe, at the cost of readability. + /// + Hash, +} + +/// +/// An / that stores +/// serialized durable operation results on a filesystem, keeping only a small pointer in +/// the checkpoint. It wraps an inner serializer that performs the actual +/// value↔bytes conversion (JSON, compressed JSON, a custom format, …), so the choice +/// of on-the-wire format is fully in the caller's control. Construct it without an inner +/// serializer () +/// to reuse the durable execution's globally-registered serializer as the inner. +/// +/// +/// +/// ⚠ Do NOT use with Lambda's ephemeral /tmp for values that must survive +/// replay. /tmp is local to a single execution environment; on replay a +/// different environment may be used and the file will not be found. Use a durable, +/// shared mount such as Amazon EFS or Amazon S3 Files, which persist across invocations +/// and are visible to concurrent function instances. +/// +/// +/// The checkpoint stores a JSON envelope that is either +/// {"data":"<base64 inner bytes>"} (inline) or {"file":"<path>"} +/// (pointer). The envelope is serialized with a source generator, so this type is +/// Native-AOT-safe as long as the inner serializer is. +/// +/// +/// Lifecycle: this serializer never deletes result files. A write for a given +/// (execution, entity) overwrites its file in place, so retries and re-serializations +/// of the same operation do not accumulate; but the files of completed or abandoned +/// executions remain on the mount. Pair the base path with an external retention policy +/// (an EFS lifecycle policy, an S3 lifecycle rule, or a scheduled cleanup keyed by the +/// per-execution directory) so storage does not grow unbounded. +/// +/// +/// This serializer must be used through a durable operation's per-operation serializer +/// slot (for example StepConfig.Serializer) so it receives a +/// . Using it as a plain +/// (for example as the assembly-registered serializer) +/// throws, because there is then no execution/entity identity to build a safe path. +/// +/// +public sealed class FileSystemSerializer : ILambdaSerializer, IDurableResultSerializer, IDefaultInnerSerializer +{ + // The durable execution checkpoint size limit is ~256 KB; leave 1 KB of headroom for + // the envelope wrapper and other checkpoint metadata. + private const int OverflowThresholdBytes = (256 * 1024) - 1024; + + private static readonly Regex DurableExecutionArnPattern = new( + @"^arn:[^:]*:lambda:[^:]*:[^:]*:function:([^:/]+):[^:/]+/durable-execution/([^/]+)/([^/]+)$", + RegexOptions.Compiled | RegexOptions.CultureInvariant, + TimeSpan.FromSeconds(1)); + + private readonly ILambdaSerializer? _inner; + private readonly string _basePath; + private readonly FileSystemStorageMode _storageMode; + private readonly FileSystemPathEncoding _pathEncoding; + + /// Creates a new . + /// + /// The serializer that converts values to/from bytes (for example the global + /// ILambdaContext.Serializer, or a compressing/encrypting wrapper). + /// + /// + /// Directory under which result files are written (for example /mnt/efs/durable). + /// Use a durable, shared mount — see the type remarks. + /// + /// When to write to a file. Defaults to . + /// How to encode path segments. Defaults to . + public FileSystemSerializer( + ILambdaSerializer inner, + string basePath, + FileSystemStorageMode storageMode = FileSystemStorageMode.Always, + FileSystemPathEncoding pathEncoding = FileSystemPathEncoding.Uri) + { + _inner = inner ?? throw new ArgumentNullException(nameof(inner)); + // Normalize once so stored file pointers are absolute and their resolution + // does not depend on the current working directory of a later invocation + // (which may run in a different execution environment). ValidatePathWithinBase + // still re-resolves for symlink safety, but the stored base is now stable. + _basePath = Path.GetFullPath(basePath ?? throw new ArgumentNullException(nameof(basePath))); + _storageMode = storageMode; + _pathEncoding = pathEncoding; + } + + /// + /// Creates a new that uses the durable execution's + /// globally-registered (the assembly-level + /// [assembly: LambdaSerializer(...)] serializer, or the one passed to + /// LambdaBootstrapBuilder.Create(handler, serializer)) as its inner serializer. + /// + /// + /// The inner serializer is supplied by the durable runtime when this instance is used + /// through a per-operation serializer slot (for example StepConfig.Serializer). + /// It saves you from having to thread ctx.Serializer in yourself when the + /// on-the-wire format is just the function's normal serializer. + /// + /// + /// Directory under which result files are written (for example /mnt/efs/durable). + /// Use a durable, shared mount — see the type remarks. + /// + /// When to write to a file. Defaults to . + /// How to encode path segments. Defaults to . + public FileSystemSerializer( + string basePath, + FileSystemStorageMode storageMode = FileSystemStorageMode.Always, + FileSystemPathEncoding pathEncoding = FileSystemPathEncoding.Uri) + { + _inner = null; + // Normalize once (see the inner-taking constructor) so a stored file pointer is + // absolute and stable across invocations, independent of the current directory. + _basePath = Path.GetFullPath(basePath ?? throw new ArgumentNullException(nameof(basePath))); + _storageMode = storageMode; + _pathEncoding = pathEncoding; + } + + // When constructed without an explicit inner serializer, the durable runtime binds the + // globally-registered serializer here before the operation runs (see the parameterless-inner + // constructor). Returns a bound copy; a caller-supplied inner always wins and is left as-is. + ILambdaSerializer IDefaultInnerSerializer.WithDefaultInner(ILambdaSerializer inner) + { + if (inner is null) throw new ArgumentNullException(nameof(inner)); + return _inner is not null + ? this + : new FileSystemSerializer(inner, _basePath, _storageMode, _pathEncoding); + } + + // ---- context-aware path (used by durable execution) ---- + + /// + public void Serialize(T value, Stream stream, DurableSerializationContext context) + { + string path; + if (_storageMode == FileSystemStorageMode.Overflow) + { + // Overflow needs the serialized size to decide inline-vs-file, so it buffers. + byte[] bytes = InnerSerialize(value); + var inline = new FileSystemEnvelope { Data = Convert.ToBase64String(bytes) }; + var inlineJson = JsonSerializer.Serialize(inline, FileSystemJsonContext.Default.FileSystemEnvelope); + if (Encoding.UTF8.GetByteCount(inlineJson) <= OverflowThresholdBytes) + { + var inlineBytes = Encoding.UTF8.GetBytes(inlineJson); + stream.Write(inlineBytes, 0, inlineBytes.Length); + return; + } + path = WriteBytesToFile(bytes, context); + } + else + { + // Always: stream the inner serialization straight to the file — no large in-memory + // intermediate, which is the whole point of offloading big payloads. + path = WriteStreamingToFile(value, context); + } + + var envelope = new FileSystemEnvelope { File = path }; + JsonSerializer.Serialize(stream, envelope, FileSystemJsonContext.Default.FileSystemEnvelope); + } + + /// + public T Deserialize(Stream stream, DurableSerializationContext context) + { + var envelope = JsonSerializer.Deserialize(stream, FileSystemJsonContext.Default.FileSystemEnvelope) + ?? throw new InvalidOperationException("FileSystemSerializer: empty or invalid envelope."); + + if (envelope.File is not null) + { + // Guard against a tampered/corrupted checkpoint pointing outside the base path. + ValidatePathWithinBase(envelope.File); + if (!File.Exists(envelope.File)) + throw new FileNotFoundException( + $"FileSystemSerializer: offloaded payload file not found: '{envelope.File}'. " + + "If the base path is Lambda's /tmp, the value cannot survive replay on a different " + + "execution environment — use a durable shared mount such as EFS or S3 Files.", + envelope.File); + + // Stream the offloaded payload straight to the inner serializer instead of + // buffering the whole file into a byte[] — symmetric with the streaming + // write path (WriteStreamingToFile), so a large offloaded value is never + // fully materialized in memory just to be read back. + // + // FileShare.Delete (in addition to Read): the write path replaces a file in + // place via File.Move(overwrite: true). On POSIX/Linux that rename-over is + // atomic and an open read handle never blocks it (the reader keeps the old + // inode). On Windows/macOS, a held read handle opened WITHOUT FileShare.Delete + // blocks the rename-over, so a concurrent replay-read racing a re-serialize of + // the same entity would throw IOException. Granting Delete share keeps those + // non-Linux readers from blocking the atomic replace; the atomic-replace + // concurrency-safety itself is a POSIX/Linux property. (Prod runs Linux, but + // dev/test frequently run Windows/macOS.) + var inner = RequireInner(); + using var fileStream = new FileStream( + envelope.File, FileMode.Open, FileAccess.Read, FileShare.Read | FileShare.Delete); + return inner.Deserialize(fileStream); + } + + if (envelope.Data is not null) + { + byte[] bytes; + try + { + bytes = Convert.FromBase64String(envelope.Data); + } + catch (FormatException ex) + { + // Sibling checks in this method throw descriptive InvalidOperationExceptions; + // a corrupted inline envelope should be no different (a bare FormatException + // gives no context about which envelope field was malformed). + throw new InvalidOperationException( + "FileSystemSerializer: inline envelope 'data' is not valid base64 — the checkpoint " + + "is corrupted or was not written by this serializer.", ex); + } + return InnerDeserialize(bytes); + } + + throw new InvalidOperationException("FileSystemSerializer: envelope has neither 'file' nor 'data'."); + } + + // ---- plain ILambdaSerializer path (only reachable outside durable execution) ---- + + void ILambdaSerializer.Serialize(T response, Stream responseStream) => + throw new NotSupportedException( + "FileSystemSerializer must be used via a durable operation's per-operation serializer " + + "(for example StepConfig.Serializer), which supplies the DurableSerializationContext needed " + + "to build a safe, unique file path. It cannot be used as a plain ILambdaSerializer."); + + T ILambdaSerializer.Deserialize(Stream requestStream) => + throw new NotSupportedException( + "FileSystemSerializer must be used via a durable operation's per-operation serializer " + + "(for example StepConfig.Serializer). It cannot be used as a plain ILambdaSerializer."); + + // ---- helpers ---- + + // The inner serializer is either the one passed to the constructor or, for the + // inner-less constructor, the globally-registered serializer bound by the durable + // runtime via IDefaultInnerSerializer. If neither is present it means this instance + // was used outside a durable operation slot, where no global serializer is available. + private ILambdaSerializer RequireInner() => + _inner ?? throw new InvalidOperationException( + "FileSystemSerializer was constructed without an inner serializer and the durable " + + "runtime did not supply a globally-registered ILambdaSerializer to use as the inner. " + + "Either pass an inner serializer to the constructor, or register one via " + + "[assembly: LambdaSerializer(typeof(...))] / LambdaBootstrapBuilder.Create(handler, serializer)."); + + private byte[] InnerSerialize(T value) + { + var inner = RequireInner(); + using var ms = new MemoryStream(); + inner.Serialize(value, ms); + return ms.ToArray(); + } + + private T InnerDeserialize(byte[] bytes) + { + var inner = RequireInner(); + using var ms = new MemoryStream(bytes); + return inner.Deserialize(ms); + } + + private string WriteStreamingToFile(T value, DurableSerializationContext context) + { + var inner = RequireInner(); + var (path, tmp) = ResolveTargetPaths(context); + try + { + using (var fileStream = new FileStream(tmp, FileMode.Create, FileAccess.Write, FileShare.None)) + { + inner.Serialize(value, fileStream); + } + File.Move(tmp, path, overwrite: true); + } + catch + { + TryDelete(tmp); + throw; + } + return path; + } + + private string WriteBytesToFile(byte[] bytes, DurableSerializationContext context) + { + var (path, tmp) = ResolveTargetPaths(context); + try + { + File.WriteAllBytes(tmp, bytes); + File.Move(tmp, path, overwrite: true); + } + catch + { + TryDelete(tmp); + throw; + } + return path; + } + + // Resolves the final file path and a unique temp path in the SAME directory. Writing to the + // temp file then atomically moving it means a concurrent replay/reader never observes a + // partially-written or truncated file. The temp name is unique so two writers of the same + // entity don't clobber each other's in-progress temp. + private (string path, string tmp) ResolveTargetPaths(DurableSerializationContext context) + { + var dir = ResolveExecutionDir(context.DurableExecutionArn); + // Guard the WRITE path the same way the read path is guarded: under Uri + // encoding the ARN-derived segments are inserted into the path unescaped + // (Uri.EscapeDataString does not escape '.', so it would NOT neutralize a + // '..' traversal segment either), so a tampered/malformed durable execution + // ARN containing '..' could otherwise resolve the per-execution directory + // outside the configured base path. + ValidateWriteDirWithinBase(dir); + Directory.CreateDirectory(dir); + var path = Path.Combine(dir, EncodeSegment(context.EntityId) + ".bin"); + var tmp = path + ".tmp-" + Guid.NewGuid().ToString("N"); + return (path, tmp); + } + + private static void TryDelete(string path) + { + try { if (File.Exists(path)) File.Delete(path); } catch { /* best effort */ } + } + + // Rejects a checkpoint file pointer that resolves outside the configured base path, so a + // tampered or corrupted envelope cannot turn deserialization into an arbitrary file read. + private void ValidatePathWithinBase(string filePath) + { + if (!IsWithinBase(filePath)) + throw new InvalidOperationException( + $"FileSystemSerializer: refusing to read an offloaded payload outside the configured " + + $"base path. Path '{filePath}' does not resolve under '{_basePath}'."); + } + + // Rejects a resolved per-execution write directory that lands outside the configured base + // path, so a tampered/malformed durable execution ARN (e.g. containing '..' traversal + // segments) cannot write offloaded payloads to an arbitrary filesystem location. + private void ValidateWriteDirWithinBase(string dir) + { + if (!IsWithinBase(dir)) + throw new InvalidOperationException( + $"FileSystemSerializer: refusing to write an offloaded payload outside the configured " + + $"base path. Resolved directory '{dir}' does not resolve under '{_basePath}'. This can " + + "happen when the durable execution ARN contains path-traversal segments."); + } + + // True when the fully-resolved candidate path lies under the fully-resolved base path. + private bool IsWithinBase(string candidate) + { + // Compare using the filesystem's casing rules. Ordinal alone would falsely + // reject an in-base path that differs only by case on Windows/macOS. + var comparison = OperatingSystem.IsWindows() || OperatingSystem.IsMacOS() + ? StringComparison.OrdinalIgnoreCase + : StringComparison.Ordinal; + + var fullBase = ResolveReal(Path.GetFullPath(_basePath)); + if (!fullBase.EndsWith(Path.DirectorySeparatorChar)) + fullBase += Path.DirectorySeparatorChar; + + var fullPath = ResolveReal(Path.GetFullPath(candidate)); + return fullPath.StartsWith(fullBase, comparison); + } + + // Best-effort resolution of a path to its real on-disk location, following a + // symlink on the leaf file/directory and on its immediate parent directory. + // Path.GetFullPath only collapses '..'/separators and does NOT follow symlinks, + // so without this a symlink planted under the base but pointing outside it would + // pass a purely lexical prefix check and defeat the containment guard. Both the + // base and the candidate are resolved the same way so a symlinked mount root + // (for example macOS /var -> /private/var) does not cause false rejections. + private static string ResolveReal(string fullPath) + { + try + { + var dir = Path.GetDirectoryName(fullPath); + if (dir != null && Directory.Exists(dir)) + { + var realDir = Directory.ResolveLinkTarget(dir, returnFinalTarget: true)?.FullName ?? dir; + fullPath = Path.Combine(realDir, Path.GetFileName(fullPath)); + } + if (File.Exists(fullPath)) + fullPath = File.ResolveLinkTarget(fullPath, returnFinalTarget: true)?.FullName ?? fullPath; + else if (Directory.Exists(fullPath)) + fullPath = Directory.ResolveLinkTarget(fullPath, returnFinalTarget: true)?.FullName ?? fullPath; + } + catch + { + // Best effort — fall back to the lexical full path. + } + return fullPath; + } + + private string ResolveExecutionDir(string arn) + { + if (_pathEncoding == FileSystemPathEncoding.Uri) + { + var match = DurableExecutionArnPattern.Match(arn); + if (match.Success) + { + return Path.Combine( + _basePath, + match.Groups[1].Value, // function name + match.Groups[2].Value, // execution name + match.Groups[3].Value); // invocation id + } + } + return Path.Combine(_basePath, EncodeSegment(arn)); + } + + private string EncodeSegment(string value) => + _pathEncoding == FileSystemPathEncoding.Hash + ? Sha256Hex(value) + : Uri.EscapeDataString(value); + + private static string Sha256Hex(string value) + { + var hash = SHA256.HashData(Encoding.UTF8.GetBytes(value)); + return Convert.ToHexString(hash).ToLowerInvariant(); + } +} + +/// The checkpoint envelope written by . +internal sealed class FileSystemEnvelope +{ + /// Inline payload (base64 of the inner-serialized bytes). Set when stored inline. + [JsonPropertyName("data")] + public string? Data { get; set; } + + /// Path to the file holding the inner-serialized bytes. Set when offloaded. + [JsonPropertyName("file")] + public string? File { get; set; } +} + +[JsonSourceGenerationOptions(DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)] +[JsonSerializable(typeof(FileSystemEnvelope))] +internal partial class FileSystemJsonContext : JsonSerializerContext +{ +} diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/IDurableResultSerializer.cs b/Libraries/src/Amazon.Lambda.DurableExecution/IDurableResultSerializer.cs index 86e90b59c..7fc0d5223 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/IDurableResultSerializer.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/IDurableResultSerializer.cs @@ -18,7 +18,7 @@ namespace Amazon.Lambda.DurableExecution; /// conveys only the value and a stream — /// it has no way to tell the serializer which operation or execution a value belongs to. /// Serializers that offload results to external storage (for example -/// FileSystemSerializer) need that identity to build a stable, unique +/// ) need that identity to build a stable, unique /// location and to avoid different operations clobbering one another. Serializers that /// do not need it simply implement and /// are used unchanged. diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs index 628256eda..f727e7085 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/ChildContextOperation.cs @@ -61,7 +61,7 @@ internal sealed class ChildContextOperation : DurableOperation /// verbatim, rather than re-serializing the (already round-tripped) return value. /// Re-serializing the return value double-transforms a non-round-tripping serializer /// (fresh vs replay diverge) and, for a context-aware serializer such as - /// FileSystemSerializer, writes a SECOND file at the parent's per-unit id + /// , writes a SECOND file at the parent's per-unit id /// that orphans the child's file. null for virtual (Flat) children, for /// overflow-replay re-execution, and before a fresh success has been serialized. /// diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/IDefaultInnerSerializer.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/IDefaultInnerSerializer.cs new file mode 100644 index 000000000..ca1f635dd --- /dev/null +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/IDefaultInnerSerializer.cs @@ -0,0 +1,22 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using Amazon.Lambda.Core; + +namespace Amazon.Lambda.DurableExecution; + +/// +/// Implemented by a per-operation serializer that wraps an inner +/// but may have been constructed without one, asking the +/// durable runtime to supply the globally-registered serializer as its inner. The runtime +/// calls when it resolves the effective serializer for an +/// operation (see the serializer resolution in DurableContext). +/// +internal interface IDefaultInnerSerializer +{ + /// + /// Returns a serializer bound to as its inner serializer. An + /// implementation that already has an explicitly-provided inner returns itself unchanged. + /// + ILambdaSerializer WithDefaultInner(ILambdaSerializer inner); +} diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/LambdaSerializerHelper.cs b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/LambdaSerializerHelper.cs index 261d2a33a..c9f0a492f 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/LambdaSerializerHelper.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/Internal/LambdaSerializerHelper.cs @@ -18,6 +18,16 @@ internal static class LambdaSerializerHelper public static ILambdaSerializer GetRequired(ILambdaContext lambdaContext) => lambdaContext.Serializer ?? throw new InvalidOperationException(MissingSerializerMessage); + /// + /// If was constructed to defer to the globally-registered + /// serializer for its inner format (see , e.g. the + /// inner-less constructor), binds + /// as its inner and returns the bound serializer; + /// otherwise returns unchanged. + /// + public static ILambdaSerializer WithDefaultInner(ILambdaSerializer serializer, ILambdaSerializer defaultInner) => + serializer is IDefaultInnerSerializer d ? d.WithDefaultInner(defaultInner) : serializer; + /// /// Serializes a durable operation result. If implements /// , the context-aware overload is used so the diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/FileSystemSerializerTests.cs b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/FileSystemSerializerTests.cs new file mode 100644 index 000000000..4aa090d5b --- /dev/null +++ b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/FileSystemSerializerTests.cs @@ -0,0 +1,431 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +using System.IO.Compression; +using System.Text; +using System.Text.Json; +using Amazon.Lambda.Core; +using Amazon.Lambda.Serialization.SystemTextJson; +using Xunit; + +namespace Amazon.Lambda.DurableExecution.Tests; + +/// +/// Unit tests for : storage modes (Always/Overflow), +/// path encodings (Uri/Hash), envelope shape, missing-file behavior, per-entity file +/// separation, composition with a compressing inner serializer, and the plain-path guard. +/// +public class FileSystemSerializerTests : IDisposable +{ + private readonly string _base = Path.Combine(Path.GetTempPath(), "fsser-" + Guid.NewGuid().ToString("N")); + private readonly ILambdaSerializer _json = new DefaultLambdaJsonSerializer(); + + public void Dispose() + { + try { if (Directory.Exists(_base)) Directory.Delete(_base, recursive: true); } catch { /* best effort */ } + } + + public sealed class Poco + { + public int Id { get; set; } + public string? Name { get; set; } + public override bool Equals(object? o) => o is Poco p && p.Id == Id && p.Name == Name; + public override int GetHashCode() => Id; + } + + private static DurableSerializationContext DurableArnCtx(string entity = "op-1") => + new(entity, "arn:aws:lambda:us-east-1:123456789012:function:fn:1/durable-execution/exec-1/inv-1"); + + private static string Serialize(FileSystemSerializer s, T value, DurableSerializationContext ctx) + { + using var ms = new MemoryStream(); + ((IDurableResultSerializer)s).Serialize(value, ms, ctx); + return Encoding.UTF8.GetString(ms.ToArray()); + } + + private static T Deserialize(FileSystemSerializer s, string envelope, DurableSerializationContext ctx) + { + using var ms = new MemoryStream(Encoding.UTF8.GetBytes(envelope)); + return ((IDurableResultSerializer)s).Deserialize(ms, ctx); + } + + [Fact] + public void Always_WritesFile_EnvelopeIsPointer_AndRoundTrips() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Always); + var value = new Poco { Id = 1, Name = "alice" }; + + var envelope = Serialize(s, value, DurableArnCtx()); + + Assert.Contains("\"file\"", envelope); + Assert.DoesNotContain("\"data\"", envelope); + Assert.NotEmpty(Directory.GetFiles(_base, "*.bin", SearchOption.AllDirectories)); + Assert.Equal(value, Deserialize(s, envelope, DurableArnCtx())); + } + + [Fact] + public void Overflow_SmallValue_StoredInline_NoFile() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Overflow); + var value = new Poco { Id = 7, Name = "small" }; + + var envelope = Serialize(s, value, DurableArnCtx()); + + Assert.Contains("\"data\"", envelope); + Assert.DoesNotContain("\"file\"", envelope); + Assert.False(Directory.Exists(_base) && Directory.GetFiles(_base, "*.bin", SearchOption.AllDirectories).Length > 0); + Assert.Equal(value, Deserialize(s, envelope, DurableArnCtx())); + } + + [Fact] + public void Overflow_LargeValue_OverflowsToFile_AndRoundTrips() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Overflow); + var big = new string('x', 300 * 1024); // > ~256KB threshold once serialized + + var envelope = Serialize(s, big, DurableArnCtx()); + + Assert.Contains("\"file\"", envelope); + Assert.Equal(big, Deserialize(s, envelope, DurableArnCtx())); + } + + [Fact] + public void Deserialize_MissingFile_ThrowsFileNotFound() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Always); + var envelope = Serialize(s, new Poco { Id = 2, Name = "gone" }, DurableArnCtx()); + + foreach (var f in Directory.GetFiles(_base, "*.bin", SearchOption.AllDirectories)) + File.Delete(f); + + Assert.Throws(() => Deserialize(s, envelope, DurableArnCtx())); + } + + [Fact] + public void Deserialize_FilePointerOutsideBase_Throws() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Always); + // A tampered/corrupted envelope pointing outside the base path must be rejected, + // not read (guards against arbitrary file reads on the mounted filesystem). + var evilPath = Path.Combine(Path.GetTempPath(), "evil-" + Guid.NewGuid().ToString("N") + ".bin"); + var envelope = "{\"file\":" + JsonSerializer.Serialize(evilPath) + "}"; + Assert.Throws(() => Deserialize(s, envelope, DurableArnCtx())); + } + + [Fact] + public void UriPathEncoding_BuildsPerExecutionDirsFromArn() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Always, FileSystemPathEncoding.Uri); + Serialize(s, new Poco { Id = 3, Name = "n" }, DurableArnCtx("entity-3")); + + var expectedDir = Path.Combine(_base, "fn", "exec-1", "inv-1"); + Assert.True(Directory.Exists(expectedDir), $"expected per-execution dir {expectedDir}"); + Assert.NotEmpty(Directory.GetFiles(expectedDir, "*.bin")); + } + + [Fact] + public void HashPathEncoding_UsesSha256HexSegments() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Always, FileSystemPathEncoding.Hash); + Serialize(s, new Poco { Id = 4, Name = "n" }, DurableArnCtx("entity-4")); + + var file = Assert.Single(Directory.GetFiles(_base, "*.bin", SearchOption.AllDirectories)); + // Directory segment (ARN hash) and file stem (entity hash) are 64-char lowercase hex. + var dirName = new DirectoryInfo(Path.GetDirectoryName(file)!).Name; + var stem = Path.GetFileNameWithoutExtension(file); + Assert.Matches("^[0-9a-f]{64}$", dirName); + Assert.Matches("^[0-9a-f]{64}$", stem); + } + + [Fact] + public void DistinctEntityIds_WriteDistinctFiles() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Always); + Serialize(s, new Poco { Id = 1, Name = "a" }, DurableArnCtx("op#0")); + Serialize(s, new Poco { Id = 2, Name = "b" }, DurableArnCtx("op#1")); + + var files = Directory.GetFiles(_base, "*.bin", SearchOption.AllDirectories); + Assert.Equal(2, files.Length); + } + + [Fact] + public void GzipInnerSerializer_Composes_CompressesFile_AndRoundTrips() + { + var gzip = new GzipJsonSerializer(_json); + var compressing = new FileSystemSerializer(gzip, _base, FileSystemStorageMode.Always); + var plain = new FileSystemSerializer(_json, _base + "-plain", FileSystemStorageMode.Always); + + var value = new string('a', 50 * 1024); // highly compressible + + var envelope = Serialize(compressing, value, DurableArnCtx("gz")); + Serialize(plain, value, DurableArnCtx("gz")); + + var compressedFile = Assert.Single(Directory.GetFiles(_base, "*.bin", SearchOption.AllDirectories)); + var plainFile = Assert.Single(Directory.GetFiles(_base + "-plain", "*.bin", SearchOption.AllDirectories)); + + Assert.True(new FileInfo(compressedFile).Length < new FileInfo(plainFile).Length, + "gzip inner serializer should produce a smaller file than the plain JSON inner"); + Assert.Equal(value, Deserialize(compressing, envelope, DurableArnCtx("gz"))); + + try { Directory.Delete(_base + "-plain", true); } catch { /* best effort */ } + } + + [Fact] + public void PlainLambdaSerializerPath_Throws() + { + var s = (ILambdaSerializer)new FileSystemSerializer(_json, _base); + using var ms = new MemoryStream(); + Assert.Throws(() => s.Serialize(new Poco { Id = 1 }, ms)); + Assert.Throws(() => s.Deserialize(new MemoryStream(Encoding.UTF8.GetBytes("{}")))); + } + + // ---- V7a / V4: crafted ARN must not escape the base path on the WRITE side ---- + + [Fact] + public void Serialize_CraftedArnWithTraversal_ThrowsOnWrite_DoesNotEscapeBase() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Always, FileSystemPathEncoding.Uri); + + // Matches the durable-execution ARN shape, but the captured function/execution + // segments are ".." — resolving the per-execution write directory would traverse + // above the configured base path. Uri encoding does NOT neutralize ".", so the + // write path must reject this (the read path already had ValidatePathWithinBase). + var evilArn = "arn:aws:lambda:us-east-1:123456789012:function:..:1/durable-execution/../inv-1"; + var ctx = new DurableSerializationContext("op-1", evilArn); + + var ex = Assert.Throws(() => Serialize(s, new Poco { Id = 1 }, ctx)); + Assert.Contains("base path", ex.Message); + + // Nothing was written outside (or inside) the base. + Assert.False(Directory.Exists(_base) && Directory.GetFiles(_base, "*.bin", SearchOption.AllDirectories).Length > 0); + } + + // ---- V7b: ARN that doesn't match the pattern falls back to a single encoded segment ---- + + [Fact] + public void UriPathEncoding_NonMatchingArn_FallsBackToSingleEncodedDirectorySegment() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Always, FileSystemPathEncoding.Uri); + + // Not a durable-execution ARN; it even contains slashes. The whole value must be + // URL-encoded into ONE directory segment directly under the base (slashes escaped + // to %2F), never split into nested directories. + var arn = "not-a-durable-execution-arn/with/slashes"; + var ctx = new DurableSerializationContext("entity-b", arn); + + var envelope = Serialize(s, new Poco { Id = 5, Name = "n" }, ctx); + + var expectedDir = Path.Combine(_base, Uri.EscapeDataString(arn)); + Assert.True(Directory.Exists(expectedDir), $"expected single fallback dir {expectedDir}"); + Assert.NotEmpty(Directory.GetFiles(expectedDir, "*.bin")); + // The base has exactly one immediate child directory (the encoded segment). + Assert.Single(Directory.GetDirectories(_base)); + Assert.Equal(new Poco { Id = 5, Name = "n" }, Deserialize(s, envelope, ctx)); + } + + // ---- V7d: overflow inline/file boundary is inclusive at OverflowThresholdBytes ---- + + /// Inner serializer that emits a fixed number of raw bytes, so a test can size the + /// overflow envelope to the byte and pin the inline/file boundary. + private sealed class FixedSizeSerializer : ILambdaSerializer + { + private readonly int _rawBytes; + public FixedSizeSerializer(int rawBytes) => _rawBytes = rawBytes; + public void Serialize(T response, Stream responseStream) => responseStream.Write(new byte[_rawBytes], 0, _rawBytes); + public T Deserialize(Stream requestStream) => default!; + } + + [Fact] + public void Overflow_Boundary_LargestInlineStaysInline_NextSizeOverflows() + { + // OverflowThresholdBytes = 256*1024 - 1024 (private). Decision: envelopeBytes + // <= threshold => inline. Base64 quantizes the payload to multiples of 4 bytes, + // so the exact threshold byte isn't individually addressable; instead we pin the + // transition: the largest raw payload whose envelope is <= threshold stays inline, + // and one raw byte more (which pushes the envelope past threshold) overflows. + const int threshold = (256 * 1024) - 1024; + + var inlineRaw = LargestRawInline(threshold); + + var inlineEnvelope = SerializeWithFixedInner(inlineRaw); + Assert.Contains("\"data\"", inlineEnvelope); + Assert.DoesNotContain("\"file\"", inlineEnvelope); + + var overflowEnvelope = SerializeWithFixedInner(inlineRaw + 1); + Assert.Contains("\"file\"", overflowEnvelope); + Assert.DoesNotContain("\"data\"", overflowEnvelope); + + string SerializeWithFixedInner(int rawBytes) + { + var s = new FileSystemSerializer(new FixedSizeSerializer(rawBytes), _base, FileSystemStorageMode.Overflow); + return Serialize(s, 0, DurableArnCtx("boundary-" + rawBytes)); + } + } + + // Largest raw byte count whose inline envelope ({"data":""}) is <= threshold. + private static int LargestRawInline(int threshold) + { + // Envelope overhead around the base64 body: {"data":"..."} = 9 + 2 = 11 bytes. + for (var n = threshold; n > 0; n--) + { + var base64Len = ((n + 2) / 3) * 4; + if (11 + base64Len <= threshold) + return n; + } + return 0; + } + + // ---- V6: corrupted inline base64 surfaces a descriptive InvalidOperationException ---- + + [Fact] + public void Deserialize_CorruptedInlineBase64_ThrowsDescriptiveInvalidOperationException() + { + var s = new FileSystemSerializer(_json, _base, FileSystemStorageMode.Overflow); + // A bare FormatException from Convert.FromBase64String would be inconsistent with + // this method's sibling descriptive InvalidOperationException checks. + var envelope = "{\"data\":\"not valid base64 !!!\"}"; + + var ex = Assert.Throws(() => Deserialize(s, envelope, DurableArnCtx())); + Assert.Contains("base64", ex.Message); + Assert.IsType(ex.InnerException); + } + + // ---- V5: a relative basePath is normalized to an absolute pointer in the ctor ---- + + [Fact] + public void Ctor_RelativeBasePath_ProducesAbsoluteFilePointer() + { + var relative = "fsser-rel-" + Guid.NewGuid().ToString("N"); + var s = new FileSystemSerializer(_json, relative, FileSystemStorageMode.Always); + try + { + var envelope = Serialize(s, new Poco { Id = 1, Name = "n" }, DurableArnCtx()); + + using var doc = JsonDocument.Parse(envelope); + var file = doc.RootElement.GetProperty("file").GetString()!; + // Normalizing _basePath in the ctor makes the stored pointer absolute, so a later + // invocation with a different CWD resolves it to the same file. + Assert.True(Path.IsPathFullyQualified(file), $"pointer should be absolute but was '{file}'"); + } + finally + { + try { Directory.Delete(Path.GetFullPath(relative), recursive: true); } catch { /* best effort */ } + } + } + + // ---- Comment 4: read handle must not block a concurrent atomic replace ---- + + /// Inner serializer whose Deserialize signals that the read stream is + /// open, then blocks until released — lets a test hold the real read FileStream open + /// and attempt a concurrent atomic replace (File.Move overwrite) over it. + private sealed class BlockingReadSerializer : ILambdaSerializer + { + private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); + public ManualResetEventSlim Reading { get; } = new(false); + public ManualResetEventSlim Proceed { get; } = new(false); + + public void Serialize(T response, Stream responseStream) => _inner.Serialize(response, responseStream); + + public T Deserialize(Stream requestStream) + { + Reading.Set(); + if (!Proceed.Wait(TimeSpan.FromSeconds(10))) + throw new TimeoutException("Proceed signal was not received."); + return _inner.Deserialize(requestStream); + } + } + + [Fact] + public async Task Deserialize_ReadHandleOpen_DoesNotBlockConcurrentAtomicReplace() + { + // NOTE: this test only meaningfully exercises the FileShare.Delete regression on + // Windows/macOS. On POSIX/Linux (CI), rename-over-an-open-fd (File.Move overwrite) + // ALWAYS succeeds regardless of the reader's share flags — the reader keeps the old + // inode — so on Linux the Assert.Null(moveEx) below would pass even WITHOUT + // FileShare.Delete, making it tautological there. It is the Windows/macOS run + // (dev/test) that actually guards the fix, where a read handle opened without + // FileShare.Delete blocks the rename-over and throws IOException. + // + // The read path opens the offloaded file with FileShare.Read | FileShare.Delete so + // a concurrent re-serialize (File.Move overwrite, the write path's atomic replace) + // can proceed while a replay-read is in flight. On Linux the open fd references the + // original inode regardless; on Windows/macOS a read handle opened WITHOUT + // FileShare.Delete blocks the rename-over and throws IOException. This test holds + // the real read FileStream open and asserts the replace succeeds. + var blocking = new BlockingReadSerializer(); + var s = new FileSystemSerializer(blocking, _base, FileSystemStorageMode.Always); + + var envelope = Serialize(s, new Poco { Id = 1, Name = "orig" }, DurableArnCtx("op#0")); + using var doc = JsonDocument.Parse(envelope); + var filePath = doc.RootElement.GetProperty("file").GetString()!; + + // Open our real read FileStream, then block inside the inner deserialize while the + // handle is still open. + var readTask = Task.Run(() => Deserialize(s, envelope, DurableArnCtx("op#0"))); + Assert.True(blocking.Reading.Wait(TimeSpan.FromSeconds(10)), "the read never opened the file handle"); + + // Replace the file the way the write path does: a sibling temp file moved over the + // target while the reader holds it open. + var tmp = filePath + ".tmp-" + Guid.NewGuid().ToString("N"); + File.WriteAllBytes(tmp, Encoding.UTF8.GetBytes("{\"Id\":2,\"Name\":\"replaced\"}")); + var moveEx = Record.Exception(() => File.Move(tmp, filePath, overwrite: true)); + Assert.Null(moveEx); // FileShare.Delete keeps the in-flight reader from blocking the replace + + // Release the reader; via its already-open handle it still reads the ORIGINAL bytes. + blocking.Proceed.Set(); + var read = await readTask; + Assert.Equal(new Poco { Id = 1, Name = "orig" }, read); + } + + [Fact] + public void InnerLessConstructor_BoundToDefaultInner_RoundTrips() + { + // The inner-less constructor defers to the globally-registered serializer, which the + // durable runtime binds via IDefaultInnerSerializer.WithDefaultInner before use. + var innerLess = new FileSystemSerializer(_base, FileSystemStorageMode.Always); + var bound = (FileSystemSerializer)((IDefaultInnerSerializer)innerLess).WithDefaultInner(_json); + var value = new Poco { Id = 9, Name = "bound" }; + + var envelope = Serialize(bound, value, DurableArnCtx()); + + Assert.Contains("\"file\"", envelope); + Assert.Equal(value, Deserialize(bound, envelope, DurableArnCtx())); + } + + [Fact] + public void WithDefaultInner_ExplicitInnerWins_ReturnsSameInstance() + { + // A caller-supplied inner is never overridden by the runtime's default. + var explicitInner = new FileSystemSerializer(_json, _base); + var result = ((IDefaultInnerSerializer)explicitInner).WithDefaultInner(new DefaultLambdaJsonSerializer()); + Assert.Same(explicitInner, result); + } + + [Fact] + public void InnerLessConstructor_WithoutBinding_Throws() + { + // Used without the runtime binding a default inner (e.g. outside a durable operation + // slot), there is no serializer to convert the value — fail with a clear message. + var innerLess = new FileSystemSerializer(_base, FileSystemStorageMode.Always); + Assert.Throws( + () => Serialize(innerLess, new Poco { Id = 1, Name = "x" }, DurableArnCtx())); + } + + private sealed class GzipJsonSerializer : ILambdaSerializer + { + private readonly ILambdaSerializer _json; + public GzipJsonSerializer(ILambdaSerializer json) => _json = json; + + public void Serialize(T response, Stream responseStream) + { + using var gz = new GZipStream(responseStream, CompressionLevel.Optimal, leaveOpen: true); + _json.Serialize(response, gz); + } + + public T Deserialize(Stream requestStream) + { + using var gz = new GZipStream(requestStream, CompressionMode.Decompress, leaveOpen: true); + return _json.Deserialize(gz); + } + } +} From a797eb020ad23bb8aea15b8b7b34903c8aeeebfe Mon Sep 17 00:00:00 2001 From: Garrett Beatty Date: Fri, 4 Sep 2026 01:11:42 +0000 Subject: [PATCH 4/5] docs(DurableExecution): document FileSystemSerializer and inner-less constructor Add a 'FileSystemSerializer' subsection to the steps custom-serializer docs covering large-result offload to a durable mount, both the inner-taking and the new inner-less constructor (runtime binds the globally-registered serializer as the inner), storage/path-encoding modes, and the retention caveat. --- .../docs/core/steps.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/steps.md b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/steps.md index 0eb07c551..e00b5893e 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/steps.md +++ b/Libraries/src/Amazon.Lambda.DurableExecution/docs/core/steps.md @@ -165,3 +165,32 @@ var report = await ctx.StepAsync( The serializer is part of the workflow's deterministic definition: it is re-resolved on every replay, so a step must be able to deserialize a result it previously serialized. Changing a step's serializer for an in-flight execution in a way that cannot read the stored payload will break replay. The override is Native-AOT safe as long as the serializer you supply is (for example `SourceGeneratorLambdaJsonSerializer`). > The same optional `Serializer` property is available on `CallbackConfig` (used to deserialize the callback payload), `InvokeConfig` (payload and result), `WaitForConditionConfig` (the checkpointed state), and `ChildContextConfig` (the child result). `MapConfig` and `ParallelConfig` expose `ItemSerializer` for each item/branch **result** (the aggregated batch envelope — per-item statuses and completion reason — is SDK-internal and not user-serialized). In every case, `null` means the globally-registered serializer is used. + +#### Offloading large results with `FileSystemSerializer` + +When a step's result would be too large to keep in the checkpoint (the durable execution checkpoint size limit is ~256 KB), assign a `FileSystemSerializer` to the per-operation `Serializer` slot. It writes the serialized result to a durable, shared mount (Amazon EFS or Amazon S3 Files — **not** Lambda's ephemeral `/tmp`, which does not survive replay) and keeps only a small file pointer in the checkpoint. `FileSystemSerializer` wraps an *inner* `ILambdaSerializer` that does the actual value↔bytes conversion, so you stay in control of the on-the-wire format. + +```csharp +var report = await ctx.StepAsync( + async (_, ct) => await BuildLargeReportAsync(ct), + name: "report", + config: new StepConfig + { + // Inner serializer supplied explicitly. + Serializer = new FileSystemSerializer( + inner: new SourceGeneratorLambdaJsonSerializer(), + basePath: "/mnt/efs/durable") + }); +``` + +When the on-the-wire format is just the function's normal serializer, use the inner-less constructor and let the durable runtime supply the globally-registered `ILambdaSerializer` (the assembly-level `[assembly: LambdaSerializer(...)]`, or the one passed to `LambdaBootstrapBuilder.Create(handler, serializer)`) as the inner — you no longer have to thread `ctx.Serializer` in yourself: + +```csharp +config: new StepConfig +{ + // No inner: the durable runtime binds the global serializer as the inner. + Serializer = new FileSystemSerializer(basePath: "/mnt/efs/durable") +} +``` + +An explicitly-supplied inner always wins over the global one. The inner is bound only when `FileSystemSerializer` is used through a per-operation serializer slot; used without a bound inner (for example as the assembly-registered serializer) it throws `InvalidOperationException`. `FileSystemStorageMode` (`Always` vs. `Overflow`) controls whether every value is written to a file or only those that would overflow the inline checkpoint; `FileSystemPathEncoding` controls how paths are derived. `FileSystemSerializer` never deletes result files, so pair the base path with an external retention policy (an EFS lifecycle policy, an S3 lifecycle rule, or a scheduled cleanup). This works the same way for any per-operation serializer slot — `CallbackConfig`/`InvokeConfig`/`WaitForConditionConfig`/`ChildContextConfig.Serializer` and `MapConfig`/`ParallelConfig.ItemSerializer`. From 7d04f74c17a6aa79eaa465570d2040ece503a02b Mon Sep 17 00:00:00 2001 From: Garrett Beatty Date: Tue, 8 Sep 2026 14:44:23 -0400 Subject: [PATCH 5/5] fix(DurableExecution): resolve symlinks at every path component in FileSystemSerializer containment check (#2563) IsWithinBase resolved the base path through ResolveReal, which only followed a symlink on the leaf and its immediate parent. When the configured base path itself is a symlink (e.g. an EFS mount exposed as /mnt/link -> /mnt/real), the base resolved to /mnt/real while candidate paths built under /mnt/link kept the unresolved symlink component (it sits above their leaf/parent). The resulting StartsWith prefix check failed, so every offloaded read and write was rejected. Resolve symlinks at every existing component of the path so the base and the candidates are canonicalized identically regardless of where a symlink sits. Add a test with a symlinked base leaf; it fails before this change. Co-authored-by: Garrett Beatty --- .../FileSystemSerializer.cs | 54 +++++++++++++------ .../FileSystemSerializerTests.cs | 50 +++++++++++++++++ 2 files changed, 89 insertions(+), 15 deletions(-) diff --git a/Libraries/src/Amazon.Lambda.DurableExecution/FileSystemSerializer.cs b/Libraries/src/Amazon.Lambda.DurableExecution/FileSystemSerializer.cs index eb007e698..fefce5814 100644 --- a/Libraries/src/Amazon.Lambda.DurableExecution/FileSystemSerializer.cs +++ b/Libraries/src/Amazon.Lambda.DurableExecution/FileSystemSerializer.cs @@ -407,32 +407,56 @@ private bool IsWithinBase(string candidate) } // Best-effort resolution of a path to its real on-disk location, following a - // symlink on the leaf file/directory and on its immediate parent directory. - // Path.GetFullPath only collapses '..'/separators and does NOT follow symlinks, - // so without this a symlink planted under the base but pointing outside it would - // pass a purely lexical prefix check and defeat the containment guard. Both the - // base and the candidate are resolved the same way so a symlinked mount root - // (for example macOS /var -> /private/var) does not cause false rejections. + // symlink at EVERY existing component of the path — not just the leaf and its + // immediate parent. Path.GetFullPath only collapses '..'/separators and does NOT + // follow symlinks, so without this a symlink planted under the base but pointing + // outside it would pass a purely lexical prefix check and defeat the containment + // guard. Resolving every component also means the base and the candidate are + // canonicalized identically no matter where a symlink sits in the path: a + // symlinked mount root used *as* the base (for example an EFS mount exposed as + // /mnt/link -> /mnt/real) resolves the same way the candidate paths built under + // it do, so containment is decided on real on-disk locations and legitimate + // reads/writes are not falsely rejected. A symlink above the base (for example + // macOS /var -> /private/var) is likewise resolved symmetrically on both sides. + // Components that do not exist yet (a per-execution directory validated before it + // is created) cannot be a symlink, so they are appended lexically. private static string ResolveReal(string fullPath) { try { - var dir = Path.GetDirectoryName(fullPath); - if (dir != null && Directory.Exists(dir)) + var root = Path.GetPathRoot(fullPath); + if (string.IsNullOrEmpty(root)) + return fullPath; // Not rooted (shouldn't happen post-GetFullPath); nothing to resolve against. + + var current = root; + var rest = fullPath.Substring(root.Length); + foreach (var segment in rest.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar)) { - var realDir = Directory.ResolveLinkTarget(dir, returnFinalTarget: true)?.FullName ?? dir; - fullPath = Path.Combine(realDir, Path.GetFileName(fullPath)); + if (segment.Length == 0) + continue; + + current = Path.Combine(current, segment); + + // Follow a symlink at this component (returnFinalTarget walks a chain + // of links to its ultimate target). A non-existent component or a + // regular file/dir yields null and is kept as-is. + var resolved = Directory.Exists(current) + ? Directory.ResolveLinkTarget(current, returnFinalTarget: true)?.FullName + : File.Exists(current) + ? File.ResolveLinkTarget(current, returnFinalTarget: true)?.FullName + : null; + + if (resolved != null) + current = resolved; } - if (File.Exists(fullPath)) - fullPath = File.ResolveLinkTarget(fullPath, returnFinalTarget: true)?.FullName ?? fullPath; - else if (Directory.Exists(fullPath)) - fullPath = Directory.ResolveLinkTarget(fullPath, returnFinalTarget: true)?.FullName ?? fullPath; + + return current; } catch { // Best effort — fall back to the lexical full path. + return fullPath; } - return fullPath; } private string ResolveExecutionDir(string arn) diff --git a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/FileSystemSerializerTests.cs b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/FileSystemSerializerTests.cs index 4aa090d5b..c6f5bf7fc 100644 --- a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/FileSystemSerializerTests.cs +++ b/Libraries/test/Amazon.Lambda.DurableExecution.Tests/FileSystemSerializerTests.cs @@ -200,6 +200,56 @@ public void Serialize_CraftedArnWithTraversal_ThrowsOnWrite_DoesNotEscapeBase() Assert.False(Directory.Exists(_base) && Directory.GetFiles(_base, "*.bin", SearchOption.AllDirectories).Length > 0); } + // ---- Symlinked base: a base path whose leaf is a symlink must not falsely reject reads/writes ---- + + [Fact] + public void SymlinkedBasePath_RoundTripsAndStaysContained() + { + // Model a real-world durable mount exposed through a symlink (e.g. an EFS mount + // surfaced as /mnt/link -> /mnt/real). The base path handed to the serializer is + // the *symlink*; every candidate path is built underneath it. The containment + // guard resolves the symlinked base to its real target, so it must resolve the + // candidates through the same symlink or it would reject every read and write. + var realDir = Path.Combine(Path.GetTempPath(), "fsser-real-" + Guid.NewGuid().ToString("N")); + var linkDir = Path.Combine(Path.GetTempPath(), "fsser-link-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(realDir); + try + { + try + { + Directory.CreateSymbolicLink(linkDir, realDir); + } + catch (Exception ex) when (ex is UnauthorizedAccessException or IOException or PlatformNotSupportedException) + { + // Creating symlinks can require elevation (Windows without Developer Mode). + // The behavior under test is filesystem-level, so skip where we cannot set up. + return; + } + + var s = new FileSystemSerializer(_json, linkDir, FileSystemStorageMode.Always); + var value = new Poco { Id = 42, Name = "efs" }; + + // Write through the symlinked base — must not throw "outside the configured base path". + var envelope = Serialize(s, value, DurableArnCtx()); + Assert.Contains("\"file\"", envelope); + + // Read back through the same base — must not be rejected by the containment guard. + var round = Deserialize(s, envelope, DurableArnCtx()); + Assert.Equal(value, round); + + // The payload physically landed under the real target directory. + Assert.True( + Directory.GetFiles(realDir, "*.bin", SearchOption.AllDirectories).Length > 0, + "offloaded payload should be written under the symlink's real target"); + } + finally + { + // Delete the link itself (non-recursive) so cleanup never follows into the target. + try { Directory.Delete(linkDir, recursive: false); } catch { /* best effort */ } + try { Directory.Delete(realDir, recursive: true); } catch { /* best effort */ } + } + } + // ---- V7b: ARN that doesn't match the pattern falls back to a single encoded segment ---- [Fact]