Feature/durable result serializer - #2561
Merged
Merged
Conversation
* feat(DurableExecution): add per-operation serializer override Add an optional `ILambdaSerializer? Serializer` to StepConfig, CallbackConfig, InvokeConfig, WaitForConditionConfig<TState>, and ChildContextConfig. DurableContext resolves `config?.Serializer ?? ILambdaContext.Serializer` per operation; the operation classes are otherwise unchanged. When Serializer is null (default), behavior is identical to today (the globally-registered serializer). Boundary serialization (workflow input, handler return, service envelopes) is untouched. Reuses the existing ILambdaSerializer contract (no new SerDes interface); AOT-safe because T stays concrete at each call site. Additive / non-breaking. - Unit tests (PerOperationSerializerTests) covering per-op serialize, default fallback, and replay-deserialize for Step/Callback/Invoke/ChildContext/WaitForCondition. - Integration test (TestFunctions/PerOperationSerializerFunction + PerOperationSerializerTest) verifying per-step routing end-to-end (camelCase override vs PascalCase default). - docs/core updates and an AutoVer change file (Minor). Separate feature from #2540 (FileSystem offload). Phase 1 (single-result ops); Map/Parallel two-level slots are a follow-up (Phase 2). * feat(DurableExecution): add per-item serializer for Map/Parallel Add an optional `ILambdaSerializer? ItemSerializer` to MapConfig<TItem> and ParallelConfig. DurableContext.RunMap/RunParallel resolve `config.ItemSerializer ?? ILambdaContext.Serializer` and pass it as the serializer used for per-item/branch results (each unit's child-context checkpoint and the inline copy on the operation's summary). The aggregated batch envelope (per-unit statuses + completion reason) is a source-generated structure and is unchanged — there is no separate whole-result serializer (matches the Java SDK). Additive / non-breaking. - Unit tests: Map_WithItemSerializer, Map_NoItemSerializer, Parallel_WithItemSerializer. - docs/core (parallel.md, steps.md) + AutoVer change file. Phase 2 of the per-operation serializer feature (Phase 1 = single-result ops). * test(DurableExecution): cloud integration test for Map/Parallel ItemSerializer Deploys MapParallelItemSerializerFunction and asserts from event history that map items and a parallel branch configured with a camelCase ItemSerializer produce camelCase per-item child-context result payloads, while a control step (global serializer) stays PascalCase. Validates Phase 2 end-to-end on AWS. * address review: consistent serializer resolution, config headers, serializer unit coverage - DurableContext: Parallel now resolves the effective serializer via LambdaSerializerHelper.GetRequired(LambdaContext) like every other operation, instead of an inline 'LambdaContext.Serializer ?? throw' (consistent error path). - Add the standard copyright/SPDX header to MapConfig.cs and WaitForConditionConfig.cs. - Add unit tests: Invoke fresh (non-replay) path serializes the request payload via the per-op serializer; Map/Parallel ItemSerializer is used to deserialize cached per-item/per-branch results on replay. * test(DurableExecution): per-op serdes conformance + fresh-success serializer round-trip (#2556) * test(DurableExecution): conformance handlers for per-item/result serdes (map 9-14, parallel 8-15, invoke 5-16) Add conformance handlers exercising the new per-operation serializer slots: - map/MapCustomSerdes (9-14): MapConfig.ItemSerializer wraps each item result - parallel/ParallelCustomSerdes (8-15): ParallelConfig.ItemSerializer wraps each branch result - invoke/InvokeCustomResultSerdes (5-16): InvokeConfig.Serializer uppercases the result on deserialize Remove the corresponding NotImplemented declarations and wire the resources in template_map/parallel/invoke.yaml. All three pass against real AWS via the runner. Packaging fix: add Conformance/Directory.Build.props supplying the SDK's runtime NuGet dependencies (AWSSDK.Lambda, Microsoft.Extensions.Logging.Abstractions) to every handler, since those transitive package assets do not flow into a handler's net8.0 framework-dependent publish (WrapAsyncCore threw FileNotFoundException at runtime). Drop the now-redundant per-handler AWSSDK.Lambda references. * feat(DurableExecution): round-trip step/child results through the serializer on fresh success On a fresh (non-replay) success, StepOperation and ChildContextOperation now return the value deserialized from the just-written checkpoint instead of the original in-memory object, matching replay semantics. This makes a custom per-operation serializer's transform observable in the operation result on the first execution, not only on replay (a non-round-tripping serializer previously had no effect on the fresh result). Overflow (replay-children) child results are unaffected — the payload is stripped and the value is recovered by re-execution. Behavior change documented in the AutoVer change file. * test(DurableExecution): real per-op serdes conformance handlers for step 1-6 and child 3-14 Refactor StepCustomSerdes (1-6) to use a real StepConfig.Serializer (uppercase-on-serialize) instead of transforming inside the step body, and add child/ChildCustomSerdes (3-14) using ChildContextConfig.Serializer. Both rely on the fresh-success round-trip so the serialize-side transform reaches the result. Remove the 3-14 NotImplemented and wire the resource in template_child.yaml. Both pass against real AWS via the runner. * address review: harden conformance serde handlers - Custom serializers now guard typeof(T) == typeof(string) and throw a clear NotSupportedException for unsupported result types (instead of an opaque InvalidCastException from the (T)(object) cast). - Read with an explicit UTF-8 StreamReader for symmetry with the UTF-8 bytes written. - MapCustomSerdes: drop the async-without-await iteration lambda (CS1998); return Task.FromResult(...) so no per-item async state machine is allocated. - InvokeCustomResultSerdes: correct the header comment — the serializer serializes the outbound request payload normally on the initial execution and applies the uppercase transform on deserialize (which, for a chained invoke, happens on replay). The handlers still transform the raw serialized payload, which is the behavior the conformance requirements assert (e.g. invoke 5-16 ExpectedResult '"HELLO"'); the string path is byte-identical, so the suite results are unchanged. * fix(DurableExecution): mark fresh-success round-trip change Major The fresh-success serializer round-trip changes the observable return value (fresh deserialized instance instead of the object the body produced, and a non-round-tripping serializer's transform now visible on the first run). At the released 1.0.0 baseline this is a breaking change and must be Major, not Minor. * chore(DurableExecution): normalize 12a4a1f7 change file (strip trailing newline) to match canonical content
…l determinism (#2559) Introduces the serializer engine that the FileSystemSerializer builds on, split out of the original filesystem PR (#2558) for reviewability: - Optional IDurableResultSerializer + DurableSerializationContext (EntityId + DurableExecutionArn) and LambdaSerializerHelper dispatch; plain ILambdaSerializer serializers are unaffected (byte-identical fallback). - Extends the fresh-success serializer round-trip (established for Step/Child in the conformance work) to Map/Parallel Flat per-item results, eliminating a fresh-vs-replay divergence for non-round-tripping ItemSerializers. - Map/Parallel Nested parent now inlines each child's ORIGINAL SUCCEED payload verbatim instead of re-serializing the round-tripped value. - Terminal handling: a fresh-success round-trip deserialize failure (Step / Child / Flat) is terminal (no retry, body never re-run); Step post-SUCCEED enqueue failure routed through the same terminal path. These changes are triggered by the per-operation serializer feature (#2555), not by FileSystemSerializer specifically; FileSystemSerializer is the leaf that exercises them. Note: two <see cref="FileSystemSerializer"/> doc links are rendered as <c>...</c> here and restored to cref links in the FileSystemSerializer PR that introduces the type.
…to a filesystem (#2540) (#2558) * feat(DurableExecution): FileSystemSerializer for offloading large results Adds FileSystemSerializer (implements ILambdaSerializer + IDurableResultSerializer): stores a durable operation's serialized result on a filesystem (EFS / S3 Files, NOT Lambda /tmp) and keeps only a small {file}|{data} envelope in the checkpoint. Wraps an inner ILambdaSerializer so callers control the on-the-wire format; supports Always/Overflow storage modes and Uri/Hash path encoding; envelope is source-generated (AOT-safe). The plain ILambdaSerializer path throws (offload needs the durable context). Stacked on the serializer-engine PR; restores the two <see cref="FileSystemSerializer"/> doc links now that the type exists. Includes the FileSystemSerializer AutoVer (Minor) change file and unit tests. * address review: inner-less FileSystemSerializer ctor uses global serializer normj asked for a constructor that omits the inner ILambdaSerializer and falls back to the durable execution's globally-registered serializer. Adds FileSystemSerializer(basePath, storageMode, pathEncoding) which leaves _inner null. The durable runtime binds the global serializer as the inner when it resolves the effective per-operation serializer, via a new internal IDefaultInnerSerializer capability (DurableContext.WithDefaultInner at the Step/ChildContext/WaitForCondition/Parallel/Map resolution sites). An explicitly-supplied inner always wins; used without a bound inner it throws a clear InvalidOperationException.
…constructor Add a 'FileSystemSerializer' subsection to the steps custom-serializer docs covering large-result offload to a durable mount, both the inner-taking and the new inner-less constructor (runtime binds the globally-registered serializer as the inner), storage/path-encoding modes, and the retention caveat.
GarrettBeatty
marked this pull request as ready for review
September 4, 2026 14:38
jnunn-aws
self-requested a review
September 4, 2026 16:06
jnunn-aws
reviewed
Sep 4, 2026
jnunn-aws
left a comment
jnunn-aws
left a comment
Contributor
There was a problem hiding this comment.
Please update empty PR description.
jnunn-aws
reviewed
Sep 4, 2026
| } | ||
| ``` | ||
|
|
||
| An explicitly-supplied inner always wins over the global one. The inner is bound only when `FileSystemSerializer` is used through a per-operation serializer slot; used without a bound inner (for example as the assembly-registered serializer) it throws `InvalidOperationException`. `FileSystemStorageMode` (`Always` vs. `Overflow`) controls whether every value is written to a file or only those that would overflow the inline checkpoint; `FileSystemPathEncoding` controls how paths are derived. `FileSystemSerializer` never deletes result files, so pair the base path with an external retention policy (an EFS lifecycle policy, an S3 lifecycle rule, or a scheduled cleanup). This works the same way for any per-operation serializer slot — `CallbackConfig`/`InvokeConfig`/`WaitForConditionConfig<TState>`/`ChildContextConfig.Serializer` and `MapConfig<TItem>`/`ParallelConfig.ItemSerializer`. |
Contributor
There was a problem hiding this comment.
FileSystemSerializer does not work on Callback/Invoke slots
This paragraph says the offload mechanism "works the same way for any per-operation serializer slot — CallbackConfig/InvokeConfig/WaitForConditionConfig<TState>/ChildContextConfig.Serializer and MapConfig<TItem>/ParallelConfig.ItemSerializer." That's not true for Callback and Invoke.
CallbackOperation.cs:214 and InvokeOperation.cs:138,147 call the serializer through the plain ILambdaSerializer methods (_serializer.Serialize/Deserialize), with no DurableSerializationContext. Those methods on FileSystemSerializer throw NotSupportedException (FileSystemSerializer.cs:270-279), and the inner-less variant additionally throws at RequireInner() (287-292). So a FileSystemSerializer in a Callback/Invoke slot fails at runtime.
Offloading may also be a less natural fit for these two: the Invoke payload is read by the callee, and the Callback payload is supplied by the external system, so a local result file has no counterpart there. Could scope the offload docs to the Step/ChildContext/WaitForCondition/Map/Parallel result slots, and note that Callback/Invoke support a plain ILambdaSerializer override.
Contributor
Author
There was a problem hiding this comment.
That's not true for Callback and Invoke.
thats true and intentionally this way. the reason being is these call other lambda functions and it doesnt make sense to use the file system serializer in this case
Contributor
Author
There was a problem hiding this comment.
ill leave this for now and update the wording in a follow up if needed.
peterrsongg
approved these changes
Sep 4, 2026
…leSystemSerializer containment check (#2563) IsWithinBase resolved the base path through ResolveReal, which only followed a symlink on the leaf and its immediate parent. When the configured base path itself is a symlink (e.g. an EFS mount exposed as /mnt/link -> /mnt/real), the base resolved to /mnt/real while candidate paths built under /mnt/link kept the unresolved symlink component (it sits above their leaf/parent). The resulting StartsWith prefix check failed, so every offloaded read and write was rejected. Resolve symlinks at every existing component of the path so the base and the candidates are canonicalized identically regardless of where a symlink sits. Add a test with a symlinked base leaf; it fails before this change. Co-authored-by: Garrett Beatty <GarrettBeatty@users.noreply.github.com>
jnunn-aws
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
merges already approved prs for per operation serializer and file system serializer