Python: fix(workflows): preserve all trace contexts in FanInEdgeRunner aggregation - #7557
Python: fix(workflows): preserve all trace contexts in FanInEdgeRunner aggregation#7557badhope (weed33834) wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes distributed tracing propagation in the Python workflow engine’s fan-in edge runner by ensuring nested fan-in aggregations preserve all upstream trace contexts and source span IDs, rather than silently dropping everything except the first element per message.
Changes:
- Update
FanInEdgeRunneraggregation to iterate overWorkflowMessage.trace_contexts/source_span_idsand aggregate all entries. - Add a regression test that simulates nested fan-in behavior by sending a message containing multiple trace contexts and asserting the target receives the full aggregated set.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| python/packages/core/agent_framework/_workflows/_edge_runner.py | Adjusts fan-in aggregation to preserve multiple trace contexts/source span IDs per incoming message. |
| python/packages/core/tests/workflow/test_edge.py | Adds a regression test to ensure fan-in preserves multiple trace contexts per message. |
| trace_contexts: list[dict[str, str]] = [] | ||
| source_span_ids: list[str] = [] | ||
| for msg in messages_to_send: | ||
| if msg.trace_contexts: | ||
| trace_contexts.extend(msg.trace_contexts) | ||
| if msg.source_span_ids: | ||
| source_span_ids.extend(msg.source_span_ids) |
There was a problem hiding this comment.
Addressed in 6fdc8b5. Trace contexts and source span IDs are now paired per-message via zip(msg_contexts, msg_span_ids, strict=False) before flattening, so a message with mismatched counts only drops its own orphans instead of shifting all subsequent pairs out of alignment.
| @handler | ||
| async def mock_aggregator_handler(self, message: list[MockMessage], ctx: WorkflowContext) -> None: | ||
| self.call_count += 1 | ||
|
|
||
| async def execute( | ||
| self, | ||
| message: WorkflowMessage, | ||
| source_executor_ids: list[str], | ||
| state: Any, | ||
| ctx: Any, | ||
| *, | ||
| trace_contexts: list[dict[str, str]] | None = None, | ||
| source_span_ids: list[str] | None = None, | ||
| ) -> None: | ||
| self.captured_trace_contexts = trace_contexts | ||
| self.captured_source_span_ids = source_span_ids | ||
| await super().execute( | ||
| message, | ||
| source_executor_ids, | ||
| state, | ||
| ctx, | ||
| trace_contexts=trace_contexts, | ||
| source_span_ids=source_span_ids, | ||
| ) |
There was a problem hiding this comment.
Addressed in 6fdc8b5. Removed the execute() override entirely. The handler now captures trace data from ctx._trace_contexts and ctx._source_span_ids on the WorkflowContext passed to it, avoiding the fragile override of Executor.execute().
|
@microsoft-github-policy-service agree |
…ation FanInEdgeRunner collected trace contexts and source span IDs using the singular backward-compat properties (msg.trace_context / msg.source_span_id), which return only the first element of the plural lists. When a message arriving at a fan-in already carries multiple trace contexts (e.g. from a prior fan-in aggregation), all but the first were silently dropped. Iterate over the plural fields (trace_contexts / source_span_ids) and extend the aggregated lists so every trace context and source span ID from every source message is preserved. This keeps distributed tracing links intact for nested fan-in topologies. Added test_fan_in_preserves_multiple_trace_contexts_per_message that sends a message with two trace contexts through a fan-in and asserts all three contexts (2 + 1) reach the target executor.
1. Pair trace_contexts and source_span_ids per-message (via zip) instead of flattening independently. This prevents misalignment when a message has mismatched counts — orphans are dropped per-message rather than shifting all subsequent pairs out of alignment. 2. Remove TraceCapturingAggregator's override of Executor.execute() (documented as "do not override"). Capture trace data from the WorkflowContext passed to the handler instead.
58fd192 to
6fdc8b5
Compare
Problem
FanInEdgeRunnercollected trace contexts and source span IDs using the singular backward-compat properties (msg.trace_context/msg.source_span_id), which return only the first element of the plural lists:When a message arriving at a fan-in already carries multiple trace contexts (e.g. from a prior fan-in aggregation in a nested topology), all but the first context per message are silently dropped. This breaks distributed tracing span links for any workflow with nested fan-in groups.
Fix
Iterate over the plural fields (
trace_contexts/source_span_ids) andextendthe aggregated lists so every trace context and source span ID from every source message is preserved:Test
Added
test_fan_in_preserves_multiple_trace_contexts_per_messagewhich:This test fails on the old code (
assert 2 == 3) and passes with the fix.All existing workflow and edge tests continue to pass (55 passed; the only 2 failures in the broader suite are pre-existing and caused by a missing
openaioptional dependency, unrelated to this change).Checklist