Skip to content

Python: fix(workflows): preserve all trace contexts in FanInEdgeRunner aggregation - #7557

Open
badhope (weed33834) wants to merge 2 commits into
microsoft:mainfrom
weed33834:fix/fanin-trace-context-aggregation
Open

Python: fix(workflows): preserve all trace contexts in FanInEdgeRunner aggregation#7557
badhope (weed33834) wants to merge 2 commits into
microsoft:mainfrom
weed33834:fix/fanin-trace-context-aggregation

Conversation

@weed33834

Copy link
Copy Markdown

Problem

FanInEdgeRunner collected trace contexts and source span IDs using the singular backward-compat properties (msg.trace_context / msg.source_span_id), which return only the first element of the plural lists:

# Before (buggy)
trace_contexts = [msg.trace_context for msg in messages_to_send if msg.trace_context]
source_span_ids = [msg.source_span_id for msg in messages_to_send if msg.source_span_id]

When a message arriving at a fan-in already carries multiple trace contexts (e.g. from a prior fan-in aggregation in a nested topology), all but the first context per message are silently dropped. This breaks distributed tracing span links for any workflow with nested fan-in groups.

Fix

Iterate over the plural fields (trace_contexts / source_span_ids) and extend the aggregated lists so every trace context and source span ID from every source message is preserved:

# After (fixed)
trace_contexts: list[dict[str, str]] = []
source_span_ids: list[str] = []
for msg in messages_to_send:
    if msg.trace_contexts:
        trace_contexts.extend(msg.trace_contexts)
    if msg.source_span_ids:
        source_span_ids.extend(msg.source_span_ids)

Test

Added test_fan_in_preserves_multiple_trace_contexts_per_message which:

  1. Sends a message carrying two trace contexts through a fan-in edge group (simulating a prior fan-in aggregation)
  2. Sends a second message with one trace context
  3. Asserts the target executor receives all 3 trace contexts and 3 source span IDs (not just 2)

This test fails on the old code (assert 2 == 3) and passes with the fix.

All existing workflow and edge tests continue to pass (55 passed; the only 2 failures in the broader suite are pre-existing and caused by a missing openai optional dependency, unrelated to this change).

Checklist

  • Code follows existing style conventions
  • Unit test added and passing
  • No breaking changes — the singular properties still exist for backward compatibility; only the fan-in aggregation path now uses the plural fields
  • No new dependencies introduced

Copilot AI lite review requested due to automatic review settings August 7, 2026 04:10
@agent-framework-automation agent-framework-automation Bot added the python Usage: [Issues, PRs], Target: Python label Aug 7, 2026
@github-actions github-actions Bot changed the title fix(workflows): preserve all trace contexts in FanInEdgeRunner aggregation Python: fix(workflows): preserve all trace contexts in FanInEdgeRunner aggregation Aug 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes distributed tracing propagation in the Python workflow engine’s fan-in edge runner by ensuring nested fan-in aggregations preserve all upstream trace contexts and source span IDs, rather than silently dropping everything except the first element per message.

Changes:

  • Update FanInEdgeRunner aggregation to iterate over WorkflowMessage.trace_contexts / source_span_ids and aggregate all entries.
  • Add a regression test that simulates nested fan-in behavior by sending a message containing multiple trace contexts and asserting the target receives the full aggregated set.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
python/packages/core/agent_framework/_workflows/_edge_runner.py Adjusts fan-in aggregation to preserve multiple trace contexts/source span IDs per incoming message.
python/packages/core/tests/workflow/test_edge.py Adds a regression test to ensure fan-in preserves multiple trace contexts per message.

Comment on lines +362 to +368
trace_contexts: list[dict[str, str]] = []
source_span_ids: list[str] = []
for msg in messages_to_send:
if msg.trace_contexts:
trace_contexts.extend(msg.trace_contexts)
if msg.source_span_ids:
source_span_ids.extend(msg.source_span_ids)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6fdc8b5. Trace contexts and source span IDs are now paired per-message via zip(msg_contexts, msg_span_ids, strict=False) before flattening, so a message with mismatched counts only drops its own orphans instead of shifting all subsequent pairs out of alignment.

Comment on lines +1203 to +1226
@handler
async def mock_aggregator_handler(self, message: list[MockMessage], ctx: WorkflowContext) -> None:
self.call_count += 1

async def execute(
self,
message: WorkflowMessage,
source_executor_ids: list[str],
state: Any,
ctx: Any,
*,
trace_contexts: list[dict[str, str]] | None = None,
source_span_ids: list[str] | None = None,
) -> None:
self.captured_trace_contexts = trace_contexts
self.captured_source_span_ids = source_span_ids
await super().execute(
message,
source_executor_ids,
state,
ctx,
trace_contexts=trace_contexts,
source_span_ids=source_span_ids,
)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6fdc8b5. Removed the execute() override entirely. The handler now captures trace data from ctx._trace_contexts and ctx._source_span_ids on the WorkflowContext passed to it, avoiding the fragile override of Executor.execute().

@weed33834

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

…ation

FanInEdgeRunner collected trace contexts and source span IDs using the
singular backward-compat properties (msg.trace_context / msg.source_span_id),
which return only the first element of the plural lists. When a message
arriving at a fan-in already carries multiple trace contexts (e.g. from
a prior fan-in aggregation), all but the first were silently dropped.

Iterate over the plural fields (trace_contexts / source_span_ids) and
extend the aggregated lists so every trace context and source span ID
from every source message is preserved. This keeps distributed tracing
links intact for nested fan-in topologies.

Added test_fan_in_preserves_multiple_trace_contexts_per_message that
sends a message with two trace contexts through a fan-in and asserts
all three contexts (2 + 1) reach the target executor.
1. Pair trace_contexts and source_span_ids per-message (via zip) instead
   of flattening independently. This prevents misalignment when a message
   has mismatched counts — orphans are dropped per-message rather than
   shifting all subsequent pairs out of alignment.

2. Remove TraceCapturingAggregator's override of Executor.execute()
   (documented as "do not override"). Capture trace data from the
   WorkflowContext passed to the handler instead.
@weed33834
badhope (weed33834) force-pushed the fix/fanin-trace-context-aggregation branch from 58fd192 to 6fdc8b5 Compare August 8, 2026 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants