Skip to content

CopilotSdkStreamAdapter default arm leaks [unknown-copilot-sdk-event: ...] placeholders into user-visible chat and persisted history (#1312 follow-up) #1323

Description

@JoshuaRowePhantom

CopilotSdkStreamAdapter default arm leaks [unknown-copilot-sdk-event: ...] placeholders into user-visible chat and persisted history

Summary

The HTML agent-chat view is polluted with literal placeholder strings of the form [unknown-copilot-sdk-event: GitHub.Copilot.<EventTypeName>], interleaved char-by-char with the real assistant text. The observed placeholder types include AssistantStreamingDeltaEvent (appears many times, one per streaming byte-count ping — this is the source of the character-by-character interleaving), AssistantMessageStartEvent, AssistantTurnStartEvent, UserMessageEvent, SystemMessageEvent, PendingMessagesModifiedEvent, SessionToolsUpdatedEvent, SessionSkillsLoadedEvent, SessionCustomAgentsUpdatedEvent, and SessionUsageInfoEvent.

Root cause: the fix for #1312 added a default: arm to CopilotSdkStreamAdapter.TranslateCopilotSdkSessionEvents that emits a ChatRole.Assistant TextContent reading "[unknown-copilot-sdk-event: <RuntimeTypeName>]" for every unmapped SessionEvent subtype. That "surface it so it isn't invisibly lost" choice was principled, but in practice the SDK emits many unmapped events per turn — most notoriously AssistantStreamingDeltaEvent, a per-chunk streaming progress ping — so the default arm now sprays user-visible placeholder text throughout every response and (via StreamingPersistenceMiddleware) persists that noise into the transcript.

Root cause

The default arm (introduced by #1312)

Phantom.Workspaces.Llm.Core/CopilotSdkStreamAdapter.cs lines 260–282:

default:
    // Fix #1312: never silently drop UNKNOWN event kinds.
    var runtimeTypeName = sessionEvent?.GetType().FullName ?? "<null>";
    var runtimeAgentId = sessionEvent?.AgentId;
    logger.LogWarning(
        "Copilot SDK adapter received an unmapped session event of type {EventType} for AgentId {AgentId}; surfacing as a generic informational update.",
        runtimeTypeName,
        string.IsNullOrEmpty(runtimeAgentId) ? "<root>" : runtimeAgentId);
    var unknown = Tag(
        new TextContent($"[{UnknownCopilotSdkEventContentType}: {runtimeTypeName}]"),
        runtimeAgentId);
    unknown.AdditionalProperties![ContentTypePropertyName] = UnknownCopilotSdkEventContentType;
    yield return new ChatResponseUpdate
    {
        Role = ChatRole.Assistant,
        Contents = [unknown],
    };
    break;

This produces a ChatRole.Assistant update carrying a plain TextContent — indistinguishable from real assistant text as far as StreamingPersistenceMiddleware and the HTML view are concerned. UnknownCopilotSdkEventContentType is applied to AdditionalProperties, but nothing on the render / persistence path filters on it, so the string bleeds straight into the user-visible transcript and into history.

Which SDK event types actually fall through

Verified against C:\dev\microsoft\copilot-sdk\dotnet\src\Generated\SessionEvents.cs. All types below are real classes in the GitHub.Copilot namespace that inherit SessionEvent, and none of them appear in the adapter's switch:

SDK event class (file:line) Why it fires
AssistantStreamingDeltaEvent (SessionEvents.cs:643) Streaming progress ping (assistant.streaming_delta) with just TotalResponseSizeBytes. Emitted per-chunk, so appears dozens of times per turn — the source of the char-by-char interleaving.
AssistantMessageStartEvent (SessionEvents.cs:669) Lifecycle: assistant message opening. Should be silent (or start a message).
AssistantTurnStartEvent (SessionEvents.cs:578) Lifecycle: turn opening. Should be silent.
UserMessageEvent (SessionEvents.cs:552) Echo of the user's own message. Should be silent (we already have it) or ignored.
SystemMessageEvent (SessionEvents.cs:956) System-role bookkeeping. Should be silent or routed to ChatRole.System.
PendingMessagesModifiedEvent (SessionEvents.cs:565) Bookkeeping. Silent.
SessionToolsUpdatedEvent (SessionEvents.cs:1334) Session metadata update. Silent.
SessionSkillsLoadedEvent (SessionEvents.cs:1360) Session metadata update. Silent.
SessionCustomAgentsUpdatedEvent (SessionEvents.cs:1373) Session metadata update. Silent.
SessionUsageInfoEvent (SessionEvents.cs:500) Usage info (not the same as the mapped AssistantUsageEvent). Silent, or map to UsageContent.

The AssistantStreamingDeltaEvent vs AssistantMessageDeltaEvent clarification

Both are real, sibling SDK classes:

  • AssistantMessageDeltaEvent — event type assistant.message_delta, payload AssistantMessageDeltaData with a DeltaContent string (SessionEvents.cs:682 and :2643). This is the actual streaming assistant text and the adapter's case AssistantMessageDeltaEvent when !string.IsNullOrEmpty(delta.Data?.DeltaContent) at line 123 already handles it correctly.
  • AssistantStreamingDeltaEvent — event type assistant.streaming_delta, payload AssistantStreamingDeltaData = { TotalResponseSizeBytes } only (SessionEvents.cs:643 and :2520). This is a byte-count progress ping, has no textual content, and is not handled — it falls through to the fix-Copilot SDK adapter silently drops unhandled SDK session events (no default arm; self-invoke bypasses framework tool loop) #1312 default arm and produces one [unknown-copilot-sdk-event: GitHub.Copilot.AssistantStreamingDeltaEvent] placeholder per streaming chunk.

The two names are easily confused, and #1312's own scope explicitly deferred reconciling the case list against the live SDK event set. This is the concrete manifestation of that deferred work.

The self-invoke bypass makes the adapter the single surface

Per #1312: CopilotSdkChatClient implements ISelfInvokingToolChatClient (CopilotSdkChatClient.cs:34), and AgentFactory.WrapWithMiddleware deliberately skips ToolResultSteeringMiddleware / FunctionInvokingChatClient for it. So whatever CopilotSdkStreamAdapter yields goes straight into the transcript. There is no downstream filter that discards UnknownCopilotSdkEventContentType, so the placeholder TextContent reaches both the HTML chat and StreamingPersistenceMiddleware, permanently corrupting history.

Affected files

File Role
Phantom.Workspaces.Llm.Core/CopilotSdkStreamAdapter.cs Primary — houses both the switch case list and the leaky default arm (lines 260–282).
Phantom.Workspaces.Llm.Core/CopilotSdkChatClient.cs Mirror non-streaming switch at line 543; also calls TranslateCopilotSdkSessionEvents at 660.
Phantom.Workspaces.Llm.Core.Tests/CopilotSdkStreamAdapterTests.cs Existing _UnknownEventKind_IsSurfacedNotDropped currently asserts the bad behaviour and must be revised.
docs/design/copilot-sdk-session-events.md Reference doc for which SDK events should be silent/mapped/error; #1312 explicitly deferred reconciling this.

Design / Fix

The fix for #1312 rightly rejected silent-drop; the fix here is to keep observability (logging) while removing the user-visible / persisted placeholder.

  1. Do not emit user-visible or persistable TextContent from the default arm. Change the default arm to log at Debug (or keep Warning once, then downgrade repeats — the streaming delta will spam Warning otherwise) but yield nothing to the ChatResponseUpdate stream. Retain the UnknownCopilotSdkEventContentType marker constant so the design intent is discoverable in the log message, but do not put the marker string into TextContent.

  2. Explicitly no-op the known-benign lifecycle / metadata events, so they don't even reach the default arm and don't spam logs. Add:

    case AssistantStreamingDeltaEvent:              // per-chunk byte-count progress ping
    case AssistantMessageStartEvent:                // lifecycle: message opening
    case AssistantTurnStartEvent:                   // lifecycle: turn opening
    case AssistantTurnEndEvent:                     // lifecycle: turn closing
    case AssistantIdleEvent:                        // lifecycle: assistant idle
    case AssistantMessageEvent:                     // whole-message duplicate of the deltas
    case UserMessageEvent:                          // echo of caller's own input
    case SystemMessageEvent:                        // system-role bookkeeping (map to System role if we want it in history)
    case PendingMessagesModifiedEvent:              // bookkeeping
    case SessionToolsUpdatedEvent:
    case SessionSkillsLoadedEvent:
    case SessionCustomAgentsUpdatedEvent:
    case SessionUsageInfoEvent:                     // consider mapping to UsageContent instead of dropping
        break;

    Names verified against C:\dev\microsoft\copilot-sdk\dotnet\src\Generated\SessionEvents.cs.

  3. Sketch:

    default:
        logger.LogDebug(
            "Copilot SDK adapter received an unmapped session event of type {EventType} for AgentId {AgentId}; no transcript output emitted.",
            sessionEvent?.GetType().FullName ?? "<null>",
            string.IsNullOrEmpty(sessionEvent?.AgentId) ? "<root>" : sessionEvent!.AgentId);
        break;
  4. Mirror in the non-streaming path (CopilotSdkChatClient.GetResponseAsync, switch at line 543): its default arm also currently logs a Warning per unmapped event; downgrade to Debug (or per-type sample) for the same reason.

  5. Update the design doc docs/design/copilot-sdk-session-events.md with the reconciled list of intentionally-consumed-silent lifecycle events and the mapping table above — this is the Copilot SDK adapter silently drops unhandled SDK session events (no default arm; self-invoke bypasses framework tool loop) #1312 (d) follow-up item that was deferred.

Relationship to #1312

This is the follow-up to #1312. #1312 explicitly deferred two things that this bug forces us to finish:

  • (a) Reconciling the adapter's switch against the live SDK event set — needed because AssistantStreamingDeltaEvent, AssistantMessageStartEvent, UserMessageEvent, PendingMessagesModifiedEvent, SessionToolsUpdatedEvent, SessionSkillsLoadedEvent, SessionCustomAgentsUpdatedEvent, SessionUsageInfoEvent, and SystemMessageEvent are all real SDK types that were never added as case arms.
  • (b) Choosing a surfacing mechanism that does not produce user-visible transcript pollution. Copilot SDK adapter silently drops unhandled SDK session events (no default arm; self-invoke bypasses framework tool loop) #1312 chose "emit a diagnostic TextContent"; in practice that puts noisy strings in front of the user and into persisted history. The fix is: log-only in the default arm; explicit silent no-op for known-benign lifecycle/metadata events.

Note on a plausible-but-wrong hypothesis worth recording: it looks at first glance like the adapter matches AssistantMessageDeltaEvent when it should have matched AssistantStreamingDeltaEvent. That is not the case — those are two different SDK classes, AssistantMessageDeltaEvent carries the DeltaContent string and IS the streaming text, and the adapter handles it correctly. The real trouble is that AssistantStreamingDeltaEvent (progress bytes only) is a separate, unmapped, high-frequency event and it is what the default arm is spraying into chat.

Expected tests

Added to Phantom.Workspaces.Llm.Core.Tests/CopilotSdkStreamAdapterTests.cs, matching the existing TranslateCopilotSdkSessionEvents_<Subject>_<Scenario>_<Outcome> naming used throughout that file.

Test Purpose
TranslateCopilotSdkSessionEvents_AssistantStreamingDeltaEvent_ProducesNoVisibleContent Feed one AssistantStreamingDeltaEvent (progress ping) and assert zero non-idle ChatResponseUpdates are yielded (and specifically no TextContent containing unknown-copilot-sdk-event).
TranslateCopilotSdkSessionEvents_AssistantMessageStartEvent_IsConsumedSilently Same but for AssistantMessageStartEvent.
TranslateCopilotSdkSessionEvents_AssistantTurnStartEvent_IsConsumedSilently Same for AssistantTurnStartEvent.
TranslateCopilotSdkSessionEvents_UserMessageEvent_IsConsumedSilently Same for UserMessageEvent.
TranslateCopilotSdkSessionEvents_PendingMessagesModifiedEvent_IsConsumedSilently Same for PendingMessagesModifiedEvent.
TranslateCopilotSdkSessionEvents_SessionToolsUpdatedEvent_IsConsumedSilently Same for SessionToolsUpdatedEvent.
TranslateCopilotSdkSessionEvents_SessionSkillsLoadedEvent_IsConsumedSilently Same for SessionSkillsLoadedEvent.
TranslateCopilotSdkSessionEvents_SessionCustomAgentsUpdatedEvent_IsConsumedSilently Same for SessionCustomAgentsUpdatedEvent.
TranslateCopilotSdkSessionEvents_SessionUsageInfoEvent_IsConsumedSilently Same for SessionUsageInfoEvent.
TranslateCopilotSdkSessionEvents_UnmappedEvent_DoesNotEmitUserVisiblePlaceholderText Feed a truly unrecognised SessionEvent subclass and assert that no ChatResponseUpdate carries a TextContent whose text contains "unknown-copilot-sdk-event".
TranslateCopilotSdkSessionEvents_UnmappedEvent_LogsDebugWithTypeAndAgentId Assert that the runtime type name and (agent id or <root> marker) still show up in a Debug-level log entry — observability preserved.
TranslateCopilotSdkSessionEvents_InterleavedStreamingDeltaAndText_ProducesOnlyText Feed [AssistantStreamingDeltaEvent, AssistantMessageDeltaEvent("Hi"), AssistantStreamingDeltaEvent, AssistantMessageDeltaEvent(" there")] and assert the visible output is exactly "Hi" and " there" — the direct regression test for the screenshot.

Existing test TranslateCopilotSdkSessionEvents_UnknownEventKind_IsSurfacedNotDropped (line 585) currently asserts the bad behaviour and needs to be revised to assert the log-only behaviour instead.

Metadata

Metadata

Labels

bugSomething isn't workingdiagnosedRoot cause identifiednext-upverified-locallyImplementation has been verified locally

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions