Too much happening on GUI thread

[JoshuaRowePhantom]

Summary

During each streaming LLM run, StreamingPersistenceMiddleware.GetStreamingResponseAsync performs an unbounded IList<ChatResponseUpdate>.ToChatResponse() coalescing pass on every yielded update (line 64), and then does persistence-store I/O on the same call stack. In GUI hosts this call stack runs on the Avalonia dispatcher thread: the profiler stack shows

System.GC.AllocateUninitializedArray<char>            (managed→native)
Microsoft.Extensions.AI ...MergeText / CoalesceContent / FinalizeResponse / ToChatResponse
Phantom.Workspaces.Llm.StreamingPersistenceMiddleware.GetStreamingResponseAsync  line 64
Microsoft.Agents.AI.ChatClientAgent.RunCoreStreamingAsync
Phantom.Workspaces.Llm.AgentChat.RunProcessLoopAsync                             line 1770
Avalonia.Threading.SendOrPostCallbackDispatcherOperation.InvokeCore
Avalonia Dispatcher.MainLoop

The heavy per-update work (a fresh char[] allocation and full text concatenation of the entire accumulated assistant message, plus a persistence-store round-trip) blocks input, rendering and animation on the UI thread for the duration of every LLM stream. The MEAI coalescing and persistence I/O are pure compute / I/O — they must run on a worker thread. Only the view-model mutations that Avalonia binds to (running-item collections, chat history) legitimately need the foreground context.

Root Cause

There are two collaborating causes; both must be understood before fixing.

Cause 1 — the process loop is bound to the foreground (UI) scheduler by design

AgentChat.StartProcessingLoop explicitly schedules RunProcessLoopAsync on this.foregroundScheduler, which in the GUI is a SynchronizationContextTaskScheduler over the Avalonia UI SynchronizationContext:

features/Phantom.Workspaces.Llm.Core/AgentChat.cs:2233-2262

private void StartProcessingLoop()
{
    ...
    // Run the process loop on the same foreground scheduler used for running-item mutations.
    // In production this is the captured UI synchronization context ...
    this.processTask = Task.Factory.StartNew(
        () => this.acceptsUserInput
            ? this.RunProcessLoopAsync(this.cts.Token)
            : this.RunHostedProcessLoopAsync(this.cts.Token),
        this.cts.Token,
        TaskCreationOptions.DenyChildAttach,
        this.foregroundScheduler).Unwrap();
}

foregroundScheduler is captured in the constructor from the current SynchronizationContext when no explicit scheduler is supplied:

features/Phantom.Workspaces.Llm.Core/AgentChat.cs:139-142

this.foregroundScheduler = request.ForegroundScheduler
    ?? (SynchronizationContext.Current is not null
        ? TaskScheduler.FromCurrentSynchronizationContext()
        : this.foregroundSchedulerPair.ExclusiveScheduler);

This binding is intentional: the loop mutates non-thread-safe, UI-observed collections (RunningItems, History, etc.) via CreateRunningItem / UpdateRunningItem / CompleteRunningItem, and those mutations must happen on the foreground context.

Cause 2 — the streaming enumeration runs synchronously on the loop's thread

RunProcessLoopAsync pumps the provider enumerator directly and does not offload it:

features/Phantom.Workspaces.Llm.Core/AgentChat.cs:1759-1785

providerEnumerator = this.StartRun(
        chatMessagesToSubmit.ToArray(),
        currentSession,
        runCancellation.Token)
    .GetAsyncEnumerator(runCancellation.Token);

while (true)
{
    pendingMoveNext = providerEnumerator.MoveNextAsync().AsTask();          // line 1770
    if (await WasCanceledBeforeCompletingAsync(pendingMoveNext, runCancellation.Token))
        throw new OperationCanceledException(runCancellation.Token);

    var hasNext = await pendingMoveNext;                                    // no ConfigureAwait(false)
    ...
    partialResponses.Notify(providerEnumerator.Current);
}

Each MoveNextAsync is invoked from the UI thread (because the loop resumes on the foreground scheduler), and any part of it that runs synchronously — or that resumes synchronously after an already-completed inner await — runs on the UI thread. The middleware body is:

features/Phantom.Workspaces.Llm.Core/StreamingPersistenceMiddleware.cs:48-78

public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
    IEnumerable<ChatMessage> messages,
    ChatOptions? options = null,
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    var buffer = new List<ChatResponseUpdate>();
    var persistedCount = 0;

    await foreach (var update in this.inner.GetStreamingResponseAsync(messages, options, cancellationToken).ConfigureAwait(false))
    {
        buffer.Add(update);

        var response = buffer.ToChatResponse();                              // line 64 — heavy
        var stableCount = update.FinishReason is not null
            ? response.Messages.Count
            : Math.Max(0, response.Messages.Count - 1);

        for (var i = persistedCount; i < stableCount; i++)
        {
            await this.PersistMessageAsync(response.Messages[i]).ConfigureAwait(false);  // I/O
        }

        persistedCount = stableCount;

        yield return update;
    }
}

Note that ConfigureAwait(false) here only affects continuations inside the async iterator — it does not change which thread the consumer's MoveNextAsync runs on. When the inner client returns an already-buffered update (a fast local await, or the very first synchronous fast-path of a streaming SDK), execution stays on the caller's thread all the way through buffer.ToChatResponse() and into PersistMessageAsync. Because the caller resumed on the Avalonia dispatcher (Cause 1), the heavy MEAI coalescing (FinalizeResponse → CoalesceContent → MergeText → AllocateUninitializedArray<char>) and the persistence-store write execute on the UI thread — exactly the stack shown by the profiler.

Additionally, ToChatResponse() is called on every update, not just at end-of-stream. It re-coalesces the entire growing buffer each time, so per-update UI-thread cost grows with response length — a very large final response coalesces its full text on the UI thread on every intermediate update as well.

Affected Files

File	Role
features/Phantom.Workspaces.Llm.Core/StreamingPersistenceMiddleware.cs	Runs ToChatResponse coalescing + persistence I/O on the caller's thread; must offload to a worker.
features/Phantom.Workspaces.Llm.Core/AgentChat.cs (lines ~1690–1830, and StartProcessingLoop at ~2233–2262)	Schedules RunProcessLoopAsync on the foreground scheduler and pumps the provider enumerator directly on that thread. Fix must ensure the streaming enumeration + coalescing does not run on the foreground context, while UI-observable mutations (CreateRunningItem / UpdateRunningItem / CompleteRunningItem, partialResponses.Notify) are still marshalled back to it.
features/Phantom.Workspaces.Llm.Core/PartialResponseConflator.cs (indirectly)	Receives per-update notifications; already marshals VM mutations, so it is the natural boundary between "background streaming" and "foreground VM updates".

Design / Fix

Guiding principle (per the owner's comment: "the StreamingPersistenceMiddleware needs to be mindful of foreground / background operations"): keep binding-affecting mutations on the foreground context; move pure compute and I/O off it.

Primary fix — offload the coalescing + persistence work inside StreamingPersistenceMiddleware

Wrap the stream-consumption body in a Task.Run so the heavy per-update work runs on the thread pool regardless of who called MoveNextAsync. The idiomatic shape for an IAsyncEnumerable-yielding middleware is a channel + background pump:

public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
    IEnumerable<ChatMessage> messages,
    ChatOptions? options = null,
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    var channel = Channel.CreateUnbounded<ChatResponseUpdate>(
        new UnboundedChannelOptions { SingleReader = true, SingleWriter = true });

    // Run the enumeration, coalescing and persistence on the thread pool, never on the caller's
    // (potentially UI) thread. The producer yields updates through the channel; the caller
    // consumes them on whatever context it prefers.
    var pump = Task.Run(async () =>
    {
        try
        {
            var buffer = new List<ChatResponseUpdate>();
            var persistedCount = 0;

            await foreach (var update in this.inner
                .GetStreamingResponseAsync(messages, options, cancellationToken)
                .ConfigureAwait(false))
            {
                buffer.Add(update);

                var response = buffer.ToChatResponse();               // now on thread pool
                var stableCount = update.FinishReason is not null
                    ? response.Messages.Count
                    : Math.Max(0, response.Messages.Count - 1);

                for (var i = persistedCount; i < stableCount; i++)
                {
                    await this.PersistMessageAsync(response.Messages[i]).ConfigureAwait(false);
                }
                persistedCount = stableCount;

                await channel.Writer.WriteAsync(update, cancellationToken).ConfigureAwait(false);
            }
            channel.Writer.Complete();
        }
        catch (Exception ex)
        {
            channel.Writer.TryComplete(ex);
        }
    }, cancellationToken);

    await foreach (var update in channel.Reader.ReadAllAsync(cancellationToken).ConfigureAwait(false))
    {
        yield return update;
    }

    await pump.ConfigureAwait(false);
}

Ordering constraint preserved: the middleware still writes each stable message before the corresponding update is delivered to the consumer, because the pump only writes to the channel after PersistMessageAsync completes. The consumer still sees updates incrementally (not batched to end-of-stream); the coalescing/persistence cost is simply no longer on the UI thread.

Alternative — offload once at the process-loop level

If a wider fix is preferred, AgentChat.StartProcessingLoop can launch the enumeration portion of RunProcessLoopAsync on the thread pool while keeping the running-item mutations marshalled to the foreground scheduler (e.g. via RunOnForegroundAsync, already used elsewhere in this file — see comment at lines 388–406). The primary fix above is preferred because it is local to the middleware, cannot regress the existing foreground-context invariants (issues #909 / #1068), and directly addresses the owner's note.

What must remain on the foreground context

• CreateRunningItem / UpdateRunningItem / CompleteRunningItem and any runningItem.Items mutation — Avalonia binds to these collections.
• History.Add / AppendUserMessagesToHistory.
• PartialResponseConflator.Notify (which internally marshals VM edits) — the loop still calls this from the foreground context; only the upstream production of ChatResponseUpdates moves to the pool.

Optional follow-up — avoid per-update full coalescing

buffer.ToChatResponse() is O(total text) per update. Once the heavy work is off the UI thread this is no longer a UX blocker, but a follow-up may want to compute the "message boundary" using role/finish-reason transitions on the buffer directly, calling ToChatResponse() only when a boundary or end-of-stream is observed. Out of scope for this fix.

Expected Tests

Add to features/Phantom.Workspaces.Llm.Core.Tests/StreamingPersistenceMiddlewareTests.cs:

Test	Scenario / expectation
StreamingPersistenceMiddleware_CoalescingRunsOffCapturedSynchronizationContext	Install a tracking SynchronizationContext on the calling thread, invoke GetStreamingResponseAsync from that thread, feed several ChatResponseUpdates through a fake inner client, and assert that every observed SynchronizationContext.Current sampled inside the ChatResponseExtensions.ToChatResponse path (via an AIContent/inner-client hook that records SynchronizationContext.Current at coalesce time) is either null or the default (thread-pool) context — never the tracking context.
StreamingPersistenceMiddleware_PersistMessageAsyncRunsOffCapturedSynchronizationContext	Same tracking-context harness; assert IAgentPersistenceStore.StoreAsync observes SynchronizationContext.Current off the tracking context on every stable-message write.
StreamingPersistenceMiddleware_YieldOrdering_StableMessagePersistedBeforeNextUpdateYielded	Existing invariant (already covered by TwoUpdates_FirstPersistedBeforeSecondYielded) must continue to hold after offloading — verify explicitly that when message N becomes stable, StoreAsync(N) completes before MoveNextAsync returns update N+1 to the consumer.
StreamingPersistenceMiddleware_CancellationTokenPropagates_ToInnerEnumerationAndPump	When the caller cancels, the offloaded pump observes cancellation and the channel completes with OperationCanceledException (or equivalent), matching the current behaviour.
AgentChat_StreamingRun_ForegroundContextObservesOnlyVmMutations	Integration-style: with a foreground SynchronizationContextTaskScheduler installed, run a scripted streaming response through AgentChat; assert that RunningItems/History observable mutations run on the foreground context (existing invariant), while a hook placed on the inner chat client records that ToChatResponse/StoreAsync did not run on it.

Naming follows the Subject_Scenario_ExpectedOutcome PascalCase convention already used in StreamingPersistenceMiddlewareTests (e.g. SingleUpdate_FinishReason_PersistedOnStreamEnd, TwoUpdates_FirstPersistedBeforeSecondYielded).

[JoshuaRowePhantom]

Probably the StreamingPersistenceMiddleware needs to be mindful of foreground / background operations.

[JoshuaRowePhantom]

Do not offload at the Process loop level; we've intentionally made it run on the foreground thread.

[JoshuaRowePhantom]

Design

Root cause

StreamingPersistenceMiddleware.GetStreamingResponseAsync performed two non-trivial pieces of work synchronously on the caller's thread for every streamed update:

1. buffer.ToChatResponse() — a full coalesce of the growing update buffer (text concatenation + a fresh allocation over the whole accumulated response so far), and
2. PersistMessageAsync(...) — a store write (IAgentPersistenceStore.StoreAsync) for each newly-stabilised message.

In the GUI host the agent process loop resumes on the Avalonia foreground scheduler by design (per the issue owner's note: "we've intentionally made it run on the foreground thread"). That means this coalescing + persistence ran on the UI SynchronizationContext, blocking input, layout and animation for the duration of the stream — worse as the response grows because ToChatResponse() is O(n) per update.

Fix

Offload only the pure compute / I/O off the caller's context, without changing the streaming cadence:

• Acquire the inner enumerator explicitly and drive it inside a per-step await Task.Run(...). Each Task.Run does: MoveNextAsync, buffer append, ToChatResponse() coalesce, and persistence of the now-stable messages — all on a thread-pool thread where SynchronizationContext.Current is null.
• The loop stays strictly one-update-per-consumer-pull (not an eager background pump), so:
  • the incremental-persistence cadence is unchanged,
  • the ordering invariant is preserved — each stable message is persisted before its triggering update is yield returned to the consumer,
  • back-pressure is identical to the previous synchronous version.
• enumerator.DisposeAsync() is guarded by try/finally; cancellationToken flows into Task.Run, MoveNextAsync, and the inner enumeration.

Note on the owner's guidance "Do not offload at the Process loop level": this fix is middleware-local — it does not move the process loop off the foreground thread. The consumer's MoveNextAsync continuation still resumes on the caller's preferred context (the foreground scheduler in the GUI host), so the downstream binding-affecting view-model mutations remain on the foreground context. Only the middleware's own coalescing + store I/O moves to the thread pool.

Why not the unbounded-channel eager pump

An unbounded Channel + background pump would run ahead of the consumer, breaking the existing lock-step timing guarantees (e.g. CancellationOfConsumer_DoesNotAbortInFlightPersistence, TwoUpdates_FirstPersistedBeforeSecondYielded). The per-step Task.Run achieves the "don't block the UI thread" goal while keeping those invariants intact.

Tests

Phantom.Workspaces.Llm.Core.Tests/StreamingPersistenceMiddlewareTests.cs:

• ..._CoalescingRunsOffCapturedSynchronizationContext — drives the middleware from a single-thread pump SynchronizationContext and asserts the inner enumeration never observes the pump context.
• ..._PersistMessageAsyncRunsOffCapturedSynchronizationContext — asserts StoreAsync never runs on the captured pump context.
• ..._YieldOrdering_StableMessagePersistedBeforeNextUpdateYielded — asserts the stable-before-yield ordering invariant is preserved.
• ..._CancellationTokenPropagates_ToInnerEnumerationAndPump — asserts the cancellation token flows into the inner enumeration.

All existing StreamingPersistenceMiddleware tests continue to pass (17/17).

[JoshuaRowePhantom]

Implementation notes

Commit SHA: a2d17a8ed001ad116bedff2442aa836135c113e3

Files changed

• Phantom.Workspaces.Llm.Core/StreamingPersistenceMiddleware.cs — reworked GetStreamingResponseAsync to drive the inner enumerator inside a per-step Task.Run, offloading the ToChatResponse() coalescing and per-message persistence onto the thread pool while keeping strict one-update-per-consumer-pull streaming.
• Phantom.Workspaces.Llm.Core.Tests/StreamingPersistenceMiddlewareTests.cs — added 4 tests + supporting helpers verifying: coalescing runs off the captured SynchronizationContext, StoreAsync runs off it, the stable-before-yield ordering invariant holds, and the cancellation token propagates into the inner enumeration.

Deviations from the issue

• The issue suggested an unbounded Channel + a background Task.Run pump. That approach pumps ahead of the consumer and breaks the existing lock-step timing guarantees (CancellationOfConsumer_DoesNotAbortInFlightPersistence, TwoUpdates_FirstPersistedBeforeSecondYielded). Instead the fix uses a per-step Task.Run, which achieves the "don't block the UI thread" goal while preserving those invariants.
• Honours the owner's guidance "Do not offload at the Process loop level": the fix is middleware-local. The consumer's MoveNextAsync continuation still resumes on the caller's preferred (foreground) context, so downstream view-model mutations stay on the foreground thread.

Scope changes

None beyond the middleware and its test file.

Verification

• dotnet build --no-incremental: 0 errors.
• StreamingPersistenceMiddlewareTests: 17/17 passed (4 new + 13 existing).

[JoshuaRowePhantom]

Verification result

Status: ✅ Verified

Scope: This verification covers only #1327 (commit a2d17a8e "Fix #1327: offload streaming persistence coalescing off the UI thread"). The branch batch/1325-and-3-more also carries #1329, #1325, #1328, which are verified separately.

Commit checked: a2d17a8e — Phantom.Workspaces.Llm.Core/StreamingPersistenceMiddleware.cs (+ tests).

Feature coverage

• GetStreamingResponseAsync now drives the inner enumerator inside a per-step Task.Run, so buffer.ToChatResponse() coalescing and PersistMessageAsync store I/O run on the thread pool (SynchronizationContext.Current == null), never on the caller's (UI) context. ✔
• Honours the owner's explicit guidance in the issue comments — "Do not offload at the Process loop level": the fix is middleware-local; the consumer's MoveNextAsync continuation still resumes on the caller's foreground context, keeping binding-affecting VM mutations on the UI thread. ✔
• Streaming cadence preserved: strictly one-update-per-consumer-pull (no eager background/channel pump), so the incremental-persistence order invariant (stable message persisted before its triggering update is yielded) and back-pressure are unchanged. ✔

Thread-safety

• buffer and persistedCount are captured and mutated only inside the sequential per-step Task.Run; each step is await-ed before the next begins, establishing a happens-before relationship — no concurrent access. ✔
• enumerator.DisposeAsync() guarded by try/finally; cancellationToken flows into Task.Run, MoveNextAsync and the inner enumeration. ✔

Code-quality criteria

• No [Skip]/quarantined tests, no unresolved TODOs. ✔
• No timing-dependent tests: new tests synchronise via TaskCompletionSource / CancellationToken.Register / a gate — no Task.Delay/Thread.Sleep/fixed timeouts/polling. ✔
• No fixed dispatcher-pass pumping: uses event-driven SingleThreadPump.PostAsync, not repeated RunAsync(DispatcherPriority.Background, …). ✔
• No synchronous task blocking (.Result/.Wait()/.GetAwaiter().GetResult()) in source or tests. ✔

Tests added (4)

• ..._CoalescingRunsOffCapturedSynchronizationContext — asserts inner enumeration/coalescing never observes the captured UI-like context.
• ..._PersistMessageAsyncRunsOffCapturedSynchronizationContext — asserts StoreAsync runs off the captured context.
• ..._YieldOrdering_StableMessagePersistedBeforeNextUpdateYielded — asserts the stable-before-yield ordering invariant survives offloading.
• ..._CancellationTokenPropagates_ToInnerEnumerationAndPump — asserts cancellation flows into the inner enumeration and surfaces OperationCanceledException.

Note on the Expected-Tests table

The issue listed a 5th, integration-level test (AgentChat_StreamingRun_ForegroundContextObservesOnlyVmMutations). It was not added, but its behaviour is fully covered elsewhere: the "off-context coalescing/persistence" half is directly (and more strongly) proven by the two new middleware SynchronizationContext tests using a UI-like SingleThreadPump, and the "foreground observes VM mutations" half is comprehensively covered by the existing AgentChatForegroundContextTests suite (RunningItems_AreNeverMutatedOffForegroundScheduler_*, InitializeAsync_*_OnForegroundScheduler, etc.). No described behaviour is left untested, so this is not counted as an outstanding gap.

Outstanding gaps

None — all criteria satisfied.

Tests

Ran .\scripts\run-tests.ps1 -TestNames StreamingPersistenceMiddlewareTests → 17/17 passed (4 new + 13 existing), Failed: 0.

The #1327 diff touches only the LLM middleware + its test file (no filesystem/Git repository layers), so needs-slow-tests is not required for this issue.

=== Test Run Summary ===
Executed : 17
Passed   : 17
Failed   : 0

[JoshuaRowePhantom]

Checked in

Commit: 61f7f48
Branch fast-forwarded: features → 61f7f48