[Feature]: Add first-class async conversation runtime for SendMessage, streaming, queueing, and interruption

[soul-soft]

Here is the English translation of your proposal, with additional details on Web usage scenarios.

---

Summary

I would like to propose that MAF provides a built‑in asynchronous message distribution model for agent conversations.

Currently, when building real interactive applications on top of AIAgent.RunStreamingAsync, applications typically need to design their own runtime layer to handle message sending, stream subscription, cancellation, and session‑level execution ordering.

Without such a layer, it is difficult to build production‑ready chat interactions because:

• HTTP/API requests should return immediately after receiving the user’s message;
• The agent execution continues in the background;
• Streaming updates are pushed to subscribers;
• Messages in the same session are processed sequentially;
• Different sessions can run concurrently;
• The current run can be interrupted by session ID;
• Errors and cancellations are presented through the stream, rather than breaking the sending request.

Motivation

In real client applications, “SendMessage” should not generally mean “run the entire model request and wait for it to complete”.

A more practical model is:

Client sends a message
  -> Server receives the message
  -> Message is enqueued by session/thread ID
  -> API returns Accepted immediately
  -> Runtime consumes the session queue
  -> Agent.RunStreamingAsync produces updates
  -> Updates are published to subscribers

This decouples message input from stream output.

Without a built‑in runtime like this, every application has to implement its own event bus, per‑session queues, background executors, cancellation token management, and subscription protocols. This is complex and error‑prone, especially when dealing with long‑running tool calls, client timeouts, user interruptions, and stream client reconnections.

Proposed API Shape

One possible abstraction is:

public interface IAgentRunner
{
    Task SendMessageAsync(
        AIAgent agent,
        AgentRunRequest request,
        CancellationToken cancellationToken = default);

    bool Interrupt(string conversationId);
}

And an event bus abstraction:

public interface IAgentEventBus
{
    ValueTask<bool> PublishAsync(
        string conversationId,
        AgentEvent @event,
        CancellationToken cancellationToken = default);

    IAsyncEnumerable<AgentEvent> SubscribeAsync(
        string conversationId,
        CancellationToken cancellationToken = default);
}

The runtime behaviour would be:

SendMessageAsync
-> Look up the session queue
-> If the session is running: enqueue the message
-> If the session is idle: start consuming the queue
-> Return immediately

Consuming the session queue
-> Call agent.RunStreamingAsync
-> Publish message/update events
-> Publish completion/error/cancellation events
-> Continue with the next message in the queue

Expected Behaviour

• Messages in the same session/thread are executed sequentially.
• Different sessions/threads can run concurrently.
• SendMessageAsync only waits for the message to be accepted, not for the agent execution to finish.
• Streaming outputs are consumed via the subscription API.
• Cancellation/interruption can be performed by session/thread ID.
• Runtime errors are emitted as events and logged by the runtime.
• Application code does not have to manually manage background tasks per request.

Why This Matters

This makes MAF much easier to use in real‑world interactive applications.

Most chat UIs require immediate feedback: after the user sends a message, the assistant avatar appears instantly, streaming begins, and the send request should not be tied to the entire lifetime of model/tool execution. Runtime issues such as connection re‑establishment and event delivery are better handled by the framework itself.

Related Design

This is similar to a thread‑based signalling/message runtime:

• Messages are routed by session/thread ID;
• A running execution can receive subsequent messages or queue them;
• An idle thread is woken up by a new message;
• Stream updates are delivered through a separate subscription channel.

This provides a cleaner foundation for production‑grade chat, desktop clients, web clients, and long‑running agent workflows.

---

Additional Web Usage Scenarios

In modern web applications, the proposed runtime addresses several critical challenges:

• HTTP request lifecycle – A typical web API endpoint can accept the user’s message, enqueue it, and return a 202 Accepted response immediately. This avoids keeping HTTP connections open for the entire agent execution, reducing server resource pressure and preventing gateway timeouts.

• Real‑time streaming – The subscription API (SubscribeAsync) can be exposed over Server‑Sent Events (SSE) or WebSockets. The client subscribes to a conversation ID and receives a continuous flow of agent events (text chunks, tool calls, status updates, errors, completion). This enables a responsive typing‑indicator experience without polling.

• Client reconnection – If the client loses connection (e.g., network flakiness), it can simply re‑subscribe to the same conversation ID and resume receiving events from the point of interruption, because the runtime keeps a buffer or replay capability. The framework can even allow resuming with a last‑seen event ID.

• Concurrent sessions – A single web server can handle hundreds of concurrent conversations, each with its own queue and background execution. Scaling is simplified because the runtime manages per‑session state and execution independently.

• User‑initiated cancellation – In a web UI, a user might click a “Stop generating” button. The frontend calls an interrupt endpoint with the conversation ID, and the runtime cancels the current agent run and publishes a cancellation event. The UI can immediately reflect the stop, and the backend does not leave orphaned tasks.

• Long‑running tool calls – When an agent invokes an external tool that takes seconds or minutes, the stream can still emit progress events. The subscription channel keeps the client informed, and the initial HTTP request is long gone, avoiding any timeout issues.

• Horizontal scaling – With a distributed event bus (e.g., Redis Streams or Kafka), the runtime can be scaled across multiple server instances. The session queue and event publication can be made distributed, so that any instance can process messages for any conversation, and subscribers can be routed accordingly.

• Browser tab management – A user might have multiple browser tabs open for the same conversation. Each tab can subscribe independently to the same conversation ID and receive the same stream of events, ensuring consistent state across tabs without extra server load.

By providing this runtime out‑of‑the‑box, MAF would eliminate the need for web developers to reinvent a robust, fault‑tolerant message processing pipeline, and would make building interactive AI‑powered web applications significantly faster and more reliable.