LLM inference callback support

[SteveSandersonMS]

Summary

This PR adds SDK support for intercepting LLM inference requests and handling them in user code across both Node.js and .NET.

It includes the full feature work on this branch:

• wire up LLM inference provider registration and generated RPC types
• add the raw chunked callback protocol for outbound inference requests and responses
• cover plain HTTP, streaming responses, cancellation, error mapping, session-id threading, and WebSocket transport
• port the feature to the .NET SDK
• simplify the public callback surface to a single LlmRequestHandler model with forwarding helpers

What changed

Shared protocol and plumbing

• add generated RPC / session event types needed for LLM inference callbacks across the supported SDK surfaces
• add SDK-side registration for a process-global LLM inference provider
• route outbound inference requests through the callback bridge instead of requiring provider-specific hooks

Node.js SDK

• add low-level inference callback support plus the callback provider bridge
• add LlmRequestHandler and related types as the main public interception surface
• support HTTP, streaming, and WebSocket inference flows through the same handler model
• thread cancellation and session id through the callback context
• update exports and tests to reflect the simplified handler API

.NET SDK

• port the LLM inference callback stack to .NET
• add LlmRequestHandler, CopilotWebSocketHandler, and forwarding helpers for the common mutate-and-forward case
• support HTTP and WebSocket inference interception with the same callback model
• thread cancellation and session id through the request context

API shape

• collapse HTTP interception to a single send hook (SendRequestAsync / sendRequest)
• expose WebSocket interception through OpenWebSocketAsync / openWebSocket, returning a per-connection handler object
• allow consumers to mutate, drop, duplicate, or fully replace request/response messages while keeping the common forwarding case straightforward

Usage examples

C#

using GitHub.Copilot;
using System.Net.Http;

sealed class MyHandler : LlmRequestHandler
{
    protected override async Task<HttpResponseMessage> SendRequestAsync(
        HttpRequestMessage request,
        LlmRequestContext ctx)
    {
        request.Headers.Add("X-Debug-Session", ctx.SessionId ?? "none");
        return await base.SendRequestAsync(request, ctx);
    }

    protected override Task<CopilotWebSocketHandler> OpenWebSocketAsync(LlmRequestContext ctx)
        => Task.FromResult<CopilotWebSocketHandler>(new MyForwardingSocket(ctx));
}

sealed class MyForwardingSocket : ForwardingWebSocketHandler
{
    public MyForwardingSocket(LlmRequestContext ctx)
        : base(ctx)
    {
    }

    public override Task SendRequestMessageAsync(LlmWebSocketMessage message)
    {
        var text = message.GetText().Replace("model-A", "model-B");
        return base.SendRequestMessageAsync(LlmWebSocketMessage.Text(text));
    }
}

Node.js

import {
    ForwardingWebSocketHandler,
    LlmRequestContext,
    LlmRequestHandler,
} from "@github/copilot";

class MyHandler extends LlmRequestHandler {
    protected override async sendRequest(request: Request, ctx: LlmRequestContext): Promise<Response> {
        const headers = new Headers(request.headers);
        headers.set("x-debug-session", ctx.sessionId ?? "none");

        return super.sendRequest(new Request(request, { headers }), ctx);
    }

    protected override async openWebSocket(ctx: LlmRequestContext) {
        return new MyForwardingSocket(ctx);
    }
}

class MyForwardingSocket extends ForwardingWebSocketHandler {
    override sendRequestMessage(data: string | Uint8Array) {
        if (typeof data === "string") {
            return super.sendRequestMessage(data.replace("model-A", "model-B"));
        }
        return super.sendRequestMessage(data);
    }
}

Tests

This branch adds and updates coverage for:

• callback provider registration
• HTTP inference interception
• streaming inference interception
• error mapping
• consumer-initiated cancellation
• session-id threading
• WebSocket callback handling
• .NET handler and adapter behavior
• Node.js unit and e2e callback flows

Resolves github/copilot-sdk-internal#88

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[github-actions]

Cross-SDK Consistency Review

This PR adds LLM inference callback support to Go and .NET, bringing the feature to 4 of the 6 SDK implementations (Node.js and Python already had it). Overall the API shapes are well-aligned across languages. A few observations:

✅ Well-aligned across the four covered SDKs

Concept	Node.js	Python	Go	.NET
Entry class	LlmRequestHandler	LlmRequestHandler	LlmRequestHandler	LlmRequestHandler
HTTP hook	sendRequest(req, ctx)	send_request(req, ctx)	Transport field (http.RoundTripper)	SendRequestAsync(req, ctx)
WS hook	openWebSocket(ctx)	open_web_socket(ctx)	OpenWebSocket field (func)	OpenWebSocketAsync(ctx)
Context	LlmRequestContext	LlmRequestContext	LlmRequestContext	LlmRequestContext
WS handler	CopilotWebSocketHandler	CopilotWebSocketHandler	CopilotWebSocketHandler	CopilotWebSocketHandler
Forwarding WS	ForwardingWebSocketHandler	ForwardingWebSocketHandler	ForwardingWebSocketHandler	ForwardingWebSocketHandler
Config	LlmInferenceConfig { handler }	LlmInferenceConfig { handler }	LlmInferenceConfig { Handler }	LlmInferenceConfig { Handler }
Low-level interface	LlmInferenceProvider	LlmInferenceProvider	LlmInferenceProvider	ILlmInferenceProvider

Naming and structure follow each language's conventions (camelCase/snake_case/PascalCase, AbortSignal/asyncio.Event/context.Context/CancellationToken for cancellation, etc.).

⚠️ Remaining feature gap: Java and Rust SDKs

LLM inference callback support is not yet implemented in the Java and Rust SDKs. These two SDKs currently have no LlmInferenceProvider, LlmRequestHandler, or related types. If Java and Rust parity is intended, they'd need equivalent implementations. Is there a tracking issue or follow-up planned for those two?

i️ Language-idiomatic design differences (not issues)

These are expected given the languages involved:

• Go's LlmRequestHandler uses struct fields for customization (Transport http.RoundTripper, OpenWebSocket func(...)) instead of method overrides. This is the idiomatic Go approach with no class inheritance.
• Go's ForwardingWebSocketHandler adds OnSendRequestMessage and OnSendResponseMessage function fields for inline message transformation, avoiding the need for a full subtype. The other SDKs accomplish the same thing via subclassing.
• .NET uses LlmWebSocketMessage (a value-type wrapper with IsBinary/GetText()) where Node.js and Python use string | Uint8Array / str | bytes union types. Both representations carry the same semantic content.

Generated by SDK Consistency Review Agent for issue #1689 · sonnet46 2M · ◷