useChat doesn't automatically invoke client tools?

[puehringer]

From the docs I got the impression that any client tool should automatically invoke when using useChat, but I can't seem to get it working. I have an existing setup with pydantic-ai and their ag-ui adapter, but for this simple example I'm using an inline stream taken from https://dojo.ag-ui.com/pydantic-ai/feature/agentic_chat. I would expect that the changeBackground tool is called, but it isn't. At the end, it's simply stuck at input-complete and nothing else happens.

{"type":"text","content":"Change the background to something new."}
{"type":"tool-call","id":"call_kbEElxRRyz24VjEa5WU9LxZp","name":"change_background","arguments":"{\"background\":\"linear-gradient(to right, #ff7e5f, #feb47b)\"}","state":"input-complete"}

Anything I'm missing here?

Demo: https://codesandbox.io/p/sandbox/99p776

import * as React from 'react';

import { type AGUIEvent, toolDefinition } from '@tanstack/ai';
import { clientTools, stream } from '@tanstack/ai-client';
import { useChat } from '@tanstack/ai-react';
import { z } from 'zod';

export const changeBackgroundDef = toolDefinition({
  name: 'change_background',
  description:
    'Change the background color of the chat. Can be anything that the CSS background attribute accepts. Regular colors, linear of radial gradients etc.',
  inputSchema: z.object({
    background: z.string().meta({ description: 'The background. Prefer gradients. Only use when asked.' }),
  }),
});

const changeBackground = changeBackgroundDef.client((input) => {
  console.log('Changing background to:', input.background);
  return `Background changed to ${input.background}`;
});

const tools = clientTools(changeBackground);

export function AIDemo() {
  const chat = useChat({
    connection: stream(async function* mockStream(messages, data, abortSignal) {
      // This stream is directly taken from https://dojo.ag-ui.com/pydantic-ai/feature/agentic_chat
      const rawData = `
data: {"type":"RUN_STARTED","threadId":"fc542f6b-092a-4987-a62e-2a4a7184b459","runId":"752486f4-900d-468a-bf73-f1b2b97fd101","input":{"threadId":"fc542f6b-092a-4987-a62e-2a4a7184b459","runId":"752486f4-900d-468a-bf73-f1b2b97fd101","state":{},"messages":[],"tools":[{"name":"change_background","description":"Change the background color of the chat. Can be anything that the CSS background attribute accepts. Regular colors, linear of radial gradients etc.","parameters":{"type":"object","properties":{"background":{"type":"string","description":"The background. Prefer gradients. Only use when asked."}},"required":["background"]}}],"context":[{"description":"Name of the user","value":"Bob"}],"forwardedProps":{}}}
data: {"type":"TOOL_CALL_START","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","toolCallName":"change_background","parentMessageId":"ff42e05e-e1b0-4e4f-90a4-ecc667458260"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"{\\\""}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"background"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"\\\":\\\""}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"linear"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"-gradient"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"(to"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":" right"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":","}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":" #"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"ff"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"7"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"e"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"5"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"f"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":","}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":" #"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"feb"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"47"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"b"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":")"}
data: {"type":"TOOL_CALL_ARGS","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp","delta":"\\\"}"}
data: {"type":"TOOL_CALL_END","toolCallId":"call_kbEElxRRyz24VjEa5WU9LxZp"}
data: {"type":"RUN_FINISHED","threadId":"fc542f6b-092a-4987-a62e-2a4a7184b459","runId":"752486f4-900d-468a-bf73-f1b2b97fd101"}
`;
      for (const line of rawData.split('\n')) {
        if (line.startsWith('data: ')) {
          const json = line.slice('data: '.length);
          const event: AGUIEvent = JSON.parse(json);
          yield event;
        }
      }
    }),
    tools,
    threadId: 'fc542f6b-092a-4987-a62e-2a4a7184b459',
    id: '752486f4-900d-468a-bf73-f1b2b97fd101',
    onCustomEvent: (eventType, data, context) => {
      console.log('Custom event:', eventType, data, context);
    },
    onError: (e) => {
      console.error('Error in chat:', e);
    },
    onFinish: (final) => {
      console.log('Chat finished:', final);
    },
    onResponse: (response) => {
      console.log('Response:', response);
    },
    onChunk: (chunk) => {
      console.log('Received chunk:', chunk);
    },
  });

  React.useEffect(() => {
    chat.append({
      id: 'e8081a6e-1420-440f-bac9-074d10916aba',
      role: 'user',
      content: 'Change the background to something new.',
    });
  }, []);

  return <div>{chat.messages.map((message) => message.parts.map((part, index) => <div key={index}>{JSON.stringify(part)}</div>))}</div>;
}

[dbridges]

Running into the same, did you ever figure this out?

[puehringer]

@dbridges Yeah I figured this out, although I'm not sure if it's the accepted answer: with a custom backend, useChat does not automatically execute client-side tools. You have to do it yourself in onFinish:

const chat = useChat({
  // ...
  onFinish: async (message) => {
    // Find tool calls that the server returned but haven't been executed yet
    const pendingToolCalls = message.parts.flatMap((p) =>
      p.type === 'tool-call' && p.state === 'input-complete' && p.output === undefined ? [p] : []
    );

    await Promise.all(
      pendingToolCalls.map(async (part) => {
        const tool = tools.find((t) => t.name === part.name);
        if (!tool?.execute) return;

        try {
          const args = typeof part.arguments === 'string' ? JSON.parse(part.arguments) : part.arguments;
          const output = await tool.execute(args);
          await chat.addToolResult({ toolCallId: part.id, tool: part.name, output, state: 'output-available' });
        } catch (e) {
          await chat.addToolResult({ toolCallId: part.id, tool: part.name, output: null, state: 'output-error' });
        }
      })
    );
  },
});

Once you call addToolResult, it automatically sends the result back to the server which triggers the next assistant turn, completing the agentic loop. Let me know if it works for you, or if you have a better idea on how to solve this.

[dbridges]

👍 I ended up doing something similar, but surprising this is not handled automatically.