diff --git a/complex-agents/xai-patient-intake/README.md b/complex-agents/xai-patient-intake/README.md new file mode 100644 index 00000000..0903946a --- /dev/null +++ b/complex-agents/xai-patient-intake/README.md @@ -0,0 +1,96 @@ +# xAI Patient Intake + +A family-medicine front desk you can call: the agent identifies a caller against a chart, +books and moves appointments, answers practice-policy questions, collects pre-visit +clinical intake, and routes a possible emergency to urgent care — all in one conversation. + +The speech pipeline is xAI end to end. + +| Stage | Model | +| --- | --- | +| Speech to text | `xai/stt-1` | +| Reasoning | `xai/grok-4.3` (`reasoning_effort="none"`) | +| Text to speech | `xai/tts-1` (voice `carina`) | + +> The clinic is an in-memory fake. No real patient data is involved, and nothing here is a +> medical device or a source of medical advice. + +## Repository layout + +| Directory | Description | +| --- | --- | +| `patient-intake-agent` | Python worker: one agent, one conversation, eight typed tools over an in-memory practice. | +| `frontend` | Next.js app that dispatches the worker and renders the conversation, transcript, and tool calls. | + +Each directory has its own README with deeper notes on architecture and customization. + +## Design + +One agent, one conversation, one fixed tool surface — no handoffs, no task framework, no +workflow state machine. The model holds the conversation in its own context and passes +what it has learned to typed tools when it needs to read or change practice state. + +| Tool | Purpose | +| --- | --- | +| `read_practice_information` | Read the complete published practice guide | +| `find_open_times` | Search real slots using typed patient and scheduling facts | +| `book_appointment` | Register a new patient when necessary and book their chosen slot | +| `manage_appointment` | List, cancel, or reschedule an existing appointment | +| `take_message` | Route a refill, results, billing, referral, nurse, or records request | +| `update_insurance` | Save details from a current insurance card | +| `record_previsit_intake` | Save one completed set of pre-visit answers | +| `record_emergency_escalation` | Record a possible emergency and end ordinary work | + +Every tool re-verifies identity from its arguments rather than trusting remembered state, +so a caller can book a visit, report a symptom, and update insurance in any order without +a phase machine deciding what is allowed next. + +Practice policy stays out of the prompt: `patient-intake-agent/src/clinic/practice_info/` +holds the published guide as Markdown, and one argument-free tool returns all of it, +leaving interpretation to the model instead of a category table. + +## Quick start + +1. **Configure LiveKit** + - Create or reuse a LiveKit Cloud project. + - Grab `LIVEKIT_URL`, `LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET`. + - The agent registers as `xai-patient-intake`; the frontend's dispatch allowlist in + `frontend/app/api/agent/connection_details/route.ts` maps to that name. + +2. **Run the agent** + + ```bash + cd complex-agents/xai-patient-intake/patient-intake-agent + uv sync + cp .env.example .env.local # add your three LiveKit values + uv run python src/agent.py console # talk to it in the terminal, no browser needed + uv run python src/agent.py dev # or register the worker for the frontend + ``` + +3. **Run the frontend** + + ```bash + cd ../frontend + pnpm install + cp .env.example .env.local # the same three values + pnpm dev + ``` + + Visit http://localhost:3000/patient-intake and click the card. The route mints a token + with an explicit agent dispatch, so the worker joins the room the visitor just created. + +4. **Try these** + - *"What are your hours?"* → `read_practice_information` + - *"I'd like to book an appointment"* → `find_open_times`, then `book_appointment` + - *"I need a refill on my lisinopril"* → `take_message` + +## Tests + +```bash +cd patient-intake-agent +uv run pytest tests/unit -q +``` + +The unit tests assert the exact eight tools assembled in production and exercise booking, +rescheduling, message routing, intake, and emergency handling against the in-memory clinic +without touching the network. diff --git a/complex-agents/xai-patient-intake/frontend/.gitignore b/complex-agents/xai-patient-intake/frontend/.gitignore new file mode 100644 index 00000000..0e5e49d9 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/.gitignore @@ -0,0 +1,10 @@ +node_modules/ +.next/ +.env +.env.local +next-env.d.ts +*.tsbuildinfo +.vercel +.env* +.DS_Store +.DS_Store diff --git a/complex-agents/xai-patient-intake/frontend/README.md b/complex-agents/xai-patient-intake/frontend/README.md new file mode 100644 index 00000000..dbb4578b --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/README.md @@ -0,0 +1,88 @@ +# Reference agent demo + +A standalone, deployable copy of the livekit.com `/agents` experience, carrying one agent so a +partner can talk to it without an account. This deployment ships **Patient Intake**, backed by the +`xai-patient-intake` worker. + +The landing page shows the agent card. Clicking it morphs the card into a live conversation panel +— transcript, tool calls, mic controls — connected to the deployed agent. + +## Run it + +```bash +pnpm install +cp .env.example .env.local # fill in the three LiveKit values +pnpm dev +``` + +## Configuration + +Three environment variables, all server-side: + +| Variable | Value | +| --------------------- | ---------------------------------------------------------------- | +| `LIVEKIT_URL` | `wss://.livekit.cloud` for the project hosting the agent | +| `LIVEKIT_API_KEY` | API key for that project | +| `LIVEKIT_API_SECRET` | API secret for that project | + +The browser never sees these. `app/api/agent/connection_details/route.ts` mints a short-lived +participant token with an explicit agent dispatch, so the worker named in the token joins the room +the visitor just created. + +**The worker must be deployed to the same LiveKit project as those credentials.** + +## Adding another agent + +The single card is data, not structure — the page renders whatever `AGENTS` holds. + +1. Add an entry to `AGENTS` in `app/(showcase)/_components/agent-metadata.ts`. Its `name` is the + public identifier and its slug is the URL (`patient_intake` → `/patient-intake`). +2. Map that name to the deployed worker name in `AGENT_DISPATCH_NAMES` in + `app/api/agent/connection_details/route.ts`. This is an allowlist — a name that isn't in it is + rejected with a 400, so a visitor can't dispatch an arbitrary worker in your project. +3. Optionally give it an accent colour in `app/(showcase)/_components/agent-themes.ts`. Agents + without one use the default cyan. + +Cards lay out as a centered row, so a second or third agent needs no layout change. + +## How this relates to livekit.com + +The components under `app/(showcase)/_components`, `components/agents-ui`, `components/ui`, and +`hooks/agents-ui` are copied from `apps/www` in the `livekit/web` monorepo. They are not a +rewrite — the intent is that a diff against the originals stays readable. The deliberate +differences: + +- **Routing.** The showcase is the whole site, so an agent sits at `/patient-intake` rather than + `/agents/patient-intake`, and the page fills the viewport instead of reserving room for the + marketing header and footer. +- **Layout.** livekit.com pins cards into a three-column grid that only centers correctly at + exactly three. Here they are a centered row that works at any count. +- **Unknown agents.** livekit.com falls back to its homepage agent; this app returns a 400. There + is no general-purpose agent to fall back to, and silently connecting someone to the wrong agent + is worse than an error. + +`@repo/bytes-core` and `@repo/bytes-react` are workspace-private and can't be installed here, so +the slice this page uses is vendored: + +- `styles/bytes-colors.css`, `styles/bytes-core.css`, `styles/bytes-react.css` — the design token + layer, copied verbatim. `styles/tailwind.css` is `apps/www`'s entry point with its + monorepo-only imports removed. +- `components/bytes/` — `Button` copied as-is; `Badge` and `IconButton` reproduced with their + class strings intact but their unused `ToggleTip` paths dropped. +- `lib/utils.ts` — the `cn` from bytes-react, whose extended tailwind-merge config is load + bearing. Under a plain `twMerge` the badge's `text-mono-caps` loses to the `text-xs` from its + size variant and the badge renders in sans sentence case. + +To refresh against upstream, re-copy the files and re-apply those differences. + +## Smoke test + +`scripts/smoke-test.mjs` drives a real browser through the whole path — card, click, token, +dispatch, and the agent's opening line — and fails loudly if the agent never speaks. It needs +Playwright, which is deliberately not a dependency of this app so it stays out of Vercel builds: + +```bash +pnpm add -D playwright && pnpm exec playwright install chromium +pnpm dev & +node scripts/smoke-test.mjs # or BASE=https://your-deployment.vercel.app +``` diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/[[...name]]/page.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/[[...name]]/page.tsx new file mode 100644 index 00000000..0b7c0000 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/[[...name]]/page.tsx @@ -0,0 +1,17 @@ +import { redirect } from 'next/navigation'; +import { resolveActiveAgent } from '../_components/agent-metadata'; + +// Validates the slug and redirects when invalid. The actual UI renders in ../layout.tsx (see its +// comment for why) — this component's own output is never shown. +export default async function AgentPage({ params }: { params: Promise<{ name?: string[] }> }) { + const { name } = await params; + const [slug, ...rest] = name ?? []; + + // Temporary redirect, not a 404: an invalid or coming-soon slug today may be a valid agent + // tomorrow, so this shouldn't be cached as a permanent dead link. + if (slug !== undefined && (rest.length > 0 || !resolveActiveAgent(slug))) { + redirect('/'); + } + + return null; +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentCards.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentCards.tsx new file mode 100644 index 00000000..5d4259cd --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentCards.tsx @@ -0,0 +1,137 @@ +'use client'; + +import { useEffect, useState } from 'react'; +import { type AgentState } from '@livekit/components-react'; +import Link from 'next/link'; +import { SiGithub } from '@icons-pack/react-simple-icons'; +import { Badge, Button, cn } from '@/components/bytes'; +import { motion } from 'motion/react'; + +import { agentAccentStyle } from '@/app/(showcase)/_components/agent-themes'; +import { AgentAudioVisualizerGrid } from '@/components/voice-agent/agent-audio-visualizer-grid'; +import { slugFromAgentName, type AgentMetadata } from './agent-metadata'; +import { agentMorphName, MORPH_SPRING } from './utils'; + +const CARD_BASE = + 'border-separator1 bg-bg1 flex lg:w-[23rem] max-w-full shrink-0 flex-col gap-6 rounded-xl border p-8 origin-center'; + +interface AgentCardProps { + agent: AgentMetadata; + onSelect: (reference: string) => void; + justExitedAgentName: string | null; +} + +export function AgentCard({ agent, onSelect, justExitedAgentName }: AgentCardProps) { + const reference = slugFromAgentName(agent.name); + const accentStyle = agentAccentStyle(agent.name); + const justExited = agent.name === justExitedAgentName; + // True while this card's own content is fading out, before the parent swaps to the + // conversation view. + const [isLeaving, setIsLeaving] = useState(false); + // Gates the content behind the box's own resize completing when this card is reappearing + // from a just-closed conversation; otherwise there's nothing to wait for. + const [contentRevealed, setContentRevealed] = useState(!justExited); + + const handleSelect = () => { + setIsLeaving(true); + // Let the content fade fully before the parent swaps to the conversation view — otherwise + // React batches both changes into one commit and the fade never gets a chance to paint. + setTimeout(() => onSelect(reference), 100); + }; + + useEffect(() => { + setTimeout(() => setContentRevealed(true), 150); + }); + + if (agent.comingSoon) { + return ( + // Fades in/out as the grid itself mounts/unmounts around a conversation starting or + // ending — the parent AnimatePresence (in AgentShowcase.tsx) keeps this mounted just long + // enough to play `exit` before the grid is actually removed. + + +
+

{agent.title}

+ {agent.description &&

{agent.description}

} +
+ + Coming soon + + + ); + } + + return ( + + + +
+
+

{agent.title}

+ {agent.headlineModel && ( + + {agent.headlineModel} + + )} +
+ {agent.description &&

{agent.description}

} +
+
+ + {agent.repoUrl && ( + + )} +
+ + + ); +} + +interface VisualizerProps { + state: AgentState; +} + +function Visualizer({ state }: VisualizerProps) { + return ( +
+ +
+ ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentConversation.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentConversation.tsx new file mode 100644 index 00000000..7893296d --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentConversation.tsx @@ -0,0 +1,134 @@ +'use client'; + +import { useCallback, useEffect, useRef, useState } from 'react'; +import Link from 'next/link'; +import { SiGithub } from '@icons-pack/react-simple-icons'; +import { ArrowLeftIcon, Badge, Button, IconButton } from '@/components/bytes'; +import { motion } from 'motion/react'; + +import { agentAccentStyle } from '@/app/(showcase)/_components/agent-themes'; +import { AgentControlBar } from '@/components/agents-ui/agent-control-bar'; +import { type AgentMetadata } from './agent-metadata'; +import { AgentSession } from './AgentSession'; +import { AgentTranscript } from './transcript/AgentTranscript'; +import { MORPH_SPRING } from './utils'; + +interface AgentConversationProps { + agent: AgentMetadata; + morphName: string; + onLeave: () => void; + /** + * True when this panel arrived via a click on a grid card (a real box morph is playing). False + * when this panel is the initial render of a direct page load — in that case there's no prior + * element to morph from, so `onLayoutAnimationComplete` below never fires, and gating the content + * behind it would leave the panel blank forever. + */ + isMorphing: boolean; +} + +export function AgentConversation({ + agent, + morphName, + isMorphing, + onLeave, +}: AgentConversationProps) { + const [contentRevealed, setContentRevealed] = useState(!isMorphing); + // True while the content is fading out, before the box starts morphing back into the grid + // card — mirrors ActiveAgentCard's handleSelect on the enter side. + const [isLeaving, setIsLeaving] = useState(false); + const isLeavingRef = useRef(false); + // The SDK disconnects the room on page unload (refresh, close, navigate away), + // which flips `connectionState` and would otherwise trigger the exit morph + + // router.push below on a page that's already being torn down — a visible + // flash of the grid mid-reload. Suppress that specific case. + const isUnloadingRef = useRef(false); + + const handleLeave = useCallback(() => { + // The disconnect button and the connection-state effect below can both end up calling this + // for the same disconnect (the button ends the session, which then flips connection state) — + // guard so we don't schedule the actual leave twice. + if (isLeavingRef.current || isUnloadingRef.current) { + return; + } + isLeavingRef.current = true; + setIsLeaving(true); + // Let the content fade fully before the box starts morphing back into the grid card — + // otherwise React batches both changes into one commit and the fade never gets a chance to + // paint (same reasoning as ActiveAgentCard.handleSelect on the enter side). + setTimeout(onLeave, 150); + }, [onLeave]); + + useEffect(() => { + const handleBeforeUnload = () => { + isUnloadingRef.current = true; + }; + window.addEventListener('beforeunload', handleBeforeUnload); + return () => window.removeEventListener('beforeunload', handleBeforeUnload); + }, []); + + return ( + setContentRevealed(true)} + className="border-separator1 mx-auto flex h-[65vh] max-h-[700px] min-h-[440px] w-full max-w-[720px] flex-col overflow-hidden rounded-xl border" + > + +
+
+ + + +

{agent.title}

+ {agent.headlineModel && ( + + {agent.headlineModel.toUpperCase()} + + )} +
+ {agent.repoUrl && ( + + )} +
+
+ + +
+ +
+ +
+ + + ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentSession.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentSession.tsx new file mode 100644 index 00000000..c5051a7d --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentSession.tsx @@ -0,0 +1,50 @@ +import { useEffect, useRef } from 'react'; +import { useSessionContext } from '@livekit/components-react'; + +import { SessionProvider } from '@/components/SessionProvider'; + +interface AgentHooksProps { + onLeave: () => void; +} + +function AgentHooks({ onLeave }: AgentHooksProps) { + const session = useSessionContext(); + const hasConnectedRef = useRef(false); + + useEffect(() => { + // No client-side noise filter on purpose. The worker runs the ai-coustics enhancer on the + // way into STT (see agent.py), and stacking a second denoiser ahead of it over-suppresses + // the quiet, trailing-off speech this agent is tuned to wait for. Suppressing on the server + // also covers SIP callers, who never run this code. + session.start({ tracks: { microphone: { enabled: true } } }); + return () => { + session.end(); + }; + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []); + + useEffect(() => { + if (session.isConnected) { + hasConnectedRef.current = true; + } else if (hasConnectedRef.current && session.connectionState !== 'connecting') { + onLeave(); + } + }, [session.isConnected, session.connectionState, onLeave]); + + return <>; +} + +interface AgentSessionProps { + agentName: string; + children: React.ReactNode; + onLeave: () => void; +} + +export function AgentSession({ agentName, children, onLeave }: AgentSessionProps) { + return ( + + + {children} + + ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentShowcase.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentShowcase.tsx new file mode 100644 index 00000000..ef52d47f --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/AgentShowcase.tsx @@ -0,0 +1,119 @@ +'use client'; + +import { useCallback, useEffect, useState } from 'react'; +import { usePathname, useRouter } from 'next/navigation'; +import Link from 'next/link'; +import { SiGithub } from '@icons-pack/react-simple-icons'; +import { Button, cn } from '@/components/bytes'; +import { AnimatePresence } from 'motion/react'; + +import { AGENTS, resolveActiveAgent } from './agent-metadata'; +import { AgentCard } from './AgentCards'; +import { AgentConversation } from './AgentConversation'; +import { agentMorphName, slugFromPathname } from './utils'; + +export function AgentShowcase() { + const pathname = usePathname(); + const router = useRouter(); + const urlReference = slugFromPathname(pathname); + + const [reference, setReference] = useState(urlReference); + useEffect(() => { + setReference(urlReference); + }, [urlReference]); + + const activeAgent = reference ? (resolveActiveAgent(reference) ?? null) : null; + const activeAgentName = activeAgent?.name ?? null; + + // Set right before leaving so the grid knows which specific card is reappearing from a + // just-closed conversation — only that card gates its content reveal behind the box's own + // layout animation completing; every other card (including on true first page load) renders + // immediately. + const [justExitedAgentName, setJustExitedAgentName] = useState(null); + + // True once a card has actually been clicked — distinguishes "opened via the grid" (a real box + // morph plays, so the panel's content should wait for it) from "loaded straight into a + // conversation URL" (no prior card, no morph ever starts, so onLayoutAnimationComplete would + // never fire and gated content would stay invisible forever). + const [enteredViaClick, setEnteredViaClick] = useState(false); + + const enterConversation = useCallback( + (ref: string) => { + setEnteredViaClick(true); + setReference(ref); + router.push(`/${ref}`, { scroll: false }); + }, + [router], + ); + + const leaveConversation = useCallback(() => { + setJustExitedAgentName(activeAgentName); + setReference(null); + router.push('/', { scroll: false }); + }, [router, activeAgentName]); + + const active = AGENTS.filter((agent) => !agent.comingSoon); + const comingSoon = AGENTS.filter((agent) => agent.comingSoon); + const split = Math.ceil(comingSoon.length / 2); + const agents = [...comingSoon.slice(0, split), ...active, ...comingSoon.slice(split)]; + + return ( + // `min-h-svh`, not livekit.com's `calc(100svh-8rem)`: that subtraction reserves room for the + // marketing header and footer, which this app doesn't render, and leaves the card sitting + // half a chrome's height above centre. +
+ {/* `initial={false}` skips every nested motion component's enter animation (including the + coming-soon cards' scale-in) on this component's own very first render — but has no + effect on later mounts, so the grid still animates in normally when it reappears after a + conversation closes. */} + + {activeAgent ? ( + // Higher than the grid's own z-30 below so the panel — and the active card mid-morph + // into it — always render above the grid while it's still fading out underneath during + // the AnimatePresence overlap window. +
+ +
+ ) : ( +
+ {/* livekit.com pins each card to a fixed slot in a three-column grid, which only + centers correctly at exactly three. This deployment sizes itself to however many + agents `AGENTS` holds, so the cards are centered as a row instead. */} +
+ {agents.map((agent) => ( +
+ +
+ ))} +
+ +
+ )} + +
+ ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/agent-metadata.ts b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/agent-metadata.ts new file mode 100644 index 00000000..2083f12a --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/agent-metadata.ts @@ -0,0 +1,40 @@ +export interface AgentMetadata { + name: string; + title: string; + description: string; + headlineModel?: string; + models: string[]; + repoUrl?: string; + comingSoon: boolean; +} + +/** Converts a URL slug (`patient-intake`) to a canonical agent name (`patient_intake`). */ +export function agentNameFromSlug(slug: string): string { + return slug.replace(/-/g, '_'); +} + +/** Converts a canonical agent name (`patient_intake`) to a URL slug (`patient-intake`). */ +export function slugFromAgentName(agentName: string): string { + return agentName.replace(/_/g, '-'); +} + +/** Resolves a URL slug to its agent metadata, excluding agents that aren't live yet. */ +export function resolveActiveAgent(slug: string): AgentMetadata | undefined { + const agent = AGENTS.find((a) => a.name === agentNameFromSlug(slug)); + return agent && !agent.comingSoon ? agent : undefined; +} + +// One entry per agent this deployment offers. Adding another agent here is most of the work of +// putting it on the page — it also needs an entry in the dispatch allowlist in +// app/api/agent/connection_details/route.ts, and optionally an accent in agent-themes.ts. +export const AGENTS: AgentMetadata[] = [ + { + name: 'patient_intake', + title: 'Patient Intake', + description: + 'A family-medicine front-desk agent. Identifies callers against a chart, books and moves appointments, collects pre-visit clinical intake, and triages red-flag symptoms to emergency care', + headlineModel: 'Grok 4.3', + models: ['xAI Speech to Text', 'Grok 4.3', 'xAI Text to Speech'], + comingSoon: false, + }, +]; diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/agent-themes.ts b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/agent-themes.ts new file mode 100644 index 00000000..06370af0 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/agent-themes.ts @@ -0,0 +1,50 @@ +import { type CSSProperties } from 'react'; + +/** + * Per-agent accent color. Re-themes the agent experience by overriding the `fgAccentPrimary1` + * token's CSS variable (visualizer, glow, accented copy). + */ +export interface AgentAccentTheme { + accentColorVar: string; +} + +export const DEFAULT_AGENT_THEME: AgentAccentTheme = { + accentColorVar: 'var(--lk-color-cyan-400)', +}; + +// Agents absent from this map keep the default accent. Values must come from the +// semantic token layer (`--lk-color-green`, not `--lk-color-green-500`) so each +// accent tracks `data-lk-theme` the way the token it replaces does — the semantic +// greens sit light on dark and deep on light. +const agentThemes: Record = { + patient_intake: { + accentColorVar: 'var(--lk-color-green)', + }, +}; + +export function getAgentTheme(agentName?: string | null): AgentAccentTheme { + if (agentName) { + const theme = agentThemes[agentName]; + if (theme) { + return theme; + } + } + return DEFAULT_AGENT_THEME; +} + +/** + * Inline style that re-themes one agent's accent, or undefined to leave the default accent + * alone. Apply it to a wrapper: everything accent-derived resolves + * `--lk-color-fgAccentPrimary1` lazily at the element that paints it — Tailwind's + * `--color-fgAccentPrimary1` behind `bg-fgAccentPrimary1`, and the visualizer cell glow's + * `--agent-accent-glow` (see styles/tailwind.css) — so overriding it here re-themes every + * descendant without either of them knowing about per-agent themes. + */ +export function agentAccentStyle( + agentName?: string | null, +): (CSSProperties & Record<`--${string}`, string>) | undefined { + const theme = getAgentTheme(agentName); + return theme === DEFAULT_AGENT_THEME + ? undefined + : { '--lk-color-fgAccentPrimary1': theme.accentColorVar }; +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/suggestions.ts b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/suggestions.ts new file mode 100644 index 00000000..49b944d7 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/suggestions.ts @@ -0,0 +1,38 @@ +export const AGENT_SUGGESTIONS_ATTRIBUTE = 'lk.agent.suggestions'; +export const TEXT_INPUT_TOPIC = 'lk.chat'; + +export interface Suggestion { + label: string; + value: string; +} + +export function parseSuggestions(raw: string | undefined): Suggestion[] { + if (!raw) { + return []; + } + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + return []; + } + if (!Array.isArray(parsed)) { + return []; + } + const suggestions: Suggestion[] = []; + for (const item of parsed) { + if ( + item && + typeof item === 'object' && + !Array.isArray(item) && + 'label' in item && + 'value' in item + ) { + const { label, value } = item; + if (typeof label === 'string' && typeof value === 'string') { + suggestions.push({ label, value }); + } + } + } + return suggestions; +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/AgentMessage.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/AgentMessage.tsx new file mode 100644 index 00000000..2a2e385a --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/AgentMessage.tsx @@ -0,0 +1,16 @@ +import { getMoodColor } from '../utils'; +import { SimulatedTextStream } from './SimulatedTextStream'; + +interface AgentMessageProps { + message: string; + expression: string; +} + +export function AgentMessage({ expression, message }: AgentMessageProps) { + const moodColor = expression ? getMoodColor(expression) : undefined; + return ( +

+ {message} +

+ ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/AgentTranscript.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/AgentTranscript.tsx new file mode 100644 index 00000000..ec7655f9 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/AgentTranscript.tsx @@ -0,0 +1,284 @@ +'use client'; + +import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; +import { + AGENT_SUGGESTIONS_ATTRIBUTE, + parseSuggestions, + TEXT_INPUT_TOPIC, + type Suggestion, +} from '@/app/(showcase)/_components/suggestions'; +import { useAgentSession } from '@/app/(showcase)/_components/use-agent-session'; +import { + useParticipantAttributes, + useRoomContext, + useTranscriptions, + useVoiceAssistant, +} from '@livekit/components-react'; +import { cn } from '@/components/bytes'; +import { AnimatePresence, motion } from 'motion/react'; + +import { agentAccentStyle } from '@/app/(showcase)/_components/agent-themes'; +import { Shimmer } from '@/components/shimmer'; +import { MORPH_SPRING, getMoodColor } from '../utils'; +import { AgentTurn } from './AgentTurn'; +import { MiniVisualizer } from './MiniVisualizer'; +import { Suggestions } from './Suggestions'; +import { UserMessage } from './UserMessage'; + +type TimelineItem = + | { kind: 'speech'; id: string; ts: number; name: string; text: string; expression?: string } + | { kind: 'tool'; id: string; ts: number; name: string; args: string }; + +// Attribute keys published on the `lk.transcription` text stream by the agents SDK. +const SEGMENT_ID_ATTRIBUTE = 'lk.segment_id'; +const EXPRESSION_ATTRIBUTE = 'lk.expression'; + +const TRANSCRIPT_INSET = 'px-8 lg:pr-20'; + +/** `lk.expression` is a JSON object (`{"value": "speak warmly"}`) so fields can be added later. */ +function parseExpression(raw: string | undefined): string | undefined { + if (!raw) { + return undefined; + } + try { + const parsed: unknown = JSON.parse(raw); + if (parsed && typeof parsed === 'object' && 'value' in parsed) { + const value = (parsed as { value: unknown }).value; + return typeof value === 'string' ? value : undefined; + } + } catch { + // tolerate a malformed attribute rather than dropping the segment + } + return undefined; +} + +interface AgentTranscriptProps { + className?: string; + /** Agent in this conversation, so an accent-themed one can opt out of the mood tint. */ + agentName?: string; +} + +export function AgentTranscript({ className, agentName }: AgentTranscriptProps) { + const { agent, state, audioTrack } = useVoiceAssistant(); + const { attributes } = useParticipantAttributes({ participant: agent }); + const room = useRoomContext(); + const { toolCalls } = useAgentSession(); + // Stream-based transcriptions (`lk.transcription` topic). Unlike the deprecated + // rtc Transcription API, this is the channel the SDK strips expressive markup + // from before publishing, and its attributes carry the segment's leading + // delivery/emotion tag as `lk.expression`. + const transcriptionStreams = useTranscriptions({ room }); + + // Log each segment's leading delivery/emotion tag (`lk.expression`) for debugging. + const loggedExpressionsRef = useRef(new Set()); + useEffect(() => { + for (const stream of transcriptionStreams) { + const attributes = stream.streamInfo.attributes ?? {}; + const segmentId = attributes[SEGMENT_ID_ATTRIBUTE] ?? stream.streamInfo.id; + const expression = parseExpression(attributes[EXPRESSION_ATTRIBUTE]); + if (expression && !loggedExpressionsRef.current.has(segmentId)) { + loggedExpressionsRef.current.add(segmentId); + console.log('[mood] lk.expression:', expression); + } + } + }, [transcriptionStreams]); + + const rawSuggestions = attributes?.[AGENT_SUGGESTIONS_ATTRIBUTE]; + const suggestions = useMemo(() => parseSuggestions(rawSuggestions), [rawSuggestions]); + const [dismissed, setDismissed] = useState(false); + const [sentMessages, setSentMessages] = useState<{ id: string; ts: number; text: string }[]>([]); + const sentCountRef = useRef(0); + const showSuggestions = !dismissed && suggestions.length > 0 && state === 'listening'; + + useEffect(() => { + setDismissed(false); + }, [rawSuggestions, state]); + + const handleSuggestionClick = useCallback( + (suggestion: Suggestion) => { + setDismissed(true); + const ts = Date.now(); + setSentMessages((prev) => [ + ...prev, + { id: `sent-${ts}-${sentCountRef.current++}`, ts, text: suggestion.value }, + ]); + void room?.localParticipant?.sendText(suggestion.value, { topic: TEXT_INPUT_TOPIC }); + }, + [room], + ); + + const timeline = useMemo(() => { + // One item per transcript segment. A delta stream (agent speech) grows a single + // stream per segment; a non-delta stream (user STT) publishes a fresh stream per + // update sharing the same segment id - later entries replace earlier ones. + const segments = new Map(); + for (const stream of transcriptionStreams) { + if (!stream.text) { + continue; + } + const attributes = stream.streamInfo.attributes ?? {}; + const segmentId = attributes[SEGMENT_ID_ATTRIBUTE] ?? stream.streamInfo.id; + const previous = segments.get(segmentId); + segments.set(segmentId, { + kind: 'speech', + id: segmentId, + ts: previous?.ts ?? stream.streamInfo.timestamp, + name: + stream.participantInfo.identity === room?.localParticipant?.identity ? 'you' : 'agent', + text: stream.text, + expression: parseExpression(attributes[EXPRESSION_ATTRIBUTE]) ?? previous?.expression, + }); + } + + const items: TimelineItem[] = [...segments.values()]; + for (const call of toolCalls) { + items.push({ + kind: 'tool', + id: `tool-${call.id}`, + ts: call.receivedAtMs, + name: call.name, + args: call.args, + }); + } + for (const sent of sentMessages) { + items.push({ kind: 'speech', id: sent.id, ts: sent.ts, name: 'you', text: sent.text }); + } + return items.sort((a, b) => a.ts - b.ts); + }, [transcriptionStreams, room, toolCalls, sentMessages]); + + const latestActionId = useMemo(() => { + for (let i = timeline.length - 1; i >= 0; i--) { + const item = timeline[i]; + if (item && (item.kind === 'tool' || (item.kind === 'speech' && item.name === 'agent'))) { + return item.id; + } + } + return null; + }, [timeline]); + + // Mood accent for the visualizer dots — the same latest transcript item the dots render next + // to (see `isLatestAction` below), not just whichever item most recently had an expression. + // An agent with its own accent keeps that accent for the whole conversation instead: the mood + // palette is fixed hues (one of which is the default accent), so tinting per turn would pull + // a themed agent's visualizer off its color on most turns. + const moodTinted = agentAccentStyle(agentName) === undefined; + const latestMoodColor = useMemo(() => { + const latestItem = timeline.find((item) => item.id === latestActionId); + if (moodTinted && latestItem?.kind === 'speech' && latestItem.expression) { + return getMoodColor(latestItem.expression); + } + return undefined; + }, [timeline, latestActionId, moodTinted]); + + const scrollRef = useRef(null); + + const [autoScroll, setAutoScroll] = useState(true); + const handleScroll = useCallback(() => { + const el = scrollRef.current; + if (!el) { + return; + } + const distanceFromBottom = el.scrollHeight - el.scrollTop - el.clientHeight; + setAutoScroll(distanceFromBottom < 48); + }, []); + + useEffect(() => { + const el = scrollRef.current; + if (autoScroll && el) { + el.scrollTop = el.scrollHeight; + } + }, [timeline, showSuggestions, autoScroll]); + + return ( + // `transition={MORPH_SPRING}` matters here, not just cosmetically: every other layout-animated + // box in this morph (the panel, the visualizer boxes) uses this same spring so they move in + // lockstep. Left on Motion's default spring, this would settle on its own out-of-sync timing + // while the panel is still mid-morph — since this component only exists on the conversation + // side, that desync only ever shows up when entering, not exiting. + +
+
+ + {timeline.length === 0 && ( +
+
+ + + + Connecting + +
+
+ )} + + {timeline.map((item, idx) => { + const isFirstItem = idx === 0; + const isLatestAction = item.id === latestActionId; + const kind = item.kind; + const isAgent = kind === 'speech' && item.name === 'agent'; + const isUser = kind === 'speech' && item.name !== 'agent'; + const message = kind === 'speech' ? item.text : undefined; + const toolName = kind === 'tool' ? item.name : undefined; + const expression = kind === 'speech' && isAgent ? item.expression : undefined; + + return ( + + {isUser ? ( + + ) : ( + + )} + + ); + })} + + + {showSuggestions && ( + + )} +
+
+ + ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/AgentTurn.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/AgentTurn.tsx new file mode 100644 index 00000000..9c13d6b5 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/AgentTurn.tsx @@ -0,0 +1,70 @@ +import type { AgentState, TrackReferenceOrPlaceholder } from '@livekit/components-react'; +import { cn } from '@/components/bytes'; + +import { AgentMessage } from './AgentMessage'; +import { MiniVisualizer } from './MiniVisualizer'; +import { ToolTag } from './ToolTag'; + +// Fixed gutter every agent/tool row reserves for the visualizer, whether or not this particular +// row is currently hosting it — keeps agent/tool content aligned consistently regardless of which +// row the visualizer (a shared layoutId'd element, see MiniVisualizer) currently occupies. +const VISUALIZER_GUTTER_WIDTH = 'w-[42px]'; + +interface AgentTurnProps { + state: AgentState; + kind: 'speech' | 'tool'; + message?: string; + toolName?: string; + expression?: string; + audioTrack?: TrackReferenceOrPlaceholder; + latestMoodColor?: string; + isFirstItem: boolean; + isLatestAction: boolean; +} + +export function AgentTurn({ + kind, + state, + message, + toolName, + expression, + audioTrack, + latestMoodColor, + isFirstItem, + isLatestAction, +}: AgentTurnProps) { + const isToolCall = kind === 'tool'; + + return ( +
+ {/* Always reserves the same width whether or not it's hosting the + visualizer, so agent/tool content stays aligned as the visualizer — a + shared layoutId'd element — moves between rows. Motion animates it into + position on its own; no manual position tracking needed. */} +
+ {isLatestAction && ( + + )} +
+ {isToolCall ? ( + + ) : ( + + )} +
+ ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/MiniVisualizer.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/MiniVisualizer.tsx new file mode 100644 index 00000000..28e00f63 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/MiniVisualizer.tsx @@ -0,0 +1,69 @@ +import { type CSSProperties } from 'react'; +import { type AgentState, type TrackReferenceOrPlaceholder } from '@livekit/components-react'; +import { arc, motion } from 'motion/react'; + +import { AgentAudioVisualizerGrid } from '@/components/agents-ui/agent-audio-visualizer-grid'; +import { MORPH_SPRING } from '../utils'; + +export interface MiniVisualizerProps { + state: AgentState; + moodColor?: string; + audioTrack?: TrackReferenceOrPlaceholder; + disableArcTransition?: boolean; +} + +/** + * Shares `layoutId={morphName}` with every other spot this visualizer can appear — the "not yet + * connected" placeholder centered in `AgentConversation`, and whichever transcript row is currently + * latest here — so Motion animates it smoothly between them instead of popping. + */ +export function MiniVisualizer({ + state, + moodColor, + audioTrack, + disableArcTransition = false, +}: MiniVisualizerProps) { + const style: CSSProperties = { + borderRadius: 8, + '--mood-color': moodColor, + } as React.CSSProperties; + + return ( + // `layout="position"` avoids this element ever attempting its own size/scale FLIP. + + + + ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/SimulatedTextStream.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/SimulatedTextStream.tsx new file mode 100644 index 00000000..86566fb1 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/SimulatedTextStream.tsx @@ -0,0 +1,127 @@ +import { useLayoutEffect, useRef, useState } from 'react'; +import { motion } from 'motion/react'; +import { semanticColors } from '@/lib/two-face-colors'; + +interface Chunk { + text: string; + delay: number; +} + +// Delay between each chunk's fade-in when several land in the same update, so a big buffered +// delta reveals word-by-word instead of all at once. +const STAGGER_SECONDS = 0.07; + +/** + * Splits `value` into pieces at internal whitespace runs — each run of whitespace stays attached + * to the chunk before it, so streaming text lands roughly one word per chunk instead of one giant + * blob. Whitespace at the very start or end doesn't force a split of its own. + */ +function splitIntoChunks(value: string): string[] { + const tokens = value.split(/(\s+)/).filter(Boolean); + const result: string[] = []; + let current = ''; + for (const token of tokens) { + if (/^\s+$/.test(token)) { + if (current === '') { + current = token; + } else { + result.push(current + token); + current = ''; + } + } else { + current += token; + } + } + if (current) { + result.push(current); + } + return result; +} + +interface SimulatedTextStreamProps { + /** + * Blur amount in px + */ + blur?: number; + /** + * Scale factor for the text + */ + scale?: number; + /** + * Initial color of the text + */ + initialColor?: string; + /** + * Final color of the text + */ + finalColor?: string; + children: string; +} + +/** + * Renders `text` as a list of spans — each time `text` grows, the newly appended portion is split + * into whitespace-delimited chunks and appended as new spans, while previously rendered chunks are + * left untouched. Lets a caller target just the newest chunks (e.g. to animate them in) without + * re-touching earlier ones. + */ +export function SimulatedTextStream({ + children, + blur = 20, + scale = 1.5, + initialColor, + finalColor, +}: SimulatedTextStreamProps) { + const [chunks, setChunks] = useState([]); + const previousTextRef = useRef(''); + // Absolute time (seconds) at which the next chunk's stagger would start. Lets a burst of + // chunks that arrives before the prior burst's stagger has finished playing continue that + // same sequence, rather than every burst restarting its delay from zero. Reading the clock has + // to happen in an effect rather than during render (render must stay pure), and a layout effect + // specifically so the correctly-delayed chunks land before the browser paints — otherwise + // there'd be a visible flash of the un-delayed text first. + const cursorRef = useRef(0); + const text = children; + + useLayoutEffect(() => { + const previousText = previousTextRef.current; + if (text === previousText) { + return; + } + const isAppend = text.startsWith(previousText); + const delta = isAppend ? text.slice(previousText.length) : text; + const now = performance.now() / 1000; + const startDelay = Math.max(0, cursorRef.current - now); + const additions = splitIntoChunks(delta).map((chunkText, i) => ({ + text: chunkText, + delay: startDelay + i * STAGGER_SECONDS, + })); + cursorRef.current = now + startDelay + additions.length * STAGGER_SECONDS; + previousTextRef.current = text; + setChunks((prev) => (isAppend ? [...prev, ...additions] : additions)); + }, [text]); + + return ( + <> + {chunks.map((chunk, index) => ( + + {chunk.text} + + ))} + + ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/Suggestions.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/Suggestions.tsx new file mode 100644 index 00000000..ff1e48c4 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/Suggestions.tsx @@ -0,0 +1,25 @@ +import type { Suggestion } from '@/app/(showcase)/_components/suggestions'; +import { Button } from '@/components/bytes'; + +interface SuggestionsProps { + suggestions: Suggestion[]; + handleSuggestionClick: (suggestion: Suggestion) => void; +} + +export function Suggestions({ suggestions, handleSuggestionClick }: SuggestionsProps) { + return ( +
+ {suggestions.map((suggestion, index) => ( + + + + ))} +
+ ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/ToolTag.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/ToolTag.tsx new file mode 100644 index 00000000..382ea533 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/ToolTag.tsx @@ -0,0 +1,13 @@ +import { Badge, SettingsGear1Icon } from '@/components/bytes'; + +interface ToolTagProps { + name: string; +} + +export function ToolTag({ name }: ToolTagProps) { + return ( + } className="tracking-wider"> + {name.toUpperCase()} + + ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/UserMessage.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/UserMessage.tsx new file mode 100644 index 00000000..f0d1bd63 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/transcript/UserMessage.tsx @@ -0,0 +1,15 @@ +import { SimulatedTextStream } from './SimulatedTextStream'; + +interface UserMessageProps { + message: string; +} + +export function UserMessage({ message }: UserMessageProps) { + return ( +
+

+ {message} +

+
+ ); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/use-agent-session.ts b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/use-agent-session.ts new file mode 100644 index 00000000..fbb02110 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/use-agent-session.ts @@ -0,0 +1,244 @@ +'use client'; + +// Agent activity surfaced from the LiveKit Agents session event stream. The +// framework publishes `AgentSessionMessage` protobufs on a byte stream (topic +// `lk.agent.session`); we decode two things from it: +// - tool calls (`functionToolsExecuted` → each function call's name + args) +// - the live model config (`sessionUsageUpdated` → STT/LLM/TTS provider+model) +// - latency metrics (`conversationItemAdded` → the assistant message's +// `MetricsReport`: STT delay, end-of-turn delay, LLM TTFT, TTS TTFB, e2e) +// No agent-side code is required — this is the same stream the jukebox dev +// playground consumes. +// +// A byte-stream handler is single-per-topic per room, so this hook must be +// called exactly once within a given room (here: once per TranscriptionSection). + +import { useCallback, useEffect, useRef, useState } from 'react'; +import { useRoomContext } from '@livekit/components-react'; +import { AgentSession } from '@livekit/protocol'; +import { ConnectionState, RoomEvent, type ByteStreamHandler } from 'livekit-client'; + +const TOPIC_SESSION_MESSAGES = 'lk.agent.session'; + +export interface ToolCall { + /** `callId` from the agent (stable per invocation), or a synthetic fallback. */ + id: string; + /** Tool/function name, e.g. `lookup_booking`. */ + name: string; + /** Raw call arguments, a JSON string (e.g. `{"last_name":"Smith"}`) or empty. */ + args: string; + /** Arrival time, used to order tool calls against transcript segments. */ + receivedAtMs: number; +} + +/** Model config reported by the session, derived from per-model usage. */ +export interface AgentSessionConfig { + sttProvider: string; + sttModel: string; + llmProvider: string; + llmModel: string; + ttsProvider: string; + ttsModel: string; +} + +const EMPTY_CONFIG: AgentSessionConfig = { + sttProvider: '', + sttModel: '', + llmProvider: '', + llmModel: '', + ttsProvider: '', + ttsModel: '', +}; + +/** Latency breakdown for the most recent agent turn, in milliseconds. */ +export interface AgentSessionMetrics { + sttLatencyMs: number; + eotLatencyMs: number; + llmTtftMs: number; + ttsTtfbMs: number; + e2eLatencyMs: number; +} + +export interface AgentSessionData { + toolCalls: ToolCall[]; + /** Latest known model config; fields fill in as each model reports usage. */ + config: AgentSessionConfig; + /** Latency from the latest agent turn, or null until the first turn completes. */ + metrics: AgentSessionMetrics | null; +} + +// MetricsReport delays are reported in seconds; the panel shows milliseconds. +// When a report omits a field, keep the prior value (`fallback`) rather than zeroing it. +function toMs(seconds: number | undefined, fallback = 0): number { + return seconds === undefined ? fallback : Math.round(seconds * 1000); +} + +/** Live tool calls and model config for the agent in the current room. */ +export function useAgentSession(): AgentSessionData { + const room = useRoomContext(); + const [toolCalls, setToolCalls] = useState([]); + const [config, setConfig] = useState(EMPTY_CONFIG); + const [metrics, setMetrics] = useState(null); + // callIds already recorded — guards against duplicate event delivery. + const seenRef = useRef>(new Set()); + const synthRef = useRef(0); + // Latest config, merged across usage events (an event may report only some models). + const configRef = useRef(EMPTY_CONFIG); + // Latest latency, merged across reports: the user-turn message carries STT + + // end-of-turn delays, the agent-turn message carries LLM/TTS/e2e — each report + // omits the other half, so we keep prior values for fields it doesn't include. + const metricsRef = useRef(null); + + const reset = useCallback(() => { + seenRef.current = new Set(); + synthRef.current = 0; + configRef.current = EMPTY_CONFIG; + metricsRef.current = null; + setToolCalls([]); + setConfig(EMPTY_CONFIG); + setMetrics(null); + }, []); + + useEffect(() => { + if (!room) { + return; + } + + let registered = false; + + const handleStream: ByteStreamHandler = async (reader) => { + const receivedAtMs = Date.now(); + let message; + try { + const chunks = await reader.readAll(); + let total = 0; + for (const chunk of chunks) { + total += chunk.byteLength; + } + const data = new Uint8Array(total); + let offset = 0; + for (const chunk of chunks) { + data.set(chunk, offset); + offset += chunk.byteLength; + } + message = AgentSession.AgentSessionMessage.fromBinary(data); + } catch (error) { + console.warn('[useAgentSession] failed to parse session message', error); + return; + } + + if (message.message.case !== 'event') { + return; + } + const event = message.message.value.event; + + if (event.case === 'functionToolsExecuted') { + const fresh: ToolCall[] = []; + for (const call of event.value.functionCalls) { + const id = call.callId || `tool-${++synthRef.current}`; + if (seenRef.current.has(id)) { + continue; + } + seenRef.current.add(id); + fresh.push({ id, name: call.name, args: call.arguments, receivedAtMs }); + } + if (fresh.length > 0) { + setToolCalls((prev) => [...prev, ...fresh]); + } + return; + } + + if (event.case === 'sessionUsageUpdated') { + const next = { ...configRef.current }; + for (const modelUsage of event.value.usage?.modelUsage ?? []) { + const usage = modelUsage.usage; + if (usage.case === 'stt') { + next.sttProvider = usage.value.provider; + next.sttModel = usage.value.model; + } else if (usage.case === 'llm') { + next.llmProvider = usage.value.provider; + next.llmModel = usage.value.model; + } else if (usage.case === 'tts') { + next.ttsProvider = usage.value.provider; + next.ttsModel = usage.value.model; + } + } + const changed = (Object.keys(next) as (keyof AgentSessionConfig)[]).some( + (key) => next[key] !== configRef.current[key], + ); + if (changed) { + configRef.current = next; + setConfig(next); + } + return; + } + + if (event.case === 'conversationItemAdded') { + const item = event.value.item?.item; + if (item?.case !== 'message' || !item.value.metrics) { + return; + } + const report = item.value.metrics; + const prev = metricsRef.current; + // Each report fills only its half; keep prior values for absent fields. + const next: AgentSessionMetrics = { + sttLatencyMs: toMs(report.transcriptionDelay, prev?.sttLatencyMs), + eotLatencyMs: toMs(report.endOfTurnDelay, prev?.eotLatencyMs), + llmTtftMs: toMs(report.llmNodeTtft, prev?.llmTtftMs), + ttsTtfbMs: toMs(report.ttsNodeTtfb, prev?.ttsTtfbMs), + e2eLatencyMs: toMs(report.e2eLatency, prev?.e2eLatencyMs), + }; + metricsRef.current = next; + setMetrics(next); + } + }; + + const register = () => { + if (registered) { + return; + } + try { + room.registerByteStreamHandler(TOPIC_SESSION_MESSAGES, handleStream); + registered = true; + } catch (error) { + console.warn('[useAgentSession] failed to register handler', error); + } + }; + + const unregister = () => { + if (!registered) { + return; + } + try { + room.unregisterByteStreamHandler(TOPIC_SESSION_MESSAGES); + } catch { + // Handler already torn down — nothing to do. + } + registered = false; + }; + + // Each connection starts a fresh agent session — drop stale state. + const onConnected = () => { + reset(); + register(); + }; + const onDisconnected = () => { + unregister(); + reset(); + }; + + if (room.state === ConnectionState.Connected) { + register(); + } + room.on(RoomEvent.Connected, onConnected); + room.on(RoomEvent.Disconnected, onDisconnected); + + return () => { + room.off(RoomEvent.Connected, onConnected); + room.off(RoomEvent.Disconnected, onDisconnected); + unregister(); + }; + }, [room, reset]); + + return { toolCalls, config, metrics }; +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/utils.ts b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/utils.ts new file mode 100644 index 00000000..046556b7 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/_components/utils.ts @@ -0,0 +1,117 @@ +// Shared by every layoutId'd box in the grid<->conversation morph (the outer box and the +// visualizer, in both AgentCards.tsx and AgentTranscript.tsx) so they move in lockstep instead of +// drifting apart mid-transition. +export const MORPH_SPRING = { + type: 'spring', + stiffness: 675, + damping: 75, + mass: 1, +} as const; + +export function agentMorphName(agentName: string): string { + return `agent-morph-${agentName}`; +} + +/** The agent slug, or null for the bare `/` grid route. The showcase is the whole site here, + * so an agent sits at `/patient-intake` rather than livekit.com's `/agents/patient-intake`. */ +export function slugFromPathname(pathname: string): string | null { + return pathname === '/' ? null : pathname.replace(/^\//, ''); +} + +import { twoFaceColors } from '@/lib/two-face-colors'; + +export const MOOD_GREEN = twoFaceColors.green.dark; // happy / bright +export const MOOD_YELLOW = twoFaceColors.yellow.dark; // warmth / empathy +export const MOOD_RED = twoFaceColors.red.dark; // anger +export const MOOD_BLUE = twoFaceColors.blue.dark; // sadness +export const MOOD_TRUST = twoFaceColors.accent.dark; // trust +export const MOOD_CONFUSED = twoFaceColors.purple.dark; // confused + +const MOOD_RULES: { keywords: string[]; hex: string }[] = [ + { + keywords: ['angr', 'furious', 'irrit', 'frustrat', 'annoy', 'rage'], + hex: MOOD_RED, + }, + { + keywords: ['sad', 'sorrow', 'melanchol', 'grief', 'gloom', 'despair', 'unhappy', 'mourn'], + hex: MOOD_BLUE, + }, + { + keywords: [ + 'happ', + 'bright', + 'cheer', + 'joy', + 'delight', + 'excit', + 'upbeat', + 'playful', + 'glad', + 'warm', + 'genuine', + ], + hex: MOOD_GREEN, + }, + { + keywords: [ + 'slow', + 'trust', + 'confident', + 'certain', + 'secur', + 'convinc', + 'believ', + 'reassur', + 'patient', + 'clear', + ], + hex: MOOD_TRUST, + }, + { + keywords: [ + 'soft', + 'gentl', + 'care', + 'confused', + 'nervous', + 'ambiguous', + 'doubtful', + 'indecisive', + 'genuin', + ], + hex: MOOD_CONFUSED, + }, + // { + // keywords: [ + // 'warm', + // 'empath', + // 'compassion', + // 'tender', + // 'caring', + // 'sooth', + // 'reassur', + // 'gentl', + // 'affection', + // ], + // hex: MOOD_YELLOW, + // }, +]; + +/** + * Maps a freeform `lk.expression` value (e.g. "speak happily") to a mood accent color, by scoring + * each rule on how many of its keywords appear and returning the highest-scoring rule's color. A + * tie goes to whichever rule comes first in `MOOD_RULES`. + */ +export function getMoodColor(expressionValue: string): string | undefined { + const lower = expressionValue.toLowerCase(); + let bestRule: (typeof MOOD_RULES)[number] | undefined; + let bestScore = 0; + for (const rule of MOOD_RULES) { + const score = rule.keywords.filter((keyword) => lower.includes(keyword)).length; + if (score > bestScore) { + bestScore = score; + bestRule = rule; + } + } + return bestRule?.hex; +} diff --git a/complex-agents/xai-patient-intake/frontend/app/(showcase)/layout.tsx b/complex-agents/xai-patient-intake/frontend/app/(showcase)/layout.tsx new file mode 100644 index 00000000..69f5c759 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/(showcase)/layout.tsx @@ -0,0 +1,17 @@ +import { AgentShowcase } from './_components/AgentShowcase'; + +// AgentShowcase renders here, not in [[...name]]/page.tsx, because layouts persist across +// client-side navigations within their subtree while pages remount on every navigation — even +// within the same dynamic segment template. AgentShowcase must not remount when moving between +// the grid and a conversation, since that tears down and re-establishes the live LiveKit room +// connection. +// +// `children` (the page) still has to be rendered here, even though the page itself always +// renders null: Next only turns a page's `redirect()` call into an actual response if the layout +// renders `children` somewhere — an unrendered child's redirect is silently dropped. A parent +// layout also doesn't receive a child dynamic segment's params (Next only scopes params down to +// the segment that defines them), so slug validation/redirect can't live here anyway — see +// [[...name]]/page.tsx. +export default function ShowcaseLayout({ children }: { children: React.ReactNode }) { + return ; +} diff --git a/complex-agents/xai-patient-intake/frontend/app/api/agent/connection_details/route.ts b/complex-agents/xai-patient-intake/frontend/app/api/agent/connection_details/route.ts new file mode 100644 index 00000000..ad8f5ca1 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/api/agent/connection_details/route.ts @@ -0,0 +1,121 @@ +import { NextResponse, type NextRequest } from 'next/server'; +import { RoomAgentDispatch, RoomConfiguration } from '@livekit/protocol'; +import { AccessToken, type AccessTokenOptions } from 'livekit-server-sdk'; + +type ConnectionDetails = { + serverUrl: string; + roomName: string; + participantName: string; + participantToken: string; +}; + +const LIVEKIT_URL = process.env.LIVEKIT_URL; +const LIVEKIT_API_KEY = process.env.LIVEKIT_API_KEY; +const LIVEKIT_API_SECRET = process.env.LIVEKIT_API_SECRET; + +// Maps the public agent identifier (the `?agent=` value, derived from the URL slug — e.g. +// `/patient-intake` -> `patient_intake`) to the worker name registered with the agent +// dispatcher. This is an allowlist: only agents listed here can be dispatched, so a client +// can't request an arbitrary worker in the project. Keep it in step with `AGENTS` in +// app/(showcase)/_components/agent-metadata.ts. +// +// Unlike livekit.com, an unknown name is rejected rather than falling back to a default agent — +// this deployment has no general-purpose agent to fall back to, and silently connecting a +// visitor to something other than the agent they asked for is worse than an error. +const AGENT_DISPATCH_NAMES: Record = { + patient_intake: 'xai-patient-intake', +}; + +export const revalidate = 0; + +export async function POST(request: NextRequest) { + try { + if (LIVEKIT_URL === undefined) { + throw new Error('LIVEKIT_URL is not defined'); + } + if (LIVEKIT_API_KEY === undefined) { + throw new Error('LIVEKIT_API_KEY is not defined'); + } + if (LIVEKIT_API_SECRET === undefined) { + throw new Error('LIVEKIT_API_SECRET is not defined'); + } + + const s2s = request.nextUrl.searchParams.get('s2s') === 'true'; + const useGateway = request.nextUrl.searchParams.get('useGateway') === 'true'; + + // Dispatch the agent requested via `?agent=`, resolving the public name to its worker + // name through the allowlist. + const requestedAgent = request.nextUrl.searchParams.get('agent'); + const agentName = requestedAgent ? AGENT_DISPATCH_NAMES[requestedAgent] : undefined; + if (agentName === undefined) { + return new NextResponse(`Unknown agent: ${requestedAgent ?? '(none requested)'}`, { + status: 400, + }); + } + + const participantName = 'user'; + const participantIdentity = `user_${Math.floor(Math.random() * 10_000)}`; + const roomName = `demo_${Math.floor(Math.random() * 10_000)}_${Math.floor(Math.random() * 10_000)}`; + + const participantToken = await createParticipantToken( + { identity: participantIdentity, name: participantName }, + roomName, + { useGateway, s2s, agentName }, + ); + + const data: ConnectionDetails = { + serverUrl: LIVEKIT_URL, + roomName, + participantToken, + participantName, + }; + const headers = new Headers({ + 'Cache-Control': 'no-store', + }); + return NextResponse.json(data, { headers }); + } catch (error) { + if (error instanceof Error) { + console.error(error); + return new NextResponse(error.message, { status: 500 }); + } + return new NextResponse('Unknown error', { status: 500 }); + } +} + +function createParticipantToken( + userInfo: AccessTokenOptions, + roomName: string, + opts: { useGateway: boolean; s2s: boolean; agentName: string }, +): Promise { + const at = new AccessToken(LIVEKIT_API_KEY, LIVEKIT_API_SECRET, { + ...userInfo, + ttl: '15m', + }); + + at.addGrant({ + room: roomName, + roomJoin: true, + canPublish: true, + canPublishData: true, + canSubscribe: true, + }); + + at.roomConfig = new RoomConfiguration({ + agents: [ + new RoomAgentDispatch({ + agentName: opts.agentName, + metadata: JSON.stringify({ + useGateway: opts.useGateway, + }), + }), + ], + }); + + if (opts.s2s) { + at.attributes = { + s2s: 'true', + }; + } + + return at.toJwt(); +} diff --git a/complex-agents/xai-patient-intake/frontend/app/layout.tsx b/complex-agents/xai-patient-intake/frontend/app/layout.tsx new file mode 100644 index 00000000..6c308c5e --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/app/layout.tsx @@ -0,0 +1,30 @@ +import type { Metadata } from 'next'; + +import { cn } from '@/lib/utils'; +import { displayFont, monoFont, sansFont } from '@/lib/fonts'; +import '@/styles/tailwind.css'; + +export const metadata: Metadata = { + title: 'LiveKit reference agents', + description: 'Try a LiveKit reference voice agent in your browser.', +}; + +export default function RootLayout({ children }: { children: React.ReactNode }) { + return ( + // `data-lk-theme` is what the bytes-core token layer switches on; dark is the default for + // LiveKit product surfaces. There is no theme toggle here — the marketing site owns that + // and this app is a single embedded demo. + + + {children} + + + ); +} diff --git a/complex-agents/xai-patient-intake/frontend/components/SessionProvider.tsx b/complex-agents/xai-patient-intake/frontend/components/SessionProvider.tsx new file mode 100644 index 00000000..a2879b66 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/components/SessionProvider.tsx @@ -0,0 +1,33 @@ +'use client'; + +import { useMemo } from 'react'; +import { + SessionProvider as LiveKitSessionProvider, + RoomAudioRenderer, + useSession, +} from '@livekit/components-react'; +import { TokenSource } from 'livekit-client'; + +interface SessionProviderProps { + children: React.ReactNode; + /** Dispatch a specific agent (from the `?agent=` query param) instead of the default. */ + agentName?: string; +} + +export function SessionProvider({ children, agentName }: SessionProviderProps) { + const tokenSource = useMemo(() => { + const params = new URLSearchParams({ useGateway: 'true' }); + if (agentName) { + params.set('agent', agentName); + } + return TokenSource.endpoint(`/api/agent/connection_details?${params.toString()}`); + }, [agentName]); + const session = useSession(tokenSource); + + return ( + + {children} + + + ); +} diff --git a/complex-agents/xai-patient-intake/frontend/components/agents-ui/agent-audio-visualizer-bar.tsx b/complex-agents/xai-patient-intake/frontend/components/agents-ui/agent-audio-visualizer-bar.tsx new file mode 100644 index 00000000..ce02118c --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/components/agents-ui/agent-audio-visualizer-bar.tsx @@ -0,0 +1,247 @@ +'use client'; + +import React, { + type CSSProperties, + Children, + type ComponentProps, + type ReactNode, + cloneElement, + isValidElement, + useMemo, +} from 'react'; +import { type VariantProps, cva } from 'class-variance-authority'; +import { type LocalAudioTrack, type RemoteAudioTrack } from 'livekit-client'; +import { + type AgentState, + type TrackReferenceOrPlaceholder, + useMultibandTrackVolume, +} from '@livekit/components-react'; +import { useAgentAudioVisualizerBarAnimator } from '@/hooks/agents-ui/use-agent-audio-visualizer-bar'; +import { cn } from '@/lib/utils'; + +/** + * Resizes an array of per-band volume values to exactly `count` entries. + * Excess values are trimmed from the end; if there are too few, the last + * value is duplicated to fill the remainder. An empty array is padded with 0s. + */ +export function normalizeVolumeBands(bands: number[], count: number): number[] { + if (bands.length === count) return bands; + if (bands.length > count) return bands.slice(0, count); + const lastValue = bands[bands.length - 1] ?? 0; + return [...bands, ...new Array(count - bands.length).fill(lastValue)]; +} + +function cloneSingleChild( + children: ReactNode | ReactNode[], + props?: Record, + key?: unknown, +) { + return Children.map(children, (child) => { + // Checking isValidElement is the safe way and avoids a typescript error too. + if (isValidElement(child) && Children.only(children)) { + const childProps = child.props as Record; + if (childProps.className) { + // make sure we retain classnames of both passed props and child + props ??= {}; + props.className = cn(childProps.className as string, props.className as string); + props.style = { + ...(childProps.style as CSSProperties), + ...(props.style as CSSProperties), + }; + } + return cloneElement(child, { ...props, key: key ? String(key) : undefined }); + } + return child; + }); +} + +export const AgentAudioVisualizerBarElementVariants = cva( + [ + 'rounded-full transition-colors duration-250 ease-linear', + 'bg-current/10 data-[lk-highlighted=true]:bg-current', + ], + { + variants: { + size: { + icon: 'min-h-[4px] w-[4px]', + sm: 'min-h-[8px] w-[8px]', + md: 'min-h-[16px] w-[16px]', + lg: 'min-h-[32px] w-[32px]', + xl: 'min-h-[64px] w-[64px]', + }, + }, + defaultVariants: { + size: 'md', + }, + }, +); + +export const AgentAudioVisualizerBarVariants = cva('relative flex items-center justify-center', { + variants: { + size: { + icon: 'h-[24px] gap-[2px]', + sm: 'h-[56px] gap-[4px]', + md: 'h-[112px] gap-[8px]', + lg: 'h-[224px] gap-[16px]', + xl: 'h-[448px] gap-[32px]', + }, + }, + defaultVariants: { + size: 'md', + }, +}); + +/** + * Props for the AgentAudioVisualizerBar component. + */ +export interface AgentAudioVisualizerBarProps { + /** + * The size of the visualizer. + * @defaultValue 'md' + */ + size?: 'icon' | 'sm' | 'md' | 'lg' | 'xl'; + /** + * The current state of the agent. Determines the animation pattern. + * @defaultValue 'connecting' + */ + state?: AgentState; + /** + * The color of the bars in hexidecimal format. + */ + color?: `#${string}`; + /** + * The number of bars to display in the visualizer. + * If not provided, defaults based on size: 3 for 'icon'/'sm', 5 for others. + */ + barCount?: number; + /** + * The audio track to visualize. Can be a local/remote audio track or a track reference. + */ + audioTrack?: LocalAudioTrack | RemoteAudioTrack | TrackReferenceOrPlaceholder; + /** + * Volume values (0-1) to use instead of the values computed from the audioTrack. + * The volumeBands.length should match barCount. + */ + volumeBands?: number[]; + /** + * Additional CSS class names to apply to the container. + */ + className?: string; + /** + * Custom div element to render as grid cells. Each child receives data-lk-index, + * data-lk-highlighted props and style props for height. Must be a single div element. + */ + children?: ReactNode; +} + +/** + * A bar-style audio visualizer that responds to agent state and audio levels. + * Displays animated bars that react to the current agent state (connecting, thinking, speaking, etc.) + * and audio volume when speaking. + * + * @extends ComponentProps<'div'> + * + * @example + * ```tsx + * + * ``` + */ +export function AgentAudioVisualizerBar({ + size = 'md', + state = 'connecting', + color, + barCount, + audioTrack, + volumeBands, + className, + children, + style, + ...props +}: AgentAudioVisualizerBarProps & + VariantProps & + ComponentProps<'div'>) { + const _barCount = useMemo(() => { + if (barCount) { + return barCount; + } + switch (size) { + case 'icon': + case 'sm': + return 3; + default: + return 5; + } + }, [barCount, size]); + + const multibandVolume = useMultibandTrackVolume(audioTrack, { + bands: _barCount, + loPass: 100, + hiPass: 200, + }); + const resolvedVolumeBands = volumeBands + ? normalizeVolumeBands(volumeBands, _barCount) + : multibandVolume; + + const sequencerInterval = useMemo(() => { + switch (state) { + case 'connecting': + return 2000 / _barCount; + case 'initializing': + return 2000; + case 'listening': + return 500; + case 'thinking': + return 150; + default: + return 1000; + } + }, [state, _barCount]); + + const highlightedIndices = useAgentAudioVisualizerBarAnimator( + state, + _barCount, + sequencerInterval, + ); + + const bands = useMemo( + () => (state === 'speaking' ? resolvedVolumeBands : new Array(_barCount).fill(0)), + [state, resolvedVolumeBands, _barCount], + ); + + if (children && Array.isArray(children)) { + throw new Error('AgentAudioVisualizerBar children must be a single element.'); + } + + return ( +
+ {bands.map((band: number, idx: number) => + children ? ( + + {cloneSingleChild(children, { + 'data-lk-index': idx, + 'data-lk-highlighted': highlightedIndices.includes(idx), + style: { height: `${band * 100}%` }, + })} + + ) : ( +
+ ), + )} +
+ ); +} diff --git a/complex-agents/xai-patient-intake/frontend/components/agents-ui/agent-audio-visualizer-grid.tsx b/complex-agents/xai-patient-intake/frontend/components/agents-ui/agent-audio-visualizer-grid.tsx new file mode 100644 index 00000000..630bac40 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/components/agents-ui/agent-audio-visualizer-grid.tsx @@ -0,0 +1,316 @@ +'use client'; + +import React, { + type CSSProperties, + Children, + type ComponentProps, + type ReactNode, + cloneElement, + isValidElement, + memo, + useMemo, +} from 'react'; +import { type VariantProps, cva } from 'class-variance-authority'; +import type { LocalAudioTrack, RemoteAudioTrack } from 'livekit-client'; +import { + type AgentState, + type TrackReferenceOrPlaceholder, + useMultibandTrackVolume, +} from '@livekit/components-react'; +import { + type Coordinate, + useAgentAudioVisualizerGridAnimator, +} from '@/hooks/agents-ui/use-agent-audio-visualizer-grid'; +import { cn } from '@/lib/utils'; + +/** + * Resizes an array of per-band volume values to exactly `count` entries. + * Excess values are trimmed from the end; if there are too few, the last + * value is duplicated to fill the remainder. An empty array is padded with 0s. + */ +export function normalizeVolumeBands(bands: number[], count: number): number[] { + if (bands.length === count) return bands; + if (bands.length > count) return bands.slice(0, count); + const lastValue = bands[bands.length - 1] ?? 0; + return [...bands, ...new Array(count - bands.length).fill(lastValue)]; +} + +function cloneSingleChild( + children: ReactNode | ReactNode[], + props?: Record, + key?: unknown, +) { + return Children.map(children, (child) => { + // Checking isValidElement is the safe way and avoids a typescript error too. + if (isValidElement(child) && Children.only(children)) { + const childProps = child.props as Record; + if (childProps.className) { + // make sure we retain classnames of both passed props and child + props ??= {}; + props.className = cn(childProps.className as string, props.className as string); + props.style = { + ...(childProps.style as CSSProperties), + ...(props.style as CSSProperties), + }; + } + return cloneElement(child, { ...props, key: key ? String(key) : undefined }); + } + return child; + }); +} + +export const AgentAudioVisualizerGridCellVariants = cva( + [ + 'h-1 w-1 place-self-center rounded-full bg-current/10 transition-all ease-out', + 'data-[lk-highlighted=true]:bg-current', + ], + { + variants: { + size: { + icon: ['h-[2px] w-[2px]'], + sm: ['h-[4px] w-[4px]'], + md: ['h-[8px] w-[8px]'], + lg: ['h-[12px] w-[12px]'], + xl: ['h-[16px] w-[16px]'], + }, + }, + defaultVariants: { + size: 'md', + }, + }, +); + +export const AgentAudioVisualizerGridVariants = cva('grid', { + variants: { + size: { + icon: ['gap-[2px]'], + sm: ['gap-[4px]'], + md: ['gap-[8px]'], + lg: ['gap-[12px]'], + xl: ['gap-[16px]'], + }, + }, + defaultVariants: { + size: 'md', + }, +}); + +/** + * Configuration options for the grid visualizer. + */ +export interface GridOptions { + /** + * The radius for the animation spread effect. + */ + radius?: number; + /** + * The interval in milliseconds between animation frames. + * @defaultValue 100 + */ + interval?: number; + /** + * The number of rows in the grid. + * @defaultValue 5 + */ + rowCount?: number; + /** + * The number of columns in the grid. + * @defaultValue 5 + */ + columnCount?: number; + /** + * Additional CSS class names to apply to the container. + */ + className?: string; +} + +const sizeDefaults = { + icon: 3, + sm: 5, + md: 5, + lg: 5, + xl: 5, +}; + +function useGrid( + size: VariantProps['size'] = 'md', + columnCount = sizeDefaults[size as keyof typeof sizeDefaults], + rowCount = sizeDefaults[size as keyof typeof sizeDefaults], +) { + return useMemo(() => { + const _columnCount = columnCount; + const _rowCount = rowCount ?? columnCount; + const items = new Array(_columnCount * _rowCount).fill(0).map((_, idx) => idx); + + return { columnCount: _columnCount, rowCount: _rowCount, items }; + }, [columnCount, rowCount]); +} + +interface GridCellProps { + index: number; + state: AgentState; + interval: number; + rowCount: number; + columnCount: number; + volumeBands: number[]; + highlightedCoordinate: Coordinate; + children?: ReactNode; +} + +const GridCell = memo(function GridCell({ + index, + state, + interval, + rowCount, + columnCount, + volumeBands, + highlightedCoordinate, + children, +}: GridCellProps) { + if (state === 'speaking') { + const y = Math.floor(index / columnCount); + const rowMidPoint = Math.floor(rowCount / 2); + const volumeChunks = 1 / (rowMidPoint + 1); + const distanceToMid = Math.abs(rowMidPoint - y); + const threshold = distanceToMid * volumeChunks; + const isHighlighted = (volumeBands[index % columnCount] ?? 0) >= threshold; + + return cloneSingleChild(children, { + 'data-lk-index': index, + 'data-lk-highlighted': isHighlighted, + }); + } + + const isHighlighted = + highlightedCoordinate.x === index % columnCount && + highlightedCoordinate.y === Math.floor(index / columnCount); + + const transitionDurationInSeconds = interval / (isHighlighted ? 1000 : 100); + + return cloneSingleChild(children, { + 'data-lk-index': index, + 'data-lk-highlighted': isHighlighted, + style: { + transitionDuration: `${transitionDurationInSeconds}s`, + }, + }); +}); + +/** + * Props for the AgentAudioVisualizerGrid component. + */ +export type AgentAudioVisualizerGridProps = GridOptions & { + /** + * The size of the visualizer. + * @defaultValue 'md' + */ + size?: 'icon' | 'sm' | 'md' | 'lg' | 'xl'; + /** + * The current state of the agent. Determines the animation pattern. + * @defaultValue 'connecting' + */ + state?: AgentState; + /** + * The color of the grid cells in hexidecimal format. + */ + color?: `#${string}`; + /** + * The audio track to visualize. Can be a local/remote audio track or a track reference. + */ + audioTrack?: LocalAudioTrack | RemoteAudioTrack | TrackReferenceOrPlaceholder; + /** + * Volume values (0-1) to use instead of the values computed from the audioTrack. + * The volumeBands.length should match columnCount. + */ + volumeBands?: number[]; + /** + * Additional CSS class names to apply to the container. + */ + className?: string; + /** + * Custom element to render as grid cells. Each child receives data-lk-index + * and data-lk-highlighted props. + */ + children?: ReactNode; +} & VariantProps; + +/** + * A grid-style audio visualizer that responds to agent state and audio levels. + * Displays an animated grid of cells that react to the current agent state + * and audio volume when speaking. + * + * @extends ComponentProps<'div'> + * + * @example + * ```tsx + * + * ``` + */ +export function AgentAudioVisualizerGrid({ + size = 'md', + state = 'connecting', + radius, + color, + rowCount: _rowCount = 5, + columnCount: _columnCount = 5, + interval = 100, + className, + children, + audioTrack, + volumeBands, + style, + ...props +}: AgentAudioVisualizerGridProps & ComponentProps<'div'>) { + const { columnCount, rowCount, items } = useGrid(size, _columnCount, _rowCount); + const highlightedCoordinate = useAgentAudioVisualizerGridAnimator( + state, + rowCount, + columnCount, + interval, + radius, + ); + const multibandVolume = useMultibandTrackVolume(audioTrack, { + bands: columnCount, + loPass: 100, + hiPass: 200, + }); + const resolvedVolumeBands = volumeBands + ? normalizeVolumeBands(volumeBands, columnCount) + : multibandVolume; + + if (children && Array.isArray(children)) { + throw new Error('AgentAudioVisualizerGrid children must be a single element.'); + } + + return ( +
+ {items.map((idx) => ( + + {children ??
} + + ))} +
+ ); +} diff --git a/complex-agents/xai-patient-intake/frontend/components/agents-ui/agent-control-bar.tsx b/complex-agents/xai-patient-intake/frontend/components/agents-ui/agent-control-bar.tsx new file mode 100644 index 00000000..603a3c48 --- /dev/null +++ b/complex-agents/xai-patient-intake/frontend/components/agents-ui/agent-control-bar.tsx @@ -0,0 +1,407 @@ +'use client'; + +import { useEffect, useRef, useState, type ComponentProps } from 'react'; +import { useChat } from '@livekit/components-react'; +import { Track } from 'livekit-client'; +import { Loader, MessageSquareTextIcon, SendHorizontal } from 'lucide-react'; +import { motion, type MotionProps } from 'motion/react'; + +import { cn } from '@/lib/utils'; +import { AgentDisconnectButton } from '@/components/agents-ui/agent-disconnect-button'; +import { AgentTrackControl } from '@/components/agents-ui/agent-track-control'; +import { + AgentTrackToggle, + agentTrackToggleVariants, +} from '@/components/agents-ui/agent-track-toggle'; +import { Button } from '@/components/ui/button'; +import { Toggle } from '@/components/ui/toggle'; +import { + useInputControls, + usePublishPermissions, + type UseInputControlsProps, +} from '@/hooks/agents-ui/use-agent-control-bar'; + +const LK_TOGGLE_VARIANT_1 = [ + 'data-[state=off]:bg-accent data-[state=off]:hover:bg-foreground/10', + 'data-[state=off]:[&_~_button]:bg-accent data-[state=off]:[&_~_button]:hover:bg-foreground/10', + 'data-[state=off]:border-border data-[state=off]:hover:border-foreground/12', + 'data-[state=off]:[&_~_button]:border-border data-[state=off]:[&_~_button]:hover:border-foreground/12', + 'data-[state=off]:text-destructive data-[state=off]:hover:text-destructive data-[state=off]:focus:text-destructive', + 'data-[state=off]:focus-visible:ring-foreground/12 data-[state=off]:focus-visible:border-ring', + 'dark:data-[state=off]:[&_~_button]:bg-accent dark:data-[state=off]:[&_~_button]:hover:bg-foreground/10', +]; + +const LK_TOGGLE_VARIANT_2 = [ + 'data-[state=off]:bg-accent data-[state=off]:hover:bg-foreground/10', + 'data-[state=off]:border-border data-[state=off]:hover:border-foreground/12', + 'data-[state=off]:focus-visible:border-ring data-[state=off]:focus-visible:ring-foreground/12', + 'data-[state=off]:text-foreground data-[state=off]:hover:text-foreground data-[state=off]:focus:text-foreground', + 'data-[state=on]:bg-blue-500/20 data-[state=on]:hover:bg-blue-500/30', + 'data-[state=on]:border-blue-700/10 data-[state=on]:text-blue-700 data-[state=on]:ring-blue-700/30', + 'data-[state=on]:focus-visible:border-blue-700/50', + 'dark:data-[state=on]:bg-blue-500/20 dark:data-[state=on]:text-blue-300', +]; + +const MOTION_PROPS: MotionProps = { + variants: { + hidden: { + height: 0, + opacity: 0, + marginBottom: 0, + }, + visible: { + height: 'auto', + opacity: 1, + marginBottom: 12, + }, + }, + initial: 'hidden', + transition: { + duration: 0.3, + ease: 'easeOut', + }, +}; + +interface AgentChatInputProps { + chatOpen: boolean; + onSend?: (message: string) => void; + className?: string; +} + +function AgentChatInput({ chatOpen, onSend = async () => {}, className }: AgentChatInputProps) { + const inputRef = useRef(null); + const [isSending, setIsSending] = useState(false); + const [message, setMessage] = useState(''); + const isDisabled = isSending || message.trim().length === 0; + + const handleSend = async () => { + if (isDisabled) { + return; + } + + try { + setIsSending(true); + await onSend(message.trim()); + setMessage(''); + } catch (error) { + console.error(error); + } finally { + setIsSending(false); + } + }; + + const handleKeyDown = async (e: React.KeyboardEvent) => { + if (e.key === 'Enter' && !e.shiftKey) { + e.preventDefault(); + handleSend(); + } + }; + + const handleButtonClick = async () => { + if (isDisabled) return; + await handleSend(); + }; + + useEffect(() => { + if (chatOpen) return; + // when not disabled refocus on input + inputRef.current?.focus(); + }, [chatOpen]); + + return ( +
+