Skip to content

AGENTS.md

Latest commit

 

History

History
232 lines (163 loc) · 14.9 KB

AGENTS.md

File metadata and controls

232 lines (163 loc) · 14.9 KB
Raw

AGENTS.md

Project Overview

LiveKit Agents for Node.js — a TypeScript framework for building realtime, multimodal, and voice AI agents that run on servers. This is the Node.js distribution of the LiveKit Agents framework (originally Python).

Monorepo Structure

  • agents/ — Core framework (@livekit/agents). Agent orchestration, LLM/STT/TTS abstractions, voice pipeline, metrics, telemetry, IPC/process pooling, and the CLI.
  • plugins/ — Provider plugins (@livekit/agents-plugin-*). Each implements one or more of: LLM, STT, TTS, VAD, EOU (end-of-utterance), Realtime, or Avatar.
  • examples/ — Example agents (private, not published). Run with pnpm build && node ./examples/src/<file>.ts dev.
  • tests/e2e/ — End-to-end tests via Docker (separate from unit tests co-located in each package).

Tooling: pnpm 11 workspaces, Turborepo for builds, tsup for bundling (CJS + ESM), TypeScript 5.4+, Vitest for tests, Changesets for versioning.

Common Commands

pnpm build                  # Build all packages (turbo)
pnpm build:agents           # Build only @livekit/agents and its deps
pnpm build:plugins          # Build only plugins and their deps
pnpm clean:build            # Clean dist/ dirs then rebuild
pnpm test                   # Run all tests (vitest)
pnpm test agents/src/llm            # Run tests by path
pnpm test -t "chat context"         # Run tests by name
pnpm test:watch             # Watch mode
pnpm lint                   # ESLint all packages
pnpm lint:fix               # ESLint with auto-fix
pnpm format:check           # Prettier check
pnpm format:write           # Prettier format
pnpm api:check              # API Extractor validation
pnpm api:update             # Update API declarations

Running an example agent

pnpm build && node ./examples/src/basic_agent.ts dev --log-level=debug

Required env vars: LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET, plus provider keys (e.g. OPENAI_API_KEY).

Rule (must follow before handing off to the user to run any example/agent file): Always run pnpm build first and iterate on TypeScript / dependency errors until it exits 0. Never tell the user to node ./examples/src/<file>.ts (or any .ts entrypoint) until the build succeeds — this includes after editing example files, after modifying framework code that breaks tsc, and after copy-pasting reproductions from external sources. If the build fails, fix the errors yourself (add missing deps, fix outdated API usage, update types) before asking the user to run anything.

Pre-downloading plugin assets (for Docker builds)

@livekit/agents ships a livekit-agents bin with a download-files subcommand that discovers installed @livekit/agents-plugin-* packages from node_modules and runs each plugin's downloadFiles() (e.g. HuggingFace models for the turn detector) — without loading the user's agent code. Use it in Dockerfiles to separate dependency installation from app code so the download layer can be cached:

COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
RUN npx livekit-agents download-files
COPY src/ ./src
RUN pnpm build

The download-files subcommand on cli.runApp() is deprecated for this purpose since it requires loading the agent file first.

Debugging individual plugins

Create a test file prefixed with test_ in examples/src/. No defineAgent wrapper needed — just import the plugin directly and run:

pnpm build && node ./examples/src/test_my_plugin.ts

Architecture

Voice Pipeline (agents/src/voice/)

The core of the framework. Pipeline flow: Audio In → VAD → STT → LLM → TTS → Audio Out.

Key classes and their roles:

  • Agent — Base class holding instructions, tools, and model config. Subclass to override pipeline hooks (ttsNode, realtimeAudioOutputNode, sttNode, llmNode). Use voice.Agent.default.<hook>(this, ...) to call the base implementation from overrides.
  • AgentTask — Extends Agent for composable task workflows with isolated chat context and tool scoping.
  • AgentSession — Orchestrates the full session lifecycle: connects to LiveKit room, manages turn detection, handles interruptions, collects metrics. Entry point: session.start({ agent, room }).
  • AgentActivity (agent_activity.ts, ~100KB) — Complex state machine managing individual turns: VAD trigger → STT → endpointing → LLM inference → TTS generation → playout. Supports preemptive generation (starts LLM while user still speaking).
  • SpeechHandle — Represents a unit of agent speech with lifecycle tracking and priority levels (LOW=0, NORMAL=5, HIGH=10). Sources: 'say', 'generate_reply', 'tool_response'.
  • AvatarSession — Base class for avatar plugin sessions. Registers aclose() as a job shutdown callback and warns when started after AgentSession.start() (which would replace the existing audio output). Plugin implementations extend this class and call super.start(agentSession, room) first.

Turn detection modes: "stt" | "vad" | "realtime_llm" | "manual" (configured in turn_config/).

Interruption detection: "adaptive" (ML-based via inference/interruption/) or "vad" mode.

Subdirectories: room_io/ (LiveKit Room I/O), transcription/ (word-level sync), avatar/ (avatar integration), recorder_io/ (recording), testing/ (test utilities), turn_config/ (turn/interruption/endpointing config).

LLM (agents/src/llm/)

  • ChatContext — Chronologically ordered conversation state. Supports action-aware history summarization via _summarize(llm, { keepLastTurns }) to compress old messages while preserving tool execution results.
  • Tool system — llm.tool() with Zod schemas for parameters. handoff() for multi-agent transfers.
  • FallbackAdapter — Agent-side multi-provider LLM failover with availability tracking.
  • Provider format adapters — provider_format/openai.ts, provider_format/google.ts, and provider_format/mistralai.ts for model-agnostic code.

Other Core Modules

  • STT (stt/): SpeechStream with automatic retry. FallbackAdapter for agent-side multi-provider failover. StreamAdapter converts non-streaming STT + VAD to streaming.
  • TTS (tts/): SynthesizeStream, ChunkedStream. FallbackAdapter for agent-side multi-provider failover. StreamAdapter for non-streaming providers.
  • VAD (vad.ts): Voice Activity Detection interface. Silero plugin is the primary implementation.
  • Inference (inference/): LiveKit Inference Gateway clients (LLM, STT, TTS, realtime) with server-side fallback support. Always use full provider/model format (e.g., 'openai/gpt-4o-mini'). Also includes interruption/ for adaptive interruption detection via ML models.
  • Stream (stream/): Composable Web Streams API primitives (StreamChannel, DeferredStream, MultiInputStream).
  • IPC (ipc/): Process pool for running agents in child processes. Two-way IPC: child sends inference requests back to parent.
  • Worker (worker.ts): Main process connecting to LiveKit server, receives job assignments, spawns agent processes.
  • Telemetry (telemetry/): OpenTelemetry tracing with custom span attributes (TTFT, TTFB, interruption probability, speech IDs, provider request IDs). Pino transport for structured logging.
  • Metrics (metrics/): AgentMetrics union type covering LLM, STT, TTS, VAD, EOU, Realtime, and Interruption metrics. ModelUsageCollector for aggregating per-provider usage.
  • Beta (beta/): TaskGroup for multi-task orchestration with optional chat context summarization between tasks.

Remote Sessions (voice/remote_session.ts)

Wire protocol for distributed agents. SessionTransport abstraction with RoomSessionTransport (LiveKit room channels) and TcpSessionTransport (raw TCP socket) implementations.

Plugins (plugins/)

Each extends Plugin base class, auto-registers on import via Plugin.registerPlugin(). Pattern: @livekit/agents-plugin-<provider>.

Plugin capabilities by type:

  • LLM: openai, google, baseten, mistralai
  • STT: deepgram (v1+v2), openai, baseten, sarvam (v1/v2/v3), mistralai, inworld, cartesia
  • TTS: cartesia, elevenlabs, deepgram, openai, neuphonic, resemble, rime, inworld, baseten, sarvam (v1/v2/v3), mistralai, fishaudio, hume, speechify
  • VAD: silero (ONNX-based, local)
  • EOU/Turn Detection: livekit (HuggingFace + ONNX)
  • Realtime: openai (+ responses/, ws/ modules), google, xai, phonic
  • Avatar: trugen, lemonslice, bey, anam, liveavatar, did, protoface
  • Test mocks: test (private, for unit tests)

AsyncLocalStorage Patterns

The framework uses Node.js AsyncLocalStorage for implicit context passing:

  • agentActivityStorage — Access current AgentActivity in callbacks
  • functionCallStorage — Access current FunctionCall in tool handlers
  • speechHandleStorage — Access current SpeechHandle

Code Conventions

  • Lifecycle changes: Follow the lifecycle review checklist for session helpers, shutdown, and cleanup. Apply it to Python ports too.
  • License header required on every new file: 
    // SPDX-FileCopyrightText: 2026 LiveKit, Inc.
//
// SPDX-License-Identifier: Apache-2.0
  • Prettier: single quotes, trailing commas, 100 char width, sorted imports (@trivago/prettier-plugin-sort-imports).
  • ESLint: @typescript-eslint with strict rules. Prefix unused vars with _. Use type imports (consistent-type-imports).
  • TypeScript: strict mode, noUncheckedIndexedAccess, verbatimModuleSyntax, target ES2022, module node16.
  • Time units: Use milliseconds for all time-based values by default. Only use seconds when the name explicitly ends with InS.
  • Changesets: All packages in agents/ and plugins/ release together (fixed versioning). Run pnpm changeset to add a changeset before PRing. The examples package is ignored.
  • API Extractor: Public API surface is tracked. Run pnpm api:check after changing exports and pnpm api:update to update declarations.

Testing

  • Framework: Vitest with 5s default timeout.
  • Pattern: *.test.ts files co-located with source.
  • Snapshots: Used in LLM chat/tool context tests (agents/src/llm/__snapshots__/).
  • Inference LLM tests: Always use full model names from agents/src/inference/llm.ts (e.g. 'openai/gpt-4o-mini', not 'gpt-4o-mini'). Initialize logger first: initializeLogger({ pretty: true }).
  • Test plugin: @livekit/agents-plugins-test provides mock LLM, STT, TTS for unit tests without external APIs.
  • STT testing utilities: stt.testing.FakeSTT (from @livekit/agents) provides a configurable test harness for unit testing STT infrastructure (e.g. FallbackAdapter) with scripted transcripts, exceptions, timeouts, and observability channels.
  • PR validation for major changes: Verify restaurant_agent.ts and realtime_agent.ts work properly in Agent Playground.

Porting from Python (livekit-agents)

When porting features or fixes from the Python livekit-agents repo to this JS/TS repo, follow these rules:

Time unit unification

Python uses seconds (float) for all time values. JS/TS uses milliseconds (number) by default.

When porting a Python time parameter:

  • Multiply the Python default by 1000 for the JS default (e.g. 3.0 s → 3000 ms)
  • Use setTimeout / clearTimeout directly with the ms value — do not multiply by 1000 at call sites
  • Name the field in plain form (e.g. aecWarmupDuration, userAwayTimeout) — the ms convention is implied
  • Only use seconds as the unit if the variable name explicitly ends with InS (e.g. delayInS)

Example mapping:

Python JS/TS
aec_warmup_duration: float = 3.0 aecWarmupDuration: number | null = 3000
user_away_timeout: float = 15.0 userAwayTimeout: number | null = 15000
loop.call_later(self._aec_warmup_remaining, cb) setTimeout(cb, this._aecWarmupRemaining)

CI Requirements

  • REUSE/SPDX license compliance
  • ESLint passes
  • Prettier formatting passes
  • Full build succeeds
  • Base branch: main

LiveKit documentation

Always refer to the latest documentation when working with this project. LiveKit offers both a CLI (lk docs) and an MCP server for browsing and searching its documentation.

LiveKit CLI

Install or update the CLI:

  • macOS: brew install livekit-cli (update: brew update && brew upgrade livekit-cli)
  • Linux: curl -sSL https://get.livekit.io/cli | bash
  • Windows: winget install LiveKit.LiveKitCLI (update: winget upgrade LiveKit.LiveKitCLI)

Run lk docs before your first docs lookup. Its help text explains how to use each command and how to interpret results.

LiveKit docs MCP server

For deeper coding agent integration, LiveKit also offers an MCP server.

Submit feedback to LiveKit

When you run into any problem with LiveKit, add it to your to-do list, then submit it with lk docs submit-feedback or the MCP submit_docs_feedback tool once you finish your other tasks. Useful feedback includes incomplete documentation, broken examples, missing steps, broken features or implementations, and any other gap or issue in the LiveKit ecosystem.

LiveKit agent skills

LiveKit publishes agent skills for each stage of the work: reading the docs, building, debugging, testing, writing scenarios, running simulations, and operating in production. They defer to the live documentation for API details. The developer installs them with lk skills install, which also adds the LiveKit Docs MCP server to your config, or with the LiveKit Claude Code plugin. If no LiveKit skills are available to you (such as reading-livekit-docs), recommend that the developer install them. Check the skills you have, not only the .agents/skills/ and .claude/skills/ directories, since the plugin and lk skills install --global install them elsewhere. Don't recommend a second install method on top of one the developer already uses: installing the same skills twice, for example as the plugin and with lk skills install, loads both copies, so every skill competes with itself. If your tool doesn't load skills automatically, read the matching .agents/skills/<name>/SKILL.md before you start that kind of task.