@valora-ai/voice

Auto-generated from JSDoc in packages/voice/src/mod.ts. Do not edit — run npm run docs:api.

createLocalRealtimeSession

function

function createLocalRealtimeSession(agent: VoiceAgent): LocalRealtimeSession

createVoiceAgent

function

function createVoiceAgent(opts: VoiceAgentOptions): VoiceAgent

alwaysEndOfTurn

variable

const alwaysEndOfTurn: TurnDetector

heuristicTurnDetector

variable

const heuristicTurnDetector: TurnDetector

Heuristic: not end-of-turn if the transcript trails off mid-clause.

Loadable

interface

interface Loadable

A loadable engine: the uniform create(repo?, opts?) factory the STT/LLM/TTS adapters share. E is the engine interface produced (TranscriptionModel, LanguageModel, SpeechModel). Two-plus adapters satisfy each — the seam is real, not hypothetical.

`create(repo?: string, opts?: LoadOpts): Promise<E>`

LocalRealtimeSession

interface

interface LocalRealtimeSession

`connect(): Promise<void>`

`disconnect(): Promise<void>`

`startAudioCapture(): Promise<void>`

`stopAudioCapture(): Promise<void>`

`sendText(text: string): void`

`interrupt(): void`

`mute(muted: boolean): void`

`unlock(): void`

`subscribe(onChange: () => void): () => void`

`getSnapshot(): LocalRealtimeSnapshot`

LocalRealtimeSnapshot

interface

interface LocalRealtimeSnapshot

`status: LocalRealtimeStatus`

`messages: Segment[]`

PlayerEngine

interface

interface PlayerEngine

Audio playback is an engine too — keeps the core free of AudioContext.

`play(pcm: Float32Array, sampleRate: number): Promise<void>`

Resolve when the clip finishes (or is stopped).

`stop(): void`

`resume(): void | Promise<void>`

Optional: unlock/resume output (e.g. a suspended AudioContext).

`level(): number`

Optional: instantaneous output amplitude 0–1 (RMS of what's currently playing). Lets the UI react to the agent's OWN voice while speaking (vs the mic level).

Segment

interface

interface Segment

A streamed transcript segment (cf. ReceivedTranscriptionSegment).

`id: string`

`text: string`

`final: boolean`

`at: number`

`role: user | agent`

TurnDetector

interface

interface TurnDetector

Decides whether a transcribed utterance ends the user's turn, or whether they're mid-thought and we should keep listening (the "semantic turn detection" idea). Default is always-true (pure VAD-silence turn-taking). Swap in a model-backed implementation for fewer mid-pause interruptions.

`isEndOfTurn(transcript: string): boolean | Promise<boolean>`

`holdMs(transcript: string): number | undefined`

Optional adaptive post-utterance hold: given the buffered (not-end-of-turn) transcript, how many milliseconds to wait for more speech before committing it anyway. Pure function, no side effects. Return undefined to defer to the caller's own default wait (the agent's maxTurnWaitMs). Detectors that omit this behave exactly as before — a fixed wait.

VoiceAction

interface

interface VoiceAction

`id: string`

`description?: string`

`match(text: string): boolean | Promise<boolean>`

`execute(ctx: VoiceActionContext): VoiceActionResult | Promise<VoiceActionResult>`

VoiceActionContext

interface

interface VoiceActionContext

`text: string`

`pendingText: string`

VoiceActionResult

interface

interface VoiceActionResult

`handled: boolean`

`reply?: string`

`data?: unknown`

VoiceAgent

interface

interface VoiceAgent

`state: VoiceState`

`subscribe(onChange: () => void): () => void`

useSyncExternalStore-compatible reactive snapshot — the sole notification surface.

`getSnapshot(): VoiceSnapshot`

`start(): Promise<void>`

`stop(): Promise<void>`

`dispose(): Promise<void>`

`interrupt(): void`

`mute(muted: boolean): void`

`sendText(text: string): void`

Inject a typed user turn — runs the same lifecycle as a spoken one (mixed modality).

`speak(text: string): Promise<void>`

Make the agent speak arbitrary text now, outside a turn. Resolves when playback ends.

`unlock(): void`

Unlock audio output — call from a user gesture (resumes a suspended AudioContext).

`reset(): void`

Abandon any turn, clear the transcript + metrics + buffered text; keep engines/state idle.

VoiceAgentOptions

interface

interface VoiceAgentOptions

`systemPrompt?: string`

System prompt for the LLM's replies. Default preserves the current voice assistant behavior.

`reasoning?: boolean`

Reasoning mode (D2): append an instruction asking the model to think briefly inside a <think></think> block before its spoken answer. When on, deliver() strips that block per turn (fresh think-filter instance, no cross-turn state) before it reaches the transcript or the speaker — the user only ever sees/hears the visible answer. llmFirstTokenMs still stamps on the first raw token, so reasoning latency stays visible in metrics. Default false (off — zero behavior change: tokens pass through exactly as before, so a model that emits <think> unprompted is not accidentally filtered).

`listenThreshold?: number`

`speakThreshold?: number`

`bargeInGraceMs?: number`

`minInterruptionMs?: number`

While the agent is speaking, user speech must persist this long before it interrupts — filters echo/noise blips (cf. LiveKit min_interruption_duration). 0 = immediate. Default 350ms. Barge-in during "thinking" stays immediate (no audio playing).

`bargeIn?: immediate | semantic`

Barge-in filtering strategy while speaking. 'immediate' (default) abandons the reply as soon as persistent user speech is detected (past bargeInGraceMs + minInterruptionMs) — exact current behavior. 'semantic' adds a short backchannel probe first: a pure backchannel ("yeah", "mm-hm") does NOT interrupt; real content does, same as today. Opt-in because the probe adds decision latency to genuine interruptions. Degrades to 'immediate' when no streamingStt is configured (the probe needs live interim transcription). Also inert when minInterruptionMs <= 0: that setting means "barge in on the first frame of speech", which skips the persistence window the semantic probe hangs off.

`speculativePrefill?: boolean`

Speculative prefill (A2): when the interim transcript holds still for a beat during listening, prime the LLM's prefix KV-cache with the prompt the committed reply would send, so the real reply's prefill costs only the divergent tail. Needs streamingStt (the trigger is interim segments) and an llm. No behavioral surface — replies just start faster; the prime records nothing and its errors are swallowed. Default true.

`maxSegments?: number`

`maxTurnWaitMs?: number`

Commit a buffered (mid-thought) turn after this much silence regardless of the detector — bounds the turn-detection wait. 0/Infinity disables. Default 1200ms.

`actions?: VoiceAction[]`

`onEvent?: (event: VoiceEvent) => void`

`onError?: (e: Error) => void`

Error callback. Defaults to console.error — never silent. Errors also land in snapshot.lastError either way.

`now?: () => number`

Clock used for turn latency metrics. Default Date.now — override for deterministic tests.

`firstChunkWords?: number | false`

Word budget for the first TTS chunk of each turn — flush at the first clause/sentence boundary or this many words, whichever comes first (felt-latency shortcut). false disables it (full-sentence-only, the pre-B1 behavior). Default 8. See Speaker in speaking/speaker.ts.

`fillers?: { phrases?: string[]; delayMs?: number } | false`

Pre-synthesized filler bank ("Hmm.", "Let me see.") played when thinking runs long — perceived-latency cover for slow LLM/TTS. Cancelled the instant real reply audio starts. false/omitted (default) disables the feature entirely — zero behavior change. See createFillerBank in speaking/filler.ts.

VoiceEngines

interface

interface VoiceEngines

`vad: VADEngine`

`stt: TranscriptionModel`

`llm?: LanguageModel`

`tts?: SpeechModel`

`player?: PlayerEngine`

`turnDetector?: TurnDetector`

`streamingStt?: StreamingSTT`

VoiceError

interface

interface VoiceError

The last error the agent hit, kept as state so UIs can show it (errors were previously callback-only and invisible unless the app wired onError).

`message: string`

`at: number`

VoiceMetrics

interface

interface VoiceMetrics

Per-turn latency metrics (ms). Useful as an SLA gate in integration tests.

`firstAudioMs: number | null`

Utterance-end → first TTS audio playing (the latency users feel).

`lastTurnMs: number | null`

Utterance-end → back to idle (whole turn).

`commitMs: number | null`

Final user transcript ready → turn commit decision (includes TurnDetector wait).

`sttFinalizeMs: number | null`

VAD speech-end → final user transcript ready.

`llmFirstTokenMs: number | null`

Turn commit → first LLM token.

`ttsFirstChunkMs: number | null`

First complete sentence available → first synthesized audio handed to the player.

VoiceSnapshot

interface

interface VoiceSnapshot

Reactive snapshot — always-current value for framework binding (useSyncExternalStore).

type LocalRealtimeStatus = disconnected | connecting | connected

VADEngine

type alias

type VADEngine = VadModel

VoiceActionEvent

type alias

type VoiceActionEvent = { type: action-start; at: number; id: string; text: string } | { type: action-end; at: number; id: string; handled: boolean; data?: unknown } | { type: action-error; at: number; id: string; error: Error }

VoiceEvent

type alias

type VoiceEvent = { type: state; at: number; state: VoiceState; previousState: VoiceState } | { type: speech-start; at: number } | { type: speech-end; at: number } | { type: turn-start; at: number; token: number } | { type: turn-end; at: number; token: number; metrics: VoiceMetrics } | { type: segment; at: number; segment: Segment } | { type: first-audio; at: number; token: number; metrics: VoiceMetrics } | { type: barge-in; at: number } | { type: interrupt; at: number } | { type: error; at: number; error: Error } | { type: no-speech; at: number } | { type: speech-discarded; at: number; reason: echo-guard | too-short } | VoiceActionEvent

VoiceState

type alias

type VoiceState = loading | idle | listening | thinking | speaking

The single state enum that drives the whole UI (the agent state).

@valora-ai/voice

On this page