Building a Voice AI Pipeline: Speech-to-Text, LLM Processing, and Text-to-Speech in Production

Voice AI pipelines are deceptively hard to build well. The happy path works fine in a demo: user speaks, your STT transcribes it, LLM responds, TTS reads it back. But production is a different story. You are composing three latency-sensitive external services over a real-time audio channel, and each one can fail, spike, or drift in ways that make the conversation feel broken. The user abandons within two or three awkward pauses.

This article covers the full architecture: audio capture and streaming, STT provider selection, LLM processing with streaming output, TTS synthesis, WebSocket delivery, latency optimization techniques, mid-conversation error handling, and cost modeling. All code examples are in TypeScript.

---

The Architecture Overview

Before diving into each layer, it helps to name the full signal path:

[Client microphone]
    -> WebSocket (PCM audio chunks)
    -> Voice Activity Detection (VAD)
    -> STT provider (Deepgram / Whisper / AssemblyAI)
    -> LLM (streaming tokens)
    -> Sentence boundary detection
    -> TTS provider (ElevenLabs / OpenAI TTS / Cartesia)
    -> WebSocket (MP3/Opus audio chunks)
    -> [Client speaker]

Every arrow in that chain is a latency budget. Target end-to-end latency (from end of user speech to first audio byte from the assistant) of under 1.2 seconds for a natural conversation feel. Above 2 seconds, users start to fill the silence. Above 3 seconds, they hang up or reload.

The two biggest levers are: (1) streaming at every stage rather than waiting for complete outputs, and (2) starting TTS before the LLM has finished generating.

---

Audio Capture and Streaming

On the client side, use the Web Audio API to capture microphone input and stream raw PCM to your WebSocket server. Avoid buffering entire utterances before sending: you want the STT to start working as soon as speech begins.

// client.ts
async function startVoiceStream(ws: WebSocket): Promise<void> {
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
  const audioContext = new AudioContext({ sampleRate: 16000 });
  const source = audioContext.createMediaStreamSource(stream);
  const processor = audioContext.createScriptProcessor(4096, 1, 1);

  processor.onaudioprocess = (event) => {
    const inputData = event.inputBuffer.getChannelData(0);
    // Convert Float32 to Int16 PCM for bandwidth efficiency
    const pcm16 = new Int16Array(inputData.length);
    for (let i = 0; i < inputData.length; i++) {
      pcm16[i] = Math.max(-32768, Math.min(32767, inputData[i] * 32768));
    }
    if (ws.readyState === WebSocket.OPEN) {
      ws.send(pcm16.buffer);
    }
  };

  source.connect(processor);
  processor.connect(audioContext.destination);
}

Use 16kHz mono PCM. That is the native format most STT APIs expect, and it cuts bandwidth in half compared to 44.1kHz stereo. For the WebSocket server, use ws or uWebSockets.js for Node.js. Avoid HTTP/REST round-trips for audio: the per-request overhead adds 100-200ms per chunk.

---

Voice Activity Detection

Do not send every audio frame to your STT provider. Most calls contain silence, background noise, and breath sounds. VAD gates the STT pipeline, reducing cost and preventing false transcriptions.

A practical VAD setup uses Silero VAD running locally in the browser via ONNX.js, or on the server via @ricky0123/vad-node. The key parameters:

// server-vad.ts
import { Vad } from "@ricky0123/vad-node";

const vad = await Vad.new({
  positiveSpeechThreshold: 0.6,   // confidence threshold to mark frame as speech
  negativeSpeechThreshold: 0.35,  // threshold to end a speech segment
  preSpeechPadFrames: 5,          // frames to include before speech detected
  redemptionFrames: 8,            // frames to wait before declaring speech ended
  frameSamples: 1536,
});

// Yields: { type: "speech_start" | "speech_end" | "vad_misfire" }

The preSpeechPadFrames setting matters: without it, you clip the beginning of the word. The redemptionFrames setting prevents cutting off sentences mid-thought when a speaker pauses briefly between clauses.

Server-side VAD adds about 5-10ms of latency per frame on modern hardware, which is acceptable. Client-side VAD (in the browser) saves the round-trip cost of sending silent frames to your server entirely.

---

Speech-to-Text: Provider Selection

The three meaningful choices for production are Deepgram, OpenAI Whisper (via API or self-hosted), and AssemblyAI. They differ on latency, accuracy, pricing, and streaming support.

Deepgram

Deepgram’s streaming WebSocket API is the fastest option for real-time voice. It returns interim results as speech is happening, which lets you start LLM processing before the utterance is complete (more on speculative processing below).

// stt-deepgram.ts
import { createClient, LiveTranscriptionEvents } from "@deepgram/sdk";

const deepgram = createClient(process.env.DEEPGRAM_API_KEY!);

export function createDeepgramStream(onFinal: (text: string) => void) {
  const live = deepgram.listen.live({
    model: "nova-2",
    language: "en-US",
    smart_format: true,
    interim_results: true,
    utterance_end_ms: 1000,
    vad_events: true,
    encoding: "linear16",
    sample_rate: 16000,
  });

  live.on(LiveTranscriptionEvents.Transcript, (data) => {
    const transcript = data.channel.alternatives[0].transcript;
    if (data.is_final && transcript.length > 0) {
      onFinal(transcript);
    }
  });

  return live;
}

The utterance_end_ms: 1000 setting tells Deepgram to finalize the transcript after 1 second of silence. Lower values (500ms) feel more responsive but produce more fragmented transcripts. 800-1000ms is the practical sweet spot.

OpenAI Whisper

The Whisper API (whisper-1) is file-based, not streaming. You buffer the entire utterance, send it as an MP3 or WAV file, and wait for the full transcription. This adds 500ms-1500ms of latency compared to streaming STT. The tradeoff: Whisper has excellent accuracy on accented English and technical vocabulary, and it costs roughly the same as Deepgram.

Self-hosting Whisper via faster-whisper on a GPU instance changes the equation: you get streaming via chunked inference and sub-200ms latency on utterances under 10 seconds, with no per-minute cost. The operational complexity is real though: model loading, GPU memory management, and batching logic are all on you.

AssemblyAI

AssemblyAI offers streaming via their LeMUR API and has strong punctuation and speaker diarization. It is the right choice when you need speaker identification (multi-party calls, interview recording) or when you need built-in content moderation or PII redaction in the transcript. For single-speaker conversational voice AI, it is slightly more expensive than Deepgram for comparable accuracy.

Provider Comparison

Dimension	Deepgram Nova-2	OpenAI Whisper API	AssemblyAI Streaming
Streaming	Yes (WebSocket)	No (file-based)	Yes (WebSocket)
Time to first word	~300ms	~800ms-1500ms	~400ms
Accuracy (EN)	Excellent	Best-in-class	Excellent
Accuracy (accents)	Good	Best-in-class	Good
Technical vocabulary	Good	Excellent	Good
Speaker diarization	Basic	No	Excellent
Pricing (per minute)	~$0.0043	~$0.006	~$0.0065
Self-hosting option	No	Yes (faster-whisper)	No
Best for	Low-latency conversational AI	Accuracy-critical transcription	Multi-speaker or compliance use cases

---

LLM Processing with Streaming

Once you have a final transcript, you need an LLM response. The critical requirement here is streaming: never wait for the complete response before starting TTS. You want TTS processing the first sentence while the LLM is still generating the rest.

// llm-stream.ts
import OpenAI from "openai";

const openai = new OpenAI();

export async function* streamLLMResponse(
  transcript: string,
  history: Array<{ role: "user" | "assistant"; content: string }>
): AsyncGenerator<string> {
  const stream = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      {
        role: "system",
        content:
          "You are a voice assistant. Keep responses concise (2-3 sentences max). " +
          "Avoid markdown, bullet points, and special characters. Write as you would speak.",
      },
      ...history,
      { role: "user", content: transcript },
    ],
    stream: true,
    max_tokens: 200,
  });

  for await (const chunk of stream) {
    const token = chunk.choices[0]?.delta?.content ?? "";
    if (token) yield token;
  }
}

The system prompt instruction to avoid markdown is load-bearing. TTS providers convert markdown characters literally: “bold” becomes “asterisk asterisk bold asterisk asterisk.” Train your system prompt to produce clean spoken prose.

Keep max_tokens tight for voice. A 500-token response takes 15-20 seconds to read aloud. Users will interrupt before you finish. 150-250 tokens is the right ceiling for most conversational turns.

---

Sentence Boundary Detection for TTS Triggering

The core latency optimization is triggering TTS as soon as you have a complete sentence, rather than waiting for the full LLM response. This requires detecting sentence boundaries in the streaming token output.

// sentence-splitter.ts
export class SentenceBuffer {
  private buffer = "";
  private readonly sentenceEnders = /[.!?]\s/;

  push(token: string): string[] {
    this.buffer += token;
    const sentences: string[] = [];

    let match: RegExpExecArray | null;
    while ((match = this.sentenceEnders.exec(this.buffer)) !== null) {
      const endIndex = match.index + match[0].length;
      sentences.push(this.buffer.slice(0, endIndex).trim());
      this.buffer = this.buffer.slice(endIndex);
    }

    return sentences;
  }

  flush(): string {
    const remaining = this.buffer.trim();
    this.buffer = "";
    return remaining;
  }
}

Each time push() returns a non-empty array, you dispatch those sentences to TTS immediately. By the time the LLM finishes generating, the first two or three sentences are already synthesized and playing.

This approach introduces a tradeoff: if the LLM generates a very long first sentence, you wait longer for the first audio. Prompt engineering matters here: instruct the model to open with short declarative sentences.

---

Text-to-Speech: Provider Selection

The three realistic options for production voice AI are ElevenLabs, OpenAI TTS, and Cartesia.

ElevenLabs

ElevenLabs has the highest voice quality and the best streaming API via their WebSocket endpoint. The eleven_turbo_v2_5 model targets sub-300ms time-to-first-audio-byte on their infrastructure, which is the metric that matters.

// tts-elevenlabs.ts
export async function* streamElevenLabsAudio(
  text: string,
  voiceId: string
): AsyncGenerator<Buffer> {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/text-to-speech/${voiceId}/stream`,
    {
      method: "POST",
      headers: {
        "xi-api-key": process.env.ELEVENLABS_API_KEY!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        text,
        model_id: "eleven_turbo_v2_5",
        voice_settings: {
          stability: 0.5,
          similarity_boost: 0.75,
          style: 0.0,
          use_speaker_boost: true,
        },
        output_format: "mp3_44100_128",
      }),
    }
  );

  if (!response.ok || !response.body) {
    throw new Error(`ElevenLabs error: ${response.status}`);
  }

  const reader = response.body.getReader();
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    yield Buffer.from(value);
  }
}

OpenAI TTS

OpenAI’s TTS (tts-1 and tts-1-hd) is meaningfully cheaper than ElevenLabs and returns a complete audio file rather than a true stream. You get the full MP3 back, typically in 500-900ms for short sentences. For voice AI, this makes it a batched TTS rather than a streaming one, which hurts the latency budget. The tts-1-hd model adds another 200-400ms in exchange for better audio quality.

The right use case for OpenAI TTS is low-latency responses where voice naturalness is secondary (IVR menus, structured prompts), or batch audio generation (podcast intro generation, video narration) where streaming is not the goal.

Cartesia

Cartesia’s Sonic model is worth serious evaluation for production: they claim sub-90ms time-to-first-audio-byte on their streaming API, which is faster than ElevenLabs at a lower cost per character. The voice library is smaller, and the emotional range is narrower, but for a single consistent assistant voice in a product, it competes well.

// tts-cartesia.ts
import Cartesia from "@cartesia/cartesia-js";

const cartesia = new Cartesia({ apiKey: process.env.CARTESIA_API_KEY! });

export async function* streamCartesiaAudio(
  text: string,
  voiceId: string
): AsyncGenerator<Buffer> {
  const websocket = cartesia.tts.websocket({
    container: "raw",
    encoding: "pcm_f32le",
    sampleRate: 44100,
  });

  await websocket.connect();

  const response = await websocket.send({
    model_id: "sonic-english",
    voice: { mode: "id", id: voiceId },
    transcript: text,
  });

  for await (const chunk of response) {
    if (chunk.type === "chunk" && chunk.data) {
      yield chunk.data;
    }
  }
}

TTS Provider Comparison

Dimension	ElevenLabs Turbo v2.5	OpenAI TTS-1	Cartesia Sonic
Streaming	Yes (HTTP streaming)	No (full file)	Yes (WebSocket)
Time to first audio	~300ms	~500-900ms	~90ms
Voice quality	Excellent	Good	Good
Voice variety	Very large	6 voices	Growing library
Emotional range	Excellent	Limited	Moderate
Pricing per 1K chars	~$0.18 (Starter)	~$0.015	~$0.065
Best for	Production voice AI, clone voices	Batch generation, low-cost projects	Ultra-low latency, cost-sensitive streaming

ElevenLabs costs roughly 12x more than OpenAI TTS and 3x more than Cartesia. For a conversational AI product with high turn volume, that difference is material. At 10,000 conversational turns per day (average 150 characters per TTS call), the monthly cost difference between ElevenLabs and Cartesia is approximately $2,700/month.

---

WebSocket Orchestration

The server orchestrates all three services over a single WebSocket connection with the client. Audio flows in, audio flows out, and state tracks the current conversation turn.

// voice-server.ts
import { WebSocketServer, WebSocket } from "ws";
import { createDeepgramStream } from "./stt-deepgram";
import { streamLLMResponse } from "./llm-stream";
import { SentenceBuffer } from "./sentence-splitter";
import { streamElevenLabsAudio } from "./tts-elevenlabs";

const VOICE_ID = "21m00Tcm4TlvDq8ikWAM"; // Rachel

const wss = new WebSocketServer({ port: 8080 });

wss.on("connection", (ws: WebSocket) => {
  const history: Array<{ role: "user" | "assistant"; content: string }> = [];
  let isAssistantSpeaking = false;

  const deepgramLive = createDeepgramStream(async (transcript) => {
    if (isAssistantSpeaking) {
      // User interrupted. Stop current TTS, restart pipeline.
      isAssistantSpeaking = false;
      ws.send(JSON.stringify({ type: "interrupt" }));
    }

    history.push({ role: "user", content: transcript });
    isAssistantSpeaking = true;

    const sentenceBuffer = new SentenceBuffer();
    let fullResponse = "";

    try {
      for await (const token of streamLLMResponse(transcript, history)) {
        fullResponse += token;
        const sentences = sentenceBuffer.push(token);

        for (const sentence of sentences) {
          if (!isAssistantSpeaking) break; // Interrupted
          for await (const audioChunk of streamElevenLabsAudio(sentence, VOICE_ID)) {
            if (!isAssistantSpeaking) break;
            ws.send(audioChunk);
          }
        }
      }

      // Flush remaining buffer after LLM finishes
      const remaining = sentenceBuffer.flush();
      if (remaining && isAssistantSpeaking) {
        for await (const audioChunk of streamElevenLabsAudio(remaining, VOICE_ID)) {
          if (!isAssistantSpeaking) break;
          ws.send(audioChunk);
        }
      }

      history.push({ role: "assistant", content: fullResponse });
    } catch (err) {
      ws.send(JSON.stringify({ type: "error", message: "Processing failed" }));
    } finally {
      isAssistantSpeaking = false;
      ws.send(JSON.stringify({ type: "turn_end" }));
    }
  });

  ws.on("message", (data: Buffer) => {
    deepgramLive.send(data);
  });

  ws.on("close", () => {
    deepgramLive.finish();
  });
});

The interrupt handling pattern is important. When the user speaks while the assistant is responding, you need to: stop sending audio chunks to the client, signal the client to stop playback, and restart the pipeline with the new input. This is the difference between a real conversational AI and a phone IVR system.

---

Error Handling Mid-Conversation

Three failure modes matter in production:

STT provider timeout. Deepgram’s WebSocket can disconnect under load. Reconnect automatically with exponential backoff and preserve the audio buffer so no speech is lost. Add a timeout: if no transcript arrives within 5 seconds of speech detection, restart the connection and notify the user with a brief “I missed that” message.

LLM API errors. Rate limits, timeouts, and context length overflows all happen. For rate limits, back off and retry with a synthetic “one moment” audio response generated from a pre-cached TTS clip. For context length overflows, truncate the oldest turns from history rather than failing the call.

TTS synthesis failure. If a TTS request fails mid-sentence, you have partial audio playing. The cleanest recovery is to re-synthesize the full pending response from a fallback provider (OpenAI TTS is a reasonable fallback for ElevenLabs) and insert a brief pause to cover the gap. Pre-cache a “let me think” audio clip in your fallback voice for graceful degradation.

// error-recovery.ts
export async function synthesizeWithFallback(
  text: string,
  voiceId: string
): Promise<AsyncGenerator<Buffer>> {
  try {
    return streamElevenLabsAudio(text, voiceId);
  } catch (primaryErr) {
    console.error("Primary TTS failed, falling back to OpenAI TTS", primaryErr);
    return streamOpenAITTS(text); // always returns, never throws
  }
}

---

Cost Modeling

At scale, the cost structure is dominated by TTS, not STT or LLM. Here is a rough model for 10,000 daily conversational turns at 30 seconds average turn duration:

Component	Volume	Unit cost	Monthly cost
STT (Deepgram Nova-2)	5,000 min/day	$0.0043/min	~$645
LLM (GPT-4o, 500 tokens avg)	10K req/day	~$0.005/req	~$1,500
TTS (ElevenLabs, 200 chars avg)	10K req/day	$0.18/1K chars	~$1,080
TTS (Cartesia, same volume)	10K req/day	$0.065/1K chars	~$390
WebSocket server (8 vCPU, 32GB)	1 instance	$400/mo	$400
Total (ElevenLabs path)			~$3,625/mo
Total (Cartesia path)			~$2,935/mo

The LLM is the second-largest cost. Switching from GPT-4o to Claude Haiku or Gemini Flash for conversational turns (where quality differences are small) cuts that line by 80%. The combination of Cartesia TTS and a fast small LLM brings the 10K daily turn cost to roughly $1,200/month, compared to $3,600 with ElevenLabs and GPT-4o.

---

Production Considerations

Observability. Instrument every stage with latency histograms: time from VAD speech-end to STT final result, STT result to LLM first token, LLM first token to TTS first audio byte, and total turn latency. Alert on p99 exceeding 3 seconds. The breakdown immediately shows you which provider is degraded.

History truncation. Conversation history grows unbounded in long sessions. Truncate to the last N turns, but always preserve the system prompt. A rolling window of 10 turns is sufficient for coherent conversation context in most use cases.

Audio format consistency. Negotiate audio format at WebSocket connection time. The client may support Opus (better compression, lower bandwidth) or need MP3 (broader compatibility). Store the negotiated format in session state and do not switch mid-conversation.

Rate limiting per session. Malicious or looping clients can exhaust your STT/LLM/TTS quotas in minutes. Apply per-session rate limits: maximum 2 turns per second, maximum 60 turns per session.

Silence handling. If the user says nothing for 15-20 seconds, send a brief prompt (“Are you still there?”). If silence continues, close the WebSocket cleanly. Open connections with no activity cost server resources and keep Deepgram connections open unnecessarily.

---

Closing

The latency budget for voice AI is unforgiving compared to text. The architecture decisions that matter most: streaming at every stage, sentence-boundary TTS triggering, server-side VAD, and interrupt handling. Get those right and the pipeline feels like a real conversation. Get them wrong and you have a slow, expensive IVR. The provider choices matter less than the plumbing between them.