Building a Real-Time Speech Analytics Pipeline: Transcription, Sentiment Detection, and Conversation Intelligence in Production

The surface area of a speech analytics system looks deceptively simple: audio goes in, insights come out. The production reality is a pipeline with six or seven distinct stages, each with its own latency budget, failure mode, and accuracy tradeoff. You cannot optimize the whole thing until you understand each stage independently.

This article covers the architecture for a system that ingests live audio streams, produces real-time transcripts with speaker labels, runs sentiment and topic analysis at the segment level, and generates conversation intelligence features like talk-to-listen ratio and interruption detection. Everything is sized for concurrent production workloads, not demos.

Pipeline Architecture

The pipeline has five layers, and the order matters for latency budgeting:

Audio Ingestion → Chunking + STT → Diarization → NLP Analysis → Storage + Retrieval

Each layer adds latency. Your total end-to-end latency for a “words on screen” experience is the sum of all five. A reasonable production target for a live coaching product is under 2 seconds from speech to displayed transcript. For an after-call analytics product, latency is less constrained, but accuracy requirements are higher because users will read and act on the output.

Start by writing down your latency budget before you select any vendor or model. The budget determines which tradeoffs you can accept.

Audio Ingestion and Chunking

Telephone audio (8kHz, G.711 mu-law, mono) and browser audio (16kHz or 48kHz, PCM or Opus, stereo) have completely different characteristics. Your ingestion layer needs to normalize before the first STT call.

For WebRTC-sourced audio, the browser’s MediaRecorder API produces Opus-encoded chunks at irregular intervals. For telephony integrations, you are typically receiving RTP streams from a media server (FreeSWITCH, Asterisk, or a cloud provider’s media plane). Normalize both to 16kHz, 16-bit PCM, mono before processing.

interface AudioChunk {
  streamId: string;
  sequenceNumber: number;
  timestampMs: number;
  durationMs: number;
  sampleRate: 16000 | 8000;
  channels: 1 | 2;
  format: "pcm16" | "opus" | "mulaw";
  data: Buffer;
}

interface NormalizedChunk {
  streamId: string;
  sequenceNumber: number;
  timestampMs: number;
  durationMs: number;
  pcm16: Buffer; // always 16kHz, 16-bit, mono
}

async function normalizeChunk(chunk: AudioChunk): Promise<NormalizedChunk> {
  // ffmpeg via child_process, or node-audioworklet for lower-overhead paths
  const pcm16 = await transcodeAudio(chunk.data, {
    inputFormat: chunk.format,
    inputSampleRate: chunk.sampleRate,
    inputChannels: chunk.channels,
    outputSampleRate: 16000,
    outputChannels: 1,
    outputFormat: "pcm16",
  });

  return {
    streamId: chunk.streamId,
    sequenceNumber: chunk.sequenceNumber,
    timestampMs: chunk.timestampMs,
    durationMs: chunk.durationMs,
    pcm16,
  };
}

Chunk size is a latency vs accuracy tradeoff. Streaming APIs accept incremental audio and return partial transcripts, but very small chunks (under 100ms) increase network overhead and cause more partial-word artifacts. Most production systems use 250ms chunks for streaming display and 1-second chunks for diarization-quality output.

STT Provider Selection

The three providers you will evaluate are Deepgram, AssemblyAI, and self-hosted Whisper. They optimize for different things.

Provider	Latency (streaming)	WER (telephone)	Diarization	Cost per hour	Best for
Deepgram Nova-3	~300ms	~5-8%	Yes, built-in	~$0.36	Live coaching, real-time display
AssemblyAI Universal-2	~500-800ms	~4-6%	Yes, built-in	~$0.37	Post-call analytics, higher accuracy
Whisper large-v3 (self-hosted)	2-5s (batch)	~4-7%	No (separate model)	Compute only, ~$0.10-0.20 on GPU	High volume, compliance requirements
Whisper.cpp (streaming)	~500ms with VAD	~5-9%	No	Compute only	On-prem required

For anything requiring real-time display (live call coaching, live captioning), Deepgram’s streaming API is the practical choice because latency is genuinely competitive and diarization is included. AssemblyAI’s real-time endpoint is better for post-call use cases where you replay the audio and care more about word-error rate than display latency.

Self-hosted Whisper is the right choice when you cannot send audio to third-party APIs (compliance, data residency), or when volume crosses the point where cloud STT costs exceed GPU costs. For a 40-hour/day workload, you break even on a dedicated A10G instance at roughly 30-35 days.

Streaming Transcription

Deepgram’s WebSocket API accepts audio bytes and returns JSON events. The key events are Results (partial and final transcript words) and Metadata. Wire up a connection per call session.

import WebSocket from "ws";

interface DeepgramWord {
  word: string;
  start: number;
  end: number;
  confidence: number;
  speaker?: number;
  speaker_confidence?: number;
  punctuated_word: string;
}

interface TranscriptSegment {
  streamId: string;
  speaker: number;
  text: string;
  words: DeepgramWord[];
  startMs: number;
  endMs: number;
  isFinal: boolean;
  confidence: number;
}

class DeepgramStreamingSession {
  private ws: WebSocket;
  private streamId: string;
  private onSegment: (segment: TranscriptSegment) => void;

  constructor(
    streamId: string,
    apiKey: string,
    onSegment: (segment: TranscriptSegment) => void
  ) {
    this.streamId = streamId;
    this.onSegment = onSegment;

    const params = new URLSearchParams({
      model: "nova-3",
      language: "en-US",
      encoding: "linear16",
      sample_rate: "16000",
      channels: "1",
      diarize: "true",
      punctuate: "true",
      smart_format: "true",
      interim_results: "true",
      utterance_end_ms: "1000",
    });

    this.ws = new WebSocket(
      `wss://api.deepgram.com/v1/listen?${params}`,
      { headers: { Authorization: `Token ${apiKey}` } }
    );

    this.ws.on("message", (data) => this.handleMessage(data.toString()));
  }

  private handleMessage(raw: string): void {
    const msg = JSON.parse(raw);

    if (msg.type !== "Results") return;

    const alt = msg.channel?.alternatives?.[0];
    if (!alt || !alt.words?.length) return;

    const words: DeepgramWord[] = alt.words;
    const isFinal = msg.is_final === true;

    // Group words by speaker to produce per-speaker segments
    const speakerGroups = this.groupBySpeaker(words);

    for (const group of speakerGroups) {
      this.onSegment({
        streamId: this.streamId,
        speaker: group.speaker,
        text: group.words.map((w) => w.punctuated_word).join(" "),
        words: group.words,
        startMs: group.words[0].start * 1000,
        endMs: group.words[group.words.length - 1].end * 1000,
        isFinal,
        confidence: alt.confidence,
      });
    }
  }

  private groupBySpeaker(
    words: DeepgramWord[]
  ): Array<{ speaker: number; words: DeepgramWord[] }> {
    const groups: Array<{ speaker: number; words: DeepgramWord[] }> = [];
    let current: { speaker: number; words: DeepgramWord[] } | null = null;

    for (const word of words) {
      const speaker = word.speaker ?? 0;
      if (!current || current.speaker !== speaker) {
        current = { speaker, words: [] };
        groups.push(current);
      }
      current.words.push(word);
    }

    return groups;
  }

  sendAudio(pcm16: Buffer): void {
    if (this.ws.readyState === WebSocket.OPEN) {
      this.ws.send(pcm16);
    }
  }

  close(): void {
    if (this.ws.readyState === WebSocket.OPEN) {
      this.ws.send(JSON.stringify({ type: "CloseStream" }));
    }
  }
}

One thing to get right early: the utterance_end_ms parameter controls when Deepgram emits a final result. Setting it too low (200ms) causes premature finalization on natural speech pauses. Setting it too high (2000ms) delays final segments and makes downstream analysis feel laggy. 1000ms is a reasonable starting point; tune it against recordings of your actual users.

Speaker Diarization

When diarization is built into the STT provider, you get speaker labels on each word. This is sufficient for most use cases. Where it falls apart: calls with more than two speakers, calls where one speaker has a significantly different microphone (cellular vs VoIP), and calls with heavy background noise on one channel.

For telephone calls where you control the media plane, a better architecture is to route each leg to a separate audio stream before mixing. You then know which channel is which participant without depending on diarization at all. This gives you perfect speaker separation at the cost of stream management complexity.

interface CallSession {
  sessionId: string;
  streams: Map<string, { participantId: string; role: "agent" | "customer" }>;
  startedAt: Date;
}

// When you control channel routing, map streams to participants directly
function mapStreamToParticipant(
  session: CallSession,
  streamId: string
): { participantId: string; role: "agent" | "customer" } | null {
  return session.streams.get(streamId) ?? null;
}

When you cannot split channels (browser recording of both sides, phone recording of mixed audio), rely on the provider’s diarization and post-process with a consistency pass: if speaker labels flip back and forth within a 500ms window, it is almost always a diarization error, not an actual interruption.

Sentiment and Emotion Detection

The fundamental architectural decision is whether to run sentiment on transcript segments (text-based) or directly on audio frames (acoustic features). Text-based is cheaper, lower latency, and easier to operate. Audio-based captures prosody (tone, pace, volume) that text loses entirely. Production systems that care about emotion accuracy use both.

For text-based sentiment, a fine-tuned classifier running on final transcript segments is the right layer. Do not send each word to an LLM; that is expensive and adds 300-800ms per segment. Use a dedicated classification model.

interface SentimentResult {
  segmentId: string;
  streamId: string;
  speaker: number;
  sentiment: "positive" | "neutral" | "negative";
  score: number; // -1.0 to 1.0
  emotions: {
    frustration: number;
    satisfaction: number;
    confusion: number;
    urgency: number;
  };
  analyzedAt: Date;
}

async function analyzeSentiment(
  segment: TranscriptSegment,
  classifier: SentimentClassifier
): Promise<SentimentResult> {
  // Only analyze final segments to avoid re-processing partials
  if (!segment.isFinal) {
    throw new Error("Sentiment analysis requires final segments");
  }

  // Skip very short segments (filler words, acknowledgments)
  if (segment.words.length < 4) {
    return neutralSentiment(segment);
  }

  const result = await classifier.classify(segment.text);

  return {
    segmentId: `${segment.streamId}-${segment.startMs}`,
    streamId: segment.streamId,
    speaker: segment.speaker,
    sentiment: result.label,
    score: result.score,
    emotions: result.emotions,
    analyzedAt: new Date(),
  };
}

For acoustic sentiment (detecting frustration from voice, not words), you need audio feature extraction: MFCCs, pitch variance, speaking rate, and energy. This runs on the raw audio track, not the transcript. A dedicated acoustic model (SpeechBrain, Wav2Vec-based classifiers) gives you emotion signals with roughly 200-400ms latency per segment.

The key production decision: run acoustic analysis asynchronously after the call completes rather than in real time unless your product surface actually needs it live. Acoustic model inference is GPU-intensive, and most conversation intelligence features (coaching, QA scoring) do not require sub-second emotion signals.

Topic Extraction and Summarization

Topic extraction belongs at the segment level, not the full-call level. A 30-minute call covers 15-20 distinct topics. Running topic extraction over the full transcript at the end gives you an averaged result that misses the temporal structure.

Use a sliding window over final transcript segments, extracting topics every 30-60 seconds of real conversation time:

interface TopicWindow {
  streamId: string;
  startMs: number;
  endMs: number;
  segments: TranscriptSegment[];
  topics: string[];
  summary: string;
}

class TopicExtractor {
  private buffer: TranscriptSegment[] = [];
  private windowDurationMs = 60_000; // 60 seconds of conversation
  private lastWindowEndMs = 0;

  addSegment(segment: TranscriptSegment): TopicWindow | null {
    if (!segment.isFinal) return null;

    this.buffer.push(segment);

    const bufferDuration =
      segment.endMs - (this.buffer[0]?.startMs ?? segment.startMs);

    if (bufferDuration < this.windowDurationMs) return null;

    const window = this.flushWindow();
    return window;
  }

  private flushWindow(): TopicWindow {
    const segments = [...this.buffer];
    this.buffer = [];

    return {
      streamId: segments[0].streamId,
      startMs: segments[0].startMs,
      endMs: segments[segments.length - 1].endMs,
      segments,
      topics: [], // filled by async extraction
      summary: "", // filled by async summarization
    };
  }
}

For extraction itself, a lightweight classification model (DistilBERT fine-tuned on your domain) outperforms general-purpose LLMs for latency and cost at this granularity. Reserve LLM-based summarization for post-call processing where latency is not a constraint.

Conversation Intelligence Features

These are the features that differentiate a speech analytics product from a transcription service.

Talk-to-listen ratio is straightforward: sum word durations (or segment durations) per speaker over the call window, express as a ratio. Use a rolling 5-minute window rather than the full call so trends are visible.

interface ConversationMetrics {
  sessionId: string;
  windowStartMs: number;
  windowEndMs: number;
  speakerMetrics: Map<
    number,
    {
      totalSpeakingMs: number;
      wordCount: number;
      averageWordPaceWpm: number;
      interruptionCount: number;
    }
  >;
  talkToListenRatio: number; // speaker 0 / speaker 1, for two-party calls
  longestMonologueMs: number;
  questionsAsked: number;
}

function computeTalkToListenRatio(
  segments: TranscriptSegment[],
  agentSpeaker: number
): number {
  let agentMs = 0;
  let customerMs = 0;

  for (const seg of segments) {
    if (!seg.isFinal) continue;
    const durationMs = seg.endMs - seg.startMs;
    if (seg.speaker === agentSpeaker) {
      agentMs += durationMs;
    } else {
      customerMs += durationMs;
    }
  }

  if (customerMs === 0) return 1.0;
  return agentMs / customerMs;
}

Interruption detection requires comparing the start timestamp of a new speaker segment against the end timestamp of the previous segment from a different speaker. If a new speaker starts while the previous speaker is still within their utterance end threshold, it is an interruption.

function detectInterruptions(
  segments: TranscriptSegment[],
  overlapThresholdMs = 200
): Array<{ at: number; interruptingSpeaker: number; interruptedSpeaker: number }> {
  const interruptions = [];
  const finalSegments = segments.filter((s) => s.isFinal);

  for (let i = 1; i < finalSegments.length; i++) {
    const prev = finalSegments[i - 1];
    const curr = finalSegments[i];

    if (curr.speaker === prev.speaker) continue;

    const overlap = prev.endMs - curr.startMs;
    if (overlap > overlapThresholdMs) {
      interruptions.push({
        at: curr.startMs,
        interruptingSpeaker: curr.speaker,
        interruptedSpeaker: prev.speaker,
      });
    }
  }

  return interruptions;
}

Keyword spotting runs on each final segment text using simple string matching for known terms, plus optional fuzzy matching for phonetically similar words (important when STT transcribes “cancel” as “counsel”):

interface KeywordMatch {
  keyword: string;
  matchedText: string;
  segmentId: string;
  speaker: number;
  atMs: number;
  confidence: number;
}

function spotKeywords(
  segment: TranscriptSegment,
  keywordGroups: Map<string, string[]> // groupName -> variants
): KeywordMatch[] {
  const text = segment.text.toLowerCase();
  const matches: KeywordMatch[] = [];

  for (const [group, variants] of keywordGroups) {
    for (const variant of variants) {
      if (text.includes(variant.toLowerCase())) {
        matches.push({
          keyword: group,
          matchedText: variant,
          segmentId: `${segment.streamId}-${segment.startMs}`,
          speaker: segment.speaker,
          atMs: segment.startMs,
          confidence: segment.confidence,
        });
      }
    }
  }

  return matches;
}

Storage and Retrieval Architecture

Transcripts need two different access patterns: real-time writes during the call, and full-text search and filtering after the call. These require different storage choices.

For real-time writes, append transcript segments to a time-series store or a streaming log (Kafka, Kinesis) per session. The write path must not block the pipeline; anything that adds more than 10ms of synchronous latency will degrade the user experience during live calls.

For search and retrieval, post-call jobs index the final merged transcript into a search store. Elasticsearch and OpenSearch both handle full-text search over transcripts well. For semantic search (“find all calls where the customer mentioned billing confusion”), add vector embeddings of 30-second transcript windows alongside the keyword index.

interface StoredTranscript {
  sessionId: string;
  organizationId: string;
  startedAt: Date;
  endedAt: Date;
  durationMs: number;
  participants: Array<{
    participantId: string;
    role: "agent" | "customer";
    speakerIndex: number;
  }>;
  segments: TranscriptSegment[];
  metrics: ConversationMetrics;
  sentiment: SentimentResult[];
  topics: TopicWindow[];
  keywords: KeywordMatch[];
  summary: string;
  fullText: string; // denormalized for keyword search
  embeddingVector?: number[]; // for semantic search
}

Retention architecture matters. Raw audio is expensive to store at scale. Most products keep raw audio for 30-90 days for replay, but store transcripts and metadata indefinitely. Design your schema so analytics queries never need to touch the audio store.

Handling Audio Quality Variance

Audio quality variance is the largest source of accuracy degradation in production. Common failure modes:

• Telephone audio over VoIP with codec re-encoding (G.711 resampled to G.729, then back) introduces artifacts that increase WER by 15-25%
• Background noise (call centers, open offices) causes false word insertions and deletion errors
• Non-native English speakers with strong accents can push WER to 20-30% on general models
• Double-talk (both speakers audible) causes diarization errors that cascade through downstream analysis

The practical mitigations for each:

Codec artifacts: normalize to 16kHz before STT; do not re-encode twice. If you receive pre-mixed audio from a phone provider, request lossless export if available.

Background noise: apply a voice activity detector (WebRTC VAD or Silero VAD) to filter non-speech frames before sending to STT. This also reduces STT API costs because you are not paying to transcribe silence and noise.

Accented speech: Deepgram’s Nova-3 and AssemblyAI’s Universal-2 both have significantly better accent coverage than Whisper base/small models. If accuracy on accented speech matters, benchmark against your actual call recordings before committing to a provider.

Double-talk: implement a simple energy-based detection layer to flag segments where both channels have simultaneous high energy. Log these for human review rather than passing them to automated analysis.

Cost Modeling

At scale, STT is your dominant cost. A call center processing 10,000 hours per month at Deepgram’s $0.36/hr rate is $3,600/month for transcription alone.

Additional costs per hour of audio:

• Sentiment analysis (hosted model): ~$0.02-0.05
• LLM summarization (post-call, GPT-4o-mini): ~$0.05-0.15
• Vector embedding storage (1 embedding per 30s): ~$0.001
• Full-text search indexing: ~$0.01-0.02

Total production cost is typically $0.45-0.60 per hour of call audio for a full analytics pipeline. At 10,000 hours/month, that is $4,500-6,000/month in AI/infrastructure costs, before your own compute.

Two cost levers that matter in practice: (1) only run LLM-based summarization on calls that meet a minimum duration threshold (5+ minutes), since short calls rarely produce useful summaries; and (2) use voice activity detection to skip silence billing — most calls have 20-30% silence that you do not need to pay to transcribe.

Scaling Concurrent Streams

Each active call is a stateful WebSocket connection and a set of in-memory buffers. The session manager is the component that becomes the scalability bottleneck.

Design session state to be externalized from the start. Keep the WebSocket connection in a stateless gateway, persist session state to Redis, and route events through a message queue to processing workers:

interface SessionState {
  sessionId: string;
  status: "active" | "completed" | "error";
  streamIds: string[];
  segmentCount: number;
  lastActivityAt: number; // epoch ms for TTL management
  metrics: Partial<ConversationMetrics>;
}

// Session state in Redis with 4-hour TTL
async function getOrCreateSession(
  sessionId: string,
  redis: Redis
): Promise<SessionState> {
  const existing = await redis.get(`session:${sessionId}`);
  if (existing) return JSON.parse(existing);

  const session: SessionState = {
    sessionId,
    status: "active",
    streamIds: [],
    segmentCount: 0,
    lastActivityAt: Date.now(),
    metrics: {},
  };

  await redis.setex(`session:${sessionId}`, 14400, JSON.stringify(session));
  return session;
}

At 1,000 concurrent calls, each with two audio streams at 16kHz 16-bit, you are handling roughly 64 MB/s of audio data into the pipeline before any processing. Plan your ingestion capacity accordingly, and set up per-session circuit breakers so a bad call (audio loop, continuous noise) cannot consume disproportionate processing resources.

Accuracy vs Latency Tradeoffs

Pipeline Stage	Low-Latency Approach	High-Accuracy Approach	Latency Delta
Transcription	Streaming partials, 250ms chunks	Batch after utterance, 1s+ chunks	+700-1500ms
Diarization	Provider built-in, concurrent with STT	Separate post-processing model	+1-3s
Sentiment	Lightweight classifier, final segments only	LLM-based with context window	+200-800ms
Topic extraction	Sliding window classifier	Full-call LLM extraction	+1-5s
Summarization	Streaming LLM	Full-call LLM with retries	+2-10s

The practical rule: use streaming approaches for anything displayed during the call, and high-accuracy approaches for anything displayed after. Your product’s UX requirements determine the budget, not the other way around.

A live coaching product needs streaming transcription and real-time sentiment. A QA scoring product can afford to wait 60 seconds after call end to run its full analysis. Do not build one pipeline that tries to serve both; the latency constraints are incompatible, and you end up with a system that is too slow for live use and too inaccurate for QA.

Production Considerations

Resumability: calls drop. Your pipeline must resume from the last acknowledged sequence number without re-processing already-indexed segments. Build sequence number tracking from the start; retrofitting it is painful.

Compliance and data handling: audio recordings are sensitive data in most jurisdictions. PII redaction in transcripts (credit card numbers, SSNs, health information) should run as a post-processing step before any segment is written to a searchable index. Log the redaction separately from the transcript so you can audit what was removed.

Monitoring: instrument pipeline stage latency, not just end-to-end. When tail latency spikes, you need to know whether it is the STT connection, the sentiment classifier, or the storage write. Export p50/p95/p99 per stage to your metrics system.

Graceful degradation: if the sentiment classifier is down, emit the transcript without sentiment scores. Do not fail the entire call because an optional enrichment service is unavailable. Each stage should have a defined behavior on timeout or failure, with an explicit fallback output type.

The speech analytics pipeline is a pipeline in the original Unix sense: each component does one thing, passes its output to the next, and can be replaced or upgraded independently. The complexity is in the wiring, the latency budgeting, and the operational behavior under degraded conditions. Build the stages cleanly, and the system stays maintainable as your accuracy requirements evolve.