Streaming LLM Responses in Production: Server-Sent Events, Backpressure, and Edge Delivery

Most teams bolt on LLM streaming as an afterthought. The model supports it, the SDK exposes it, they wire it up in an afternoon, and it works fine in dev. Then production arrives: clients on mobile networks, edge nodes in front of origin, multiple concurrent streams per user, and the occasional model timeout mid-sentence. The cracks appear fast.

This article covers the full picture: protocol choice, backpressure mechanics, token buffering, edge delivery, error recovery, and what streaming actually costs you compared to batching.

Why Streaming Matters at All

Latency perception is non-linear. A user waiting 4 seconds for a complete response perceives it as slow. The same user seeing the first tokens within 300ms and watching the rest arrive feels like the system is fast, even if total time-to-complete is longer.

For LLMs specifically, time-to-first-token (TTFT) is the metric that drives UX. The model starts generating immediately but takes seconds to finish. Streaming lets you hand tokens to the client as they come off the model, rather than waiting for the full completion.

That said, streaming is not free. It adds protocol complexity, makes error handling harder, and creates real infrastructure challenges at scale.

SSE vs WebSocket: The Right Tool

Both Server-Sent Events and WebSockets can carry streaming LLM output. They are not equivalent.

Server-Sent Events are unidirectional HTTP: the server pushes events, the client reads them. They use a plain text protocol over a persistent HTTP connection. Browsers handle reconnection automatically. You get this for free in any HTTP/1.1 or HTTP/2 stack.

WebSockets are bidirectional, full-duplex. They require an upgrade handshake and their own framing protocol. They work well when the client needs to interrupt the stream, send follow-up messages, or maintain a long-lived session with back-and-forth.

For LLM chat, SSE wins in most cases. The server-to-client flow is one-directional per turn. The client sends a message, the server streams a response. The next user message starts a new HTTP request. There is no need for full-duplex, and the operational simplicity of SSE is worth keeping.

Factor	SSE	WebSocket
Protocol	HTTP (text/event-stream)	Custom framing over TCP
Direction	Server to client	Bidirectional
Reconnect	Automatic (Last-Event-ID)	Manual
Proxy/CDN support	Excellent	Varies, often blocked
Load balancer sticky sessions	Not required	Required for stateful sessions
Interrupting a stream	Requires separate POST	Native (send message over same socket)
Streaming LLM per-turn	Good fit	Overkill unless multi-turn real-time

The one case where WebSockets make sense: voice or real-time collaborative interfaces where the client sends audio chunks or edits while the model is still responding. For standard chat, use SSE.

Implementing SSE Streaming in TypeScript

Here is a minimal but production-shaped Node.js/Edge handler that streams OpenAI-compatible responses to the client:

import { OpenAI } from "openai";

const client = new OpenAI();

export async function handleStream(req: Request): Promise<Response> {
  const { messages } = await req.json();

  const stream = await client.chat.completions.create({
    model: "gpt-4o",
    messages,
    stream: true,
  });

  const encoder = new TextEncoder();

  const readable = new ReadableStream({
    async start(controller) {
      try {
        for await (const chunk of stream) {
          const delta = chunk.choices[0]?.delta?.content;
          if (delta) {
            // SSE format: "data: <payload>\n\n"
            const payload = JSON.stringify({ token: delta });
            controller.enqueue(encoder.encode(`data: ${payload}\n\n`));
          }

          if (chunk.choices[0]?.finish_reason) {
            controller.enqueue(encoder.encode(`data: [DONE]\n\n`));
            controller.close();
          }
        }
      } catch (err) {
        // Encode an error event before closing
        const errorPayload = JSON.stringify({ error: "stream_error" });
        controller.enqueue(encoder.encode(`event: error\ndata: ${errorPayload}\n\n`));
        controller.close();
      }
    },
  });

  return new Response(readable, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
      "Connection": "keep-alive",
      "X-Accel-Buffering": "no", // Disable nginx buffering
    },
  });
}

Two headers that commonly get overlooked: X-Accel-Buffering: no tells nginx not to buffer the response before forwarding it. Without this, nginx will wait for the full response and your streaming is silently negated. Cache-Control: no-cache prevents CDN layers from caching event-stream responses (which would break streaming entirely).

On the client side, the EventSource API handles this cleanly:

const source = new EventSource("/api/chat", { withCredentials: true });

source.onmessage = (event) => {
  if (event.data === "[DONE]") {
    source.close();
    return;
  }
  const { token } = JSON.parse(event.data);
  appendToken(token); // Your UI update function
};

source.addEventListener("error", (event) => {
  console.error("Stream error", event);
  source.close();
  showErrorState();
});

EventSource does not support POST requests natively. For chat interfaces that need to send a message body, the common patterns are: encode the request in query params (fine for short inputs), use a fetch-based SSE reader instead of EventSource, or create a session ID with a POST first and then open a GET-based event stream.

// fetch-based SSE reader for POST requests
async function streamChat(messages: Message[]) {
  const response = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
  });

  const reader = response.body!.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const chunk = decoder.decode(value, { stream: true });
    // Parse SSE lines from chunk
    for (const line of chunk.split("\n")) {
      if (line.startsWith("data: ")) {
        const data = line.slice(6);
        if (data === "[DONE]") return;
        const { token } = JSON.parse(data);
        appendToken(token);
      }
    }
  }
}

Backpressure: When the Model Outpaces the Client

Backpressure is what happens when a producer generates data faster than a consumer can process it. With LLM streaming, the model can generate tokens faster than a slow client (mobile on 3G, congested network) can receive them.

The naive implementation just buffers everything in memory. This works until you have 500 concurrent streams and your Node process is holding gigabytes of buffered tokens for slow clients.

The Web Streams API (ReadableStream / WritableStream) has backpressure built in via its queuing strategy. When the internal queue fills, the stream signals the producer to slow down. But this only works if you respect the signal.

const readable = new ReadableStream(
  {
    async start(controller) {
      for await (const chunk of modelStream) {
        const delta = chunk.choices[0]?.delta?.content;
        if (delta) {
          // desiredSize goes negative when consumer is slow
          if (controller.desiredSize !== null && controller.desiredSize <= 0) {
            // Client is behind. You can:
            // 1. Pause reading from the model (if the SDK supports it)
            // 2. Continue buffering but track total buffered size
            // 3. Drop the connection if buffer exceeds threshold
          }
          controller.enqueue(encoder.encode(`data: ${JSON.stringify({ token: delta })}\n\n`));
        }
      }
      controller.close();
    },
  },
  new CountQueuingStrategy({ highWaterMark: 10 }) // 10 chunks before backpressure kicks in
);

In practice, you cannot pause OpenAI’s API mid-stream. Once you open the stream, tokens come. The real levers are:

1. Set a per-connection buffer cap. If buffered bytes exceed (say) 128KB, drop the connection with a 503.
2. Track slow clients in aggregate. If one user has consistently slow streams, consider falling back to polling for them.
3. Use a queue system. Write tokens to Redis Streams or a similar append log, let the client poll at its own rate. This decouples model generation speed from client consumption speed completely, at the cost of infrastructure complexity.

Token Buffering Strategies

Not every token needs to be sent immediately. Buffering a few tokens before flushing can reduce overhead for fast connections and improve perceived smoothness.

class TokenBuffer {
  private buffer: string[] = [];
  private flushTimer: ReturnType<typeof setTimeout> | null = null;
  private readonly maxSize: number;
  private readonly maxDelayMs: number;
  private readonly onFlush: (tokens: string) => void;

  constructor(options: {
    maxSize: number;
    maxDelayMs: number;
    onFlush: (tokens: string) => void;
  }) {
    this.maxSize = options.maxSize;
    this.maxDelayMs = options.maxDelayMs;
    this.onFlush = options.onFlush;
  }

  push(token: string) {
    this.buffer.push(token);
    if (this.buffer.length >= this.maxSize) {
      this.flush();
    } else if (!this.flushTimer) {
      this.flushTimer = setTimeout(() => this.flush(), this.maxDelayMs);
    }
  }

  flush() {
    if (this.flushTimer) {
      clearTimeout(this.flushTimer);
      this.flushTimer = null;
    }
    if (this.buffer.length > 0) {
      this.onFlush(this.buffer.join(""));
      this.buffer = [];
    }
  }
}

Reasonable defaults: buffer up to 5 tokens or flush every 50ms. This smooths out the token-by-token jitter without introducing noticeable delay. For mobile clients, consider a slightly larger buffer (10 tokens, 100ms) to reduce the number of tiny renders.

Edge Delivery via Cloudflare Workers

Running LLM streaming close to users via Cloudflare Workers requires understanding how Workers handle streaming. Workers use the Web Streams API natively, which is why the ReadableStream pattern above maps directly.

The main consideration at the edge: Workers have a CPU time limit (50ms default on the free tier, up to 30 seconds on paid). Streaming responses that take 10-20 seconds to complete tie up the Worker’s CPU budget. This is manageable, but you need to structure the handler correctly.

Workers are also subject to a 128MB memory limit. A stream that buffers aggressively will hit this. Keep the buffer small and flush eagerly.

// Cloudflare Worker handler
export default {
  async fetch(request: Request): Promise<Response> {
    if (request.method !== "POST") {
      return new Response("Method not allowed", { status: 405 });
    }

    const { messages } = await request.json<{ messages: Message[] }>();

    // Call your origin or directly call the model API
    const upstreamResponse = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${(globalThis as unknown as Env).OPENAI_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ model: "gpt-4o", messages, stream: true }),
    });

    if (!upstreamResponse.ok || !upstreamResponse.body) {
      return new Response("Upstream error", { status: 502 });
    }

    // Pass the stream through with CORS and no-buffer headers
    return new Response(upstreamResponse.body, {
      headers: {
        "Content-Type": "text/event-stream",
        "Cache-Control": "no-cache",
        "Access-Control-Allow-Origin": "*",
        "X-Accel-Buffering": "no",
      },
    });
  },
};

Workers act as a passthrough here: they handle auth, CORS, and rate limiting at the edge, then proxy the stream from origin. This keeps latency low (the Worker is in the same region as the user) while keeping model calls centralized.

One issue specific to Cloudflare: by default, Cloudflare caches and potentially buffers responses. For SSE, you must either disable caching with a Cache Rule, or ensure your Cache-Control: no-cache header is present (Workers respect this).

Error Handling Mid-Stream

Errors that occur after the stream has started are the hardest part. You have already sent a 200 OK and started delivering data. You cannot change the status code. Your options for signaling errors to the client are limited to in-band signals.

The SSE event type mechanism handles this cleanly:

// Server side: emit a typed error event
function encodeErrorEvent(code: string, message: string): Uint8Array {
  const payload = JSON.stringify({ code, message });
  return encoder.encode(`event: error\ndata: ${payload}\n\n`);
}

// Common error scenarios mid-stream:
// 1. Model API timeout
// 2. Rate limit hit partway through
// 3. Content filter triggered
// 4. Network issue to upstream

async function* streamWithErrorHandling(modelStream: AsyncIterable<Chunk>) {
  try {
    for await (const chunk of modelStream) {
      yield chunk;
    }
  } catch (err) {
    if (err instanceof OpenAI.APIError) {
      if (err.status === 429) {
        yield { type: "error", code: "rate_limit", message: "Rate limit reached" };
      } else if (err.status === 408 || err.message.includes("timeout")) {
        yield { type: "error", code: "timeout", message: "Model timeout" };
      } else {
        yield { type: "error", code: "upstream_error", message: "Model error" };
      }
    } else {
      yield { type: "error", code: "internal", message: "Internal error" };
    }
  }
}

On the client, listen for the typed error event and handle it specifically:

source.addEventListener("error", (event: MessageEvent) => {
  const { code, message } = JSON.parse(event.data);
  if (code === "rate_limit") {
    showRetryAfterUI(message);
  } else if (code === "timeout") {
    offerResendOption();
  } else {
    showGenericError();
  }
  source.close();
});

One edge case worth handling explicitly: partial responses. If the stream errors halfway through a sentence, you often want to show what arrived rather than blank the screen. Keep appended tokens in state, and only clear on a fresh request.

Streaming vs Batching: Cost Implications

Streaming does not change your token costs. You pay the same number of input and output tokens regardless of whether you stream or batch.

What changes is infrastructure cost and latency:

Factor	Streaming	Batching
Token cost	Same	Same
Connection duration	Long (seconds per request)	Short (full response once complete)
Concurrent connections	Higher (slow clients hold connections)	Lower
Server memory per request	Potentially higher (buffering)	Lower (fire and forget)
TTFT (user experience)	Excellent	Poor for long responses
Retry complexity	High (partial state to manage)	Low (retry is idempotent)
Works with aggressive CDN caching	No	Yes

The infrastructure cost difference depends on your hosting model. On serverless (Lambda, Workers), you pay per duration. A 10-second streaming response costs 10x a 1-second batched response in compute time. On dedicated servers, streaming increases concurrent connection count, which affects memory and file descriptor limits.

For most user-facing LLM applications, streaming’s UX benefit justifies the infrastructure cost. The exception: batch processing pipelines (generating reports, processing documents) where no human is watching a cursor. For those, batching is simpler and cheaper.

Production Checklist

A few things that are consistently missed in production streaming deployments:

Timeouts at every layer. Your LLM API client needs a timeout. Your load balancer needs an idle timeout longer than your typical stream duration (set it to at least 120 seconds). Your CDN’s origin timeout needs to match. Mismatched timeouts cause silent truncation.

Drain on shutdown. When you deploy a new version, active streams are on inflight connections. A hard process kill will truncate them mid-sentence. Implement graceful shutdown: stop accepting new connections, let active streams finish (with a max grace period of, say, 30 seconds), then exit.

Stream deduplication for retries. If a client retries (network blip, refresh), you can end up with two active streams for the same request. Store a request ID server-side, and cancel the previous stream when a duplicate arrives.

Observability. Log TTFT, total stream duration, token count, and whether the stream completed or errored. These four metrics will tell you most of what you need to know when debugging production incidents.

Streaming LLM responses is one of those features that seems simple from the SDK surface and reveals real complexity in production. The protocol choice matters, the buffering strategy matters, and the error handling mid-stream requires deliberate design. Getting these right is the difference between a system that feels fast and one that occasionally leaves users staring at a truncated sentence.