LLM Application Architecture Patterns: Request-Response, Streaming Agents, and Background Processing in TypeScript

The most common mistake in early LLM application design is picking the wrong delivery model. A team building an internal document classifier ships it as a synchronous REST endpoint, then scrambles when requests start timing out at 30 seconds. Another team wires a batch data extraction job through a streaming chat interface because that is what they had working. A third builds a customer-facing assistant that blocks the HTTP response for 45 seconds and wonders why retention is low.

The underlying model call is often not the problem. The architecture around it is.

LLM applications fit cleanly into three architectural patterns: synchronous request-response, streaming agents, and background processing pipelines. Each maps to a specific class of user experience, latency budget, and operational characteristic. Mixing them up produces systems that are hard to operate and unpleasant to use.

This article covers what each pattern looks like in TypeScript, when to reach for each one, where they fail, and how real systems combine them.

Pattern 1: Synchronous Request-Response

What it is

The client sends a request and waits for the complete response before doing anything else. Standard HTTP request-response semantics. The model call happens inline, the result is returned, the connection closes.

This is appropriate when:

• The operation completes in under 10 seconds reliably
• You need the result before proceeding (a classification, an entity extraction, a short answer)
• You do not need intermediate output visible to the user
• The caller is a backend service, not a browser waiting for a response

The basic structure

import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";

const client = new Anthropic();

const ClassificationSchema = z.object({
  category: z.enum(["billing", "technical", "account", "other"]),
  confidence: z.number().min(0).max(1),
  reasoning: z.string(),
});

type Classification = z.infer<typeof ClassificationSchema>;

async function classifyTicket(text: string): Promise<Classification> {
  const response = await client.messages.create({
    model: "claude-opus-4-6",
    max_tokens: 256,
    system: `You are a support ticket classifier. Respond with a JSON object with keys:
- category: one of "billing", "technical", "account", or "other"
- confidence: a number from 0 to 1
- reasoning: a one-sentence explanation

Respond with JSON only. No markdown, no prose.`,
    messages: [{ role: "user", content: text }],
  });

  const content = response.content[0];
  if (content.type !== "text") {
    throw new Error("Unexpected response type");
  }

  const parsed = JSON.parse(content.text);
  return ClassificationSchema.parse(parsed);
}

The critical decisions here: the max_tokens cap (256 is more than enough for structured classification output, and uncapped requests burn tokens and add latency), the strict JSON instruction in the system prompt, and Zod validation on the response before you trust it.

Timeouts and retries

Synchronous calls need hard timeouts. If your p99 latency is 8 seconds and you set a 30-second timeout, you will eventually queue up enough slow requests to exhaust your connection pool.

async function classifyWithRetry(
  text: string,
  options: { timeoutMs: number; maxRetries: number } = {
    timeoutMs: 15_000,
    maxRetries: 2,
  }
): Promise<Classification> {
  let lastError: unknown;

  for (let attempt = 0; attempt <= options.maxRetries; attempt++) {
    const controller = new AbortController();
    const timeoutId = setTimeout(
      () => controller.abort(),
      options.timeoutMs
    );

    try {
      const response = await client.messages.create(
        {
          model: "claude-opus-4-6",
          max_tokens: 256,
          system: CLASSIFY_SYSTEM_PROMPT,
          messages: [{ role: "user", content: text }],
        },
        { signal: controller.signal }
      );

      clearTimeout(timeoutId);
      return parseClassification(response);
    } catch (err) {
      clearTimeout(timeoutId);

      if (err instanceof Anthropic.APIStatusError && err.status === 429) {
        // Rate limited: back off before retrying
        const backoffMs = Math.pow(2, attempt) * 1000;
        await new Promise((r) => setTimeout(r, backoffMs));
        lastError = err;
        continue;
      }

      if (err instanceof DOMException && err.name === "AbortError") {
        lastError = new Error(`LLM call timed out after ${options.timeoutMs}ms`);
        // Timeout: retry immediately on a different connection
        continue;
      }

      // Non-retryable error (4xx other than 429): surface immediately
      throw err;
    }
  }

  throw lastError;
}

The retry logic distinguishes between rate-limit errors (back off exponentially) and timeouts (retry immediately). A 400 Bad Request is not retryable, do not loop on it.

Where synchronous breaks down

Synchronous works until it does not. The failure modes are predictable:

• P99 latency climbs as the input gets longer or the model gets busier
• Gateway timeouts at 30 seconds are hard limits you cannot negotiate around in most infrastructure
• Concurrency is bounded by thread pool or connection pool size, and LLM calls hold those connections for their full duration

When the operation regularly exceeds 10-15 seconds, or when it involves multiple chained model calls, synchronous request-response is the wrong pattern.

Pattern 2: Streaming Agents

What it is

The server starts generating output immediately and sends it to the client as it is produced. The connection stays open for the duration of the generation. For interactive applications, this is the difference between a good and a bad user experience: a user reading tokens as they arrive tolerates 30 seconds of generation much better than a user staring at a spinner for the same duration.

Streaming is appropriate when:

• A human is watching the output in real time
• The operation is interactive (chat, copilot assistance, document drafting)
• The total latency is too long for synchronous but the user needs immediate feedback
• You want to enable tool-use or multi-step reasoning while keeping the user in the loop

Server-side streaming with tool use

The more complex streaming pattern involves tool-use: the model emits tool call events mid-stream, you execute the tool, and you continue the stream with the result. This is the foundation of interactive agents.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

interface Tool {
  name: string;
  description: string;
  input_schema: object;
  execute: (input: unknown) => Promise<unknown>;
}

async function* runStreamingAgent(
  userMessage: string,
  tools: Tool[],
  maxSteps = 8
): AsyncGenerator<{ type: "text_delta"; delta: string } | { type: "done" } | { type: "error"; message: string }> {
  const messages: Anthropic.MessageParam[] = [
    { role: "user", content: userMessage },
  ];

  const toolDefs = tools.map(({ execute: _execute, ...def }) => def);

  for (let step = 0; step < maxSteps; step++) {
    try {
      const stream = await client.messages.stream({
        model: "claude-opus-4-6",
        max_tokens: 4096,
        tools: toolDefs as Anthropic.Tool[],
        messages,
      });

      let accumulatedText = "";
      const toolCalls: Anthropic.ToolUseBlock[] = [];

      for await (const event of stream) {
        if (
          event.type === "content_block_delta" &&
          event.delta.type === "text_delta"
        ) {
          accumulatedText += event.delta.text;
          yield { type: "text_delta", delta: event.delta.text };
        }

        if (event.type === "content_block_stop") {
          // Capture completed tool call blocks
          const block = await stream.finalMessage().then(
            (msg) => msg.content[event.index]
          );
          if (block?.type === "tool_use") {
            toolCalls.push(block);
          }
        }
      }

      const finalMessage = await stream.finalMessage();

      if (finalMessage.stop_reason === "end_turn") {
        yield { type: "done" };
        return;
      }

      if (finalMessage.stop_reason === "tool_use" && toolCalls.length > 0) {
        // Add assistant turn with tool calls
        messages.push({ role: "assistant", content: finalMessage.content });

        // Execute tools in parallel
        const toolResults = await Promise.allSettled(
          toolCalls.map(async (call) => {
            const tool = tools.find((t) => t.name === call.name);
            if (!tool) {
              return {
                type: "tool_result" as const,
                tool_use_id: call.id,
                content: `Error: Tool "${call.name}" not found.`,
              };
            }

            try {
              const result = await tool.execute(call.input);
              return {
                type: "tool_result" as const,
                tool_use_id: call.id,
                content: JSON.stringify(result),
              };
            } catch (err) {
              return {
                type: "tool_result" as const,
                tool_use_id: call.id,
                content: `Error: ${String(err)}`,
              };
            }
          })
        );

        const resultContent = toolResults.map((r) =>
          r.status === "fulfilled"
            ? r.value
            : {
                type: "tool_result" as const,
                tool_use_id: "unknown",
                content: "Tool execution failed.",
              }
        );

        messages.push({ role: "user", content: resultContent });
        // Continue the loop for the next generation step
        continue;
      }

      yield { type: "done" };
      return;
    } catch (err) {
      yield { type: "error", message: String(err) };
      return;
    }
  }

  yield { type: "error", message: `Agent exceeded ${maxSteps} steps` };
}

The generator yields text deltas as they arrive, allowing callers to pipe them directly to the client. Tool execution happens between steps, invisible to the model’s generation but visible to the user if you yield status events alongside text deltas.

Wiring streaming to HTTP

In a Node.js HTTP server or an edge runtime, the generator maps to a readable stream:

// Hono or similar framework handler
app.post("/api/chat", async (c) => {
  const { message } = await c.req.json<{ message: string }>();

  const encoder = new TextEncoder();

  const readable = new ReadableStream({
    async start(controller) {
      const agent = runStreamingAgent(message, getRegisteredTools());

      for await (const event of agent) {
        if (event.type === "text_delta") {
          // Server-sent events format
          const data = `data: ${JSON.stringify(event)}\n\n`;
          controller.enqueue(encoder.encode(data));
        } else if (event.type === "done" || event.type === "error") {
          const data = `data: ${JSON.stringify(event)}\n\n`;
          controller.enqueue(encoder.encode(data));
          controller.close();
          break;
        }
      }
    },
  });

  return new Response(readable, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
      Connection: "keep-alive",
    },
  });
});

On the client:

async function streamChat(message: string, onDelta: (text: string) => void) {
  const response = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ message }),
  });

  if (!response.ok || !response.body) {
    throw new Error(`HTTP ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const chunk = decoder.decode(value, { stream: true });
    for (const line of chunk.split("\n")) {
      if (!line.startsWith("data: ")) continue;
      const event = JSON.parse(line.slice(6));
      if (event.type === "text_delta") onDelta(event.delta);
      if (event.type === "done" || event.type === "error") return;
    }
  }
}

Where streaming breaks down

Streaming adds complexity at the infrastructure layer. Load balancers need to be configured for long-lived connections. Response buffering (common in nginx and many CDN configurations) defeats the purpose entirely. If your edge handles the request and buffers before forwarding to the origin, the user sees nothing until the complete response arrives.

Streaming also complicates error handling. Once you send a 200 status and start streaming, you cannot send a 500 if something breaks mid-stream. You have to encode error conditions in the stream format itself, which not all clients handle gracefully.

For server-to-server use cases where there is no user watching, streaming adds complexity with no benefit. Use synchronous calls.

Pattern 3: Background Processing

What it is

The HTTP request returns immediately with a job ID. The LLM work happens asynchronously in a worker process or queue consumer. The client polls for the result or receives it via webhook.

This is appropriate when:

• The total processing time exceeds what HTTP timeouts allow (anything over 30-60 seconds reliably)
• You are processing batches: hundreds of documents, thousands of rows, large corpora
• The user does not need to watch generation happen live
• You need to control throughput (rate limits, cost budgets, prioritization)
• The result needs to be stored, reviewed, or used in a downstream pipeline

Job submission and result retrieval

import { randomUUID } from "crypto";

// Simplified job store (use Redis or a database in production)
type JobStatus = "queued" | "running" | "completed" | "failed";

interface Job {
  id: string;
  status: JobStatus;
  input: unknown;
  output?: unknown;
  error?: string;
  createdAt: Date;
  completedAt?: Date;
}

const jobs = new Map<string, Job>();

// HTTP handler: submit work, return immediately
app.post("/api/extract", async (c) => {
  const { documentUrl } = await c.req.json<{ documentUrl: string }>();

  const jobId = randomUUID();
  const job: Job = {
    id: jobId,
    status: "queued",
    input: { documentUrl },
    createdAt: new Date(),
  };

  jobs.set(jobId, job);
  queue.push(jobId); // add to processing queue

  return c.json({ jobId, status: "queued" }, 202);
});

// HTTP handler: poll for result
app.get("/api/extract/:jobId", (c) => {
  const job = jobs.get(c.req.param("jobId"));
  if (!job) return c.json({ error: "Job not found" }, 404);

  return c.json({
    id: job.id,
    status: job.status,
    output: job.output,
    error: job.error,
  });
});

The worker

The worker pulls jobs from the queue and executes them. The key design decisions are: concurrency (how many jobs run simultaneously), error handling (what happens to failed jobs), and observability (how you know what is happening inside a 10-minute extraction run).

import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";

const client = new Anthropic();

const ExtractedDataSchema = z.object({
  invoiceNumber: z.string(),
  issueDate: z.string(),
  dueDate: z.string().optional(),
  lineItems: z.array(
    z.object({
      description: z.string(),
      quantity: z.number(),
      unitPrice: z.number(),
      total: z.number(),
    })
  ),
  totalAmount: z.number(),
  currency: z.string(),
  vendor: z.object({
    name: z.string(),
    address: z.string().optional(),
  }),
});

async function processExtractionJob(job: Job): Promise<void> {
  const { documentUrl } = job.input as { documentUrl: string };

  // Mark as running
  jobs.set(job.id, { ...job, status: "running" });

  try {
    // Fetch document content
    const docResponse = await fetch(documentUrl);
    if (!docResponse.ok) {
      throw new Error(`Failed to fetch document: HTTP ${docResponse.status}`);
    }
    const documentText = await docResponse.text();

    // Extract structured data
    const response = await client.messages.create({
      model: "claude-opus-4-6",
      max_tokens: 2048,
      system: `You are a document data extraction system. Extract the specified fields
from the provided document. Respond with a JSON object only. No markdown, no prose.
If a field is not present in the document, omit it from the response.`,
      messages: [
        {
          role: "user",
          content: `Extract invoice data from the following document:\n\n${documentText}`,
        },
      ],
    });

    const content = response.content[0];
    if (content.type !== "text") {
      throw new Error("Unexpected response type from LLM");
    }

    const extracted = ExtractedDataSchema.parse(JSON.parse(content.text));

    // Mark as completed
    jobs.set(job.id, {
      ...job,
      status: "completed",
      output: extracted,
      completedAt: new Date(),
    });
  } catch (err) {
    jobs.set(job.id, {
      ...job,
      status: "failed",
      error: String(err),
      completedAt: new Date(),
    });
  }
}

// Worker loop with controlled concurrency
async function startWorker(concurrency = 3): Promise<void> {
  const inFlight = new Set<string>();

  setInterval(async () => {
    while (inFlight.size < concurrency && queue.length > 0) {
      const jobId = queue.shift()!;
      const job = jobs.get(jobId);
      if (!job) continue;

      inFlight.add(jobId);
      processExtractionJob(job).finally(() => inFlight.delete(jobId));
    }
  }, 100);
}

The concurrency limit is doing real work here: without it, a burst of 100 submissions would fire 100 simultaneous LLM calls, blow through your rate limits, and produce a thundering herd of 429 errors.

Handling large batches with rate limiting

When processing thousands of documents, you need token-per-minute (TPM) awareness in addition to request concurrency control.

class RateLimitedQueue {
  private queue: Array<() => Promise<void>> = [];
  private running = 0;
  private requestsThisMinute = 0;
  private minuteStart = Date.now();

  constructor(
    private readonly maxConcurrent: number,
    private readonly maxRequestsPerMinute: number
  ) {}

  async add<T>(fn: () => Promise<T>): Promise<T> {
    return new Promise((resolve, reject) => {
      this.queue.push(async () => {
        try {
          resolve(await fn());
        } catch (e) {
          reject(e);
        }
      });
      this.drain();
    });
  }

  private async drain(): Promise<void> {
    while (this.running < this.maxConcurrent && this.queue.length > 0) {
      // Reset counter each minute
      const now = Date.now();
      if (now - this.minuteStart >= 60_000) {
        this.requestsThisMinute = 0;
        this.minuteStart = now;
      }

      if (this.requestsThisMinute >= this.maxRequestsPerMinute) {
        // Wait until the minute resets
        const waitMs = 60_000 - (now - this.minuteStart) + 100;
        await new Promise((r) => setTimeout(r, waitMs));
        continue;
      }

      const task = this.queue.shift()!;
      this.running++;
      this.requestsThisMinute++;

      task().finally(() => {
        this.running--;
        this.drain();
      });
    }
  }
}

const extractionQueue = new RateLimitedQueue(5, 50);

// Each document goes through the rate-limited queue
async function submitBatch(documentUrls: string[]): Promise<string[]> {
  const jobIds: string[] = [];

  for (const url of documentUrls) {
    const jobId = randomUUID();
    jobs.set(jobId, {
      id: jobId,
      status: "queued",
      input: { documentUrl: url },
      createdAt: new Date(),
    });
    jobIds.push(jobId);

    // Fire but do not await — queue handles concurrency
    extractionQueue.add(() => processExtractionJob(jobs.get(jobId)!));
  }

  return jobIds;
}

Where background processing breaks down

The main failure mode is invisible failure: a job sits in the queue, nothing processes it, and the caller keeps polling. You need dead-letter handling (jobs that fail N times move to a separate queue for inspection), visibility timeouts (jobs that start but never complete get re-queued), and monitoring on queue depth and job age.

For very long-running tasks (15+ minutes), the in-memory job store in the examples above is not sufficient. Use Redis, a database, or a purpose-built queue service. Restarts will drain the in-memory queue and lose progress.

Pattern Comparison

Dimension	Request-Response	Streaming Agent	Background Processing
Latency visibility	None until complete	Progressive	Deferred (status polling)
HTTP timeout risk	High for slow tasks	Low (long-lived connection)	None (returns 202 immediately)
User experience fit	Internal API, quick results	Interactive chat, copilots	Batch jobs, document processing
Infrastructure complexity	Low	Medium (SSE, load balancer config)	High (queue, worker, storage)
Error observability	Inline exception	Mid-stream error encoding	Job status + dead-letter queue
Throughput control	Per-request timeout	Connection concurrency	Queue depth + rate limiting
Retry semantics	Retry the full call	Restart the stream	Retry individual jobs
Cost visibility	Per call	Per session (multi-step)	Per job, aggregatable

Production Considerations

Timeouts need to be set at every layer. The application timeout, the HTTP client timeout, the load balancer timeout, and the LLM API timeout are four different numbers. If any one of them is shorter than your p99 latency, requests get cut off silently. Map them all before you go to production.

Idempotency matters for background jobs. If a job is submitted twice (duplicate webhook delivery, user double-click), you want the second submission to return the same job ID, not create a second extraction run. Hash the input and check for an existing job before creating a new one.

Token counting belongs in your observability stack. Each LLM call returns token usage. Log it on every call, tagged with the job type, the model, and whether it succeeded. This is the only way to build cost attribution by feature and to detect when an input class is unexpectedly large.

Structured output validation is not optional. The model will occasionally produce JSON that does not match your schema, or valid JSON with wrong types. Validate every response before you store or forward it. Return the validation error to the model for a self-correction attempt before failing the request.

Backpressure on the submission side. For background processing, if your queue grows faster than your worker can drain it, your system is in a degraded state. Track queue depth as a metric. Set an alert when it exceeds N jobs. Consider rejecting new submissions (503) when the queue is critically backed up rather than silently accepting work you cannot process in a reasonable time.

Connection keep-alive is not streaming. Some HTTP proxies buffer the entire response before forwarding it. If you are streaming and your p50 time-to-first-token is unchanged from your non-streaming latency, your proxy is buffering. Check proxy_buffering off (nginx) or the equivalent for your setup.

Composing Patterns in Real Systems

Real production systems use all three patterns, each where it fits.

A document review platform might look like this: the user uploads a document and submits it for analysis (background job, 202 response). The processing worker extracts structured data synchronously (request-response to the LLM, blocking inside the worker). The user returns to a chat interface to ask follow-up questions about the extracted data (streaming agent, with the extracted data injected as context).

The three patterns are not mutually exclusive. The boundary between them is: who is waiting, how long, and what happens if something goes wrong.

A synchronous call inside a background worker is fine. It is still synchronous from the worker’s perspective. A streaming agent on the client side can trigger background jobs via tool calls. The patterns compose cleanly as long as you are deliberate about which one governs the user-facing latency.

Closing

The pattern choice is an upstream architectural decision that determines your operational complexity, your error handling strategy, and your user’s experience. Getting it wrong early means refactoring under pressure when timeouts start piling up or the batch job backlog grows past your weekend tolerance.

Start with the simplest pattern that fits the latency budget. Add streaming when a human needs to watch progress. Move to background processing when the work exceeds what HTTP can hold. The LLM call itself is the same in all three cases. The architecture around it is where the real engineering work lives.