LLM Memory and Context Management: Sliding Windows, Summarization, and Long-Term Recall for Production Agents

Every LLM call is stateless. The model has no memory of what happened in prior turns unless you put it in the context window yourself. That sounds like a minor detail until you are building a production agent that needs to remember what a user said three days ago, carry a coherent multi-step task across a 30-turn conversation, or avoid re-reading 40,000 tokens of history on every request just to answer a simple follow-up question.

This is the core memory problem for LLM applications: you need selective, affordable access to the right prior context at call time. The strategies to get there span from trivial (truncate old messages) to complex (hierarchical stores with vector retrieval), and the tradeoffs between them are real enough to affect both the quality of your application and your monthly API bill.

This article covers the full range of strategies with concrete TypeScript implementations using the Anthropic and OpenAI SDKs.

---

The Problem Space

A typical conversation with a capable model has several distinct memory needs:

• Working memory: The current task state, intermediate results, tool call outputs. Needs to be in the context window right now.
• Episodic memory: What happened earlier in this conversation. Needs some representation, but not necessarily verbatim.
• Long-term memory: Facts, preferences, or history from past sessions. Cannot fit in the context window at all without filtering.
• Semantic memory: General knowledge the model already has from pretraining. Effectively free.

Context window limits force you to choose what gets in. A 200K-token window (Claude’s current maximum) sounds large until you realize that a 6-month conversation history, a full codebase, and a set of tool outputs all competing for space will exceed it. And even if they fit, you are paying for every token on every call.

The right memory strategy depends on which of these four categories matters for your workload.

---

Sliding Window: Truncation and Summarization

The simplest approach is a sliding window over the most recent N messages. When the buffer fills, you drop or compress the oldest content.

Hard Truncation

import Anthropic from "@anthropic-ai/sdk";

type Message = {
  role: "user" | "assistant";
  content: string;
};

function truncateHistory(
  messages: Message[],
  maxTokens: number,
  estimatedTokensPerChar = 0.25
): Message[] {
  let totalChars = 0;
  const maxChars = maxTokens / estimatedTokensPerChar;

  // Walk backwards from most recent, keep what fits
  const kept: Message[] = [];
  for (let i = messages.length - 1; i >= 0; i--) {
    totalChars += messages[i].content.length;
    if (totalChars > maxChars) break;
    kept.unshift(messages[i]);
  }
  return kept;
}

const client = new Anthropic();

async function chat(history: Message[], userMessage: string): Promise<string> {
  const updatedHistory: Message[] = [
    ...history,
    { role: "user", content: userMessage },
  ];

  // Reserve ~2000 tokens for the response and system prompt overhead
  const contextMessages = truncateHistory(updatedHistory, 180_000);

  const response = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 2048,
    messages: contextMessages,
  });

  return response.content[0].type === "text" ? response.content[0].text : "";
}

Hard truncation is cheap and predictable. The downside is abrupt: the model has no knowledge of what was in the dropped messages. If a user mentioned their database schema in turn 2 and asks a follow-up in turn 25, the model will answer without that context. For casual chatbots this is tolerable. For task-oriented agents it fails badly.

Where truncation breaks down

The model behaves as if the conversation started at whatever turn you kept. It will not know it is missing context. It will not ask for clarification. It will hallucinate plausible-sounding answers based on what it can see. This is the failure mode most developers do not catch until users start complaining.

---

Conversation Summarization Pipelines

The alternative to dropping messages is compressing them. When the history exceeds a threshold, summarize the oldest N turns and replace them with a compact prose summary. This preserves semantically important content at a fraction of the token cost.

import Anthropic from "@anthropic-ai/sdk";

type Message = {
  role: "user" | "assistant";
  content: string;
};

type ConversationState = {
  summary: string | null;
  recentMessages: Message[];
  totalTurns: number;
};

const client = new Anthropic();

async function summarizeMessages(messages: Message[]): Promise<string> {
  const transcript = messages
    .map((m) => `${m.role.toUpperCase()}: ${m.content}`)
    .join("\n\n");

  const response = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 512,
    messages: [
      {
        role: "user",
        content: `Summarize the following conversation excerpt. Preserve all factual details, decisions made, and open questions. Be concise but complete.\n\n${transcript}`,
      },
    ],
  });

  return response.content[0].type === "text" ? response.content[0].text : "";
}

async function addTurn(
  state: ConversationState,
  role: "user" | "assistant",
  content: string,
  summarizeAfter = 10
): Promise<ConversationState> {
  const updatedMessages: Message[] = [
    ...state.recentMessages,
    { role, content },
  ];

  // When buffer fills, summarize the oldest half
  if (updatedMessages.length >= summarizeAfter) {
    const toSummarize = updatedMessages.slice(0, Math.floor(summarizeAfter / 2));
    const keep = updatedMessages.slice(Math.floor(summarizeAfter / 2));

    const newSummary = await summarizeMessages(toSummarize);
    const combinedSummary = state.summary
      ? `${state.summary}\n\n${newSummary}`
      : newSummary;

    return {
      summary: combinedSummary,
      recentMessages: keep,
      totalTurns: state.totalTurns + 1,
    };
  }

  return {
    ...state,
    recentMessages: updatedMessages,
    totalTurns: state.totalTurns + 1,
  };
}

function buildContextMessages(state: ConversationState): Message[] {
  if (!state.summary) return state.recentMessages;

  // Inject summary as a synthetic user/assistant exchange at the start
  const summaryInjection: Message[] = [
    {
      role: "user",
      content: "Here is a summary of our conversation so far:",
    },
    {
      role: "assistant",
      content: state.summary,
    },
  ];

  return [...summaryInjection, ...state.recentMessages];
}

Key implementation decisions here:

Use a cheaper model for summarization. Haiku or GPT-4o-mini are adequate for summarization and cost a fraction of your primary model. The cost savings matter when summaries run on every threshold crossing.

Summarize incrementally, not all at once. Rolling summaries avoid the case where a single giant summarization call fails and loses all context. Each batch is small and recoverable.

Preserve decisions and open questions explicitly. Generic summaries drift toward high-level topic descriptions. Prompting the summarizer to keep facts, decisions, and open questions substantially improves retrieval quality downstream.

The main failure mode of summarization is lossy compression. If a user specified a constraint (“never use PostgreSQL”) early in the conversation and the summarizer omitted it, the model will eventually violate it. The fix is to extract and separately store structured facts rather than relying on free-text summarization to preserve everything.

---

Long-Term Memory with Vector Stores

Summarization handles the current conversation. It does not help when a user returns after three days and references something from a previous session. For that, you need persistence across sessions with retrieval at call time.

The standard approach: embed and store conversation turns (or summaries of them) in a vector database. On each new turn, retrieve the semantically closest past memories and inject them into the context.

import OpenAI from "openai";

type Memory = {
  id: string;
  userId: string;
  content: string;
  embedding: number[];
  createdAt: Date;
  sessionId: string;
};

// Minimal vector store interface -- replace with Pinecone, pgvector, etc.
interface VectorStore {
  upsert(memory: Memory): Promise<void>;
  query(
    userId: string,
    queryEmbedding: number[],
    topK: number
  ): Promise<Memory[]>;
}

const openai = new OpenAI();

async function embedText(text: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return response.data[0].embedding;
}

async function storeMemory(
  store: VectorStore,
  userId: string,
  sessionId: string,
  content: string
): Promise<void> {
  const embedding = await embedText(content);
  await store.upsert({
    id: `${userId}-${Date.now()}`,
    userId,
    sessionId,
    content,
    embedding,
    createdAt: new Date(),
  });
}

async function retrieveRelevantMemories(
  store: VectorStore,
  userId: string,
  currentMessage: string,
  topK = 5
): Promise<Memory[]> {
  const queryEmbedding = await embedText(currentMessage);
  return store.query(userId, queryEmbedding, topK);
}

function formatMemoriesForContext(memories: Memory[]): string {
  if (memories.length === 0) return "";

  const formatted = memories
    .sort((a, b) => b.createdAt.getTime() - a.createdAt.getTime())
    .map(
      (m) =>
        `[${m.createdAt.toLocaleDateString()}] ${m.content}`
    )
    .join("\n");

  return `Relevant context from past conversations:\n${formatted}`;
}

async function buildPromptWithMemory(
  store: VectorStore,
  userId: string,
  currentMessage: string,
  recentMessages: Array<{ role: "user" | "assistant"; content: string }>
): Promise<Array<{ role: "user" | "assistant"; content: string }>> {
  const memories = await retrieveRelevantMemories(
    store,
    userId,
    currentMessage
  );
  const memoryContext = formatMemoriesForContext(memories);

  if (!memoryContext) return recentMessages;

  return [
    {
      role: "user",
      content: memoryContext,
    },
    {
      role: "assistant",
      content: "Understood. I have access to that prior context.",
    },
    ...recentMessages,
  ];
}

What to store. The unit of storage matters. Storing individual messages is noisy; the embeddings for “yes” and “okay” are nearly identical and not useful. Better options: end-of-turn summaries, extracted facts (“user prefers TypeScript over Python”), or completed task outcomes. Each of these carries semantic weight that will surface meaningfully in retrieval.

When to store. Storing on every turn bloats the vector store and degrades retrieval precision over time. A better policy: store after each session ends, and store only messages that contain decisions, preferences, or facts. You can use a classifier (even a simple prompt) to filter what is worth persisting.

Retrieval precision. cosine similarity on embeddings is a blunt instrument. For users with long histories, top-K retrieval will return the most semantically similar past content, which may not be the most relevant. Adding recency weighting (blend similarity score with a decay function over creation date) and per-user filtering before querying substantially improves results.

---

Hierarchical Memory Architecture

In production, a single memory tier is rarely sufficient. A workable hierarchy for long-running agents:

Working Memory (in-context)
  Current task state, tool outputs, recent turns
  Cost: high per token
  Latency: zero (already in window)

Episodic Buffer (in-context, compressed)
  Rolling summary of current session
  Cost: moderate (summarization overhead)
  Latency: near-zero

Long-Term Store (vector retrieval)
  Cross-session facts, preferences, decisions
  Cost: low (retrieval + embedding)
  Latency: 50-200ms

Structured Store (key-value or relational)
  Explicit user preferences, configuration, account facts
  Cost: low (DB query)
  Latency: 5-20ms

type HierarchicalContext = {
  workingMemory: string; // Current task state as structured string
  sessionSummary: string | null; // Rolling summary of this session
  longTermMemories: string[]; // Retrieved from vector store
  structuredFacts: Record<string, string>; // User preferences, config
};

function assembleContext(ctx: HierarchicalContext): string {
  const parts: string[] = [];

  if (Object.keys(ctx.structuredFacts).length > 0) {
    const facts = Object.entries(ctx.structuredFacts)
      .map(([k, v]) => `${k}: ${v}`)
      .join("\n");
    parts.push(`User profile:\n${facts}`);
  }

  if (ctx.longTermMemories.length > 0) {
    parts.push(
      `Relevant past context:\n${ctx.longTermMemories.join("\n")}`
    );
  }

  if (ctx.sessionSummary) {
    parts.push(`Earlier this session:\n${ctx.sessionSummary}`);
  }

  if (ctx.workingMemory) {
    parts.push(`Current task state:\n${ctx.workingMemory}`);
  }

  return parts.join("\n\n");
}

The structured store deserves attention. For facts that are explicitly known (user timezone, language preference, account tier), a key-value lookup is more reliable than vector retrieval. Retrieval returns what is semantically similar to the query, not what is definitively true about the user. Mixing retrieval results with explicitly stored facts keeps deterministic information deterministic.

---

Working Memory Patterns for Agents

For agents that execute multi-step tasks (rather than conversational assistants), working memory is more important than episodic memory. The agent needs to track task state, intermediate results, and decisions across tool calls within a single run.

type ToolCall = {
  name: string;
  input: Record<string, unknown>;
  output: unknown;
  timestamp: number;
};

type AgentWorkingMemory = {
  taskDescription: string;
  completedSteps: string[];
  pendingSteps: string[];
  toolCallHistory: ToolCall[];
  currentHypothesis: string | null;
  blockers: string[];
};

function serializeWorkingMemory(wm: AgentWorkingMemory): string {
  const lines: string[] = [`Task: ${wm.taskDescription}`];

  if (wm.completedSteps.length > 0) {
    lines.push(`Completed:\n${wm.completedSteps.map((s) => `- ${s}`).join("\n")}`);
  }

  if (wm.pendingSteps.length > 0) {
    lines.push(`Remaining:\n${wm.pendingSteps.map((s) => `- ${s}`).join("\n")}`);
  }

  if (wm.currentHypothesis) {
    lines.push(`Current hypothesis: ${wm.currentHypothesis}`);
  }

  if (wm.blockers.length > 0) {
    lines.push(`Blockers:\n${wm.blockers.map((b) => `- ${b}`).join("\n")}`);
  }

  // Only include the last 5 tool calls to avoid bloating context
  if (wm.toolCallHistory.length > 0) {
    const recent = wm.toolCallHistory.slice(-5);
    const toolSummary = recent
      .map((tc) => `${tc.name}(${JSON.stringify(tc.input)}) => ${JSON.stringify(tc.output)}`)
      .join("\n");
    lines.push(`Recent tool calls:\n${toolSummary}`);
  }

  return lines.join("\n\n");
}

The pattern here is explicit state externalization. Instead of relying on the model to infer task progress from raw conversation history, you maintain a structured state object and serialize it into the system prompt or first user turn on each step. The model always has a clean view of where things stand without re-reading all prior turns.

Tool call history should be trimmed aggressively. The model does not need to re-read 50 prior tool outputs to decide what to do next. Keep the last 3-5 calls and summarize the rest into the completed steps list.

---

Token Budget Management

Token costs accumulate at the context level, not the message level. A 10-turn conversation with a 2,000-token system prompt, five 500-token assistant messages, and five 300-token user messages costs roughly 16,500 tokens on the final call, even if the new content added is only 200 tokens.

type TokenBudget = {
  systemPrompt: number;
  memoryContext: number;
  conversationHistory: number;
  responseBuffer: number;
  maxContextTokens: number;
};

function allocateTokenBudget(
  maxContextTokens: number,
  responseBuffer = 4096
): TokenBudget {
  const available = maxContextTokens - responseBuffer;
  return {
    systemPrompt: Math.floor(available * 0.1), // 10%
    memoryContext: Math.floor(available * 0.25), // 25%
    conversationHistory: Math.floor(available * 0.65), // 65%
    responseBuffer,
    maxContextTokens,
  };
}

// Rough token estimation: 1 token ~ 4 chars for English prose
function estimateTokens(text: string): number {
  return Math.ceil(text.length / 4);
}

function fitContentToTokenBudget(content: string, budget: number): string {
  const estimated = estimateTokens(content);
  if (estimated <= budget) return content;

  // Truncate to budget (character level approximation)
  const maxChars = budget * 4;
  return content.slice(0, maxChars) + "\n[truncated]";
}

The allocation percentages above are starting points, not rules. A coding assistant where the system prompt includes tool definitions might need 20-30% for the system prompt. An agent that does deep historical recall might need 40% for memory context. Instrument your actual token usage per call and adjust.

Track input token cost per session, not just per call. A session that burns 500K tokens across 50 turns will surprise you when you see the invoice if you are only watching individual call costs.

---

Tradeoffs

Dimension	Hard Truncation	Summarization	Vector Retrieval
Context fidelity	Low (drops messages)	Medium (lossy compression)	High for retrieved items
Latency overhead	None	200-800ms per summary	50-200ms per query
Cost overhead	None	~0.5-2% of primary call cost	~0.1-0.5% (embedding cost)
Cross-session recall	No	No	Yes
Implementation complexity	Trivial	Moderate	High
Works for task agents	Poor	Adequate	Good
Failure mode	Silent context loss	Fact loss under compression	Retrieval misses

---

Production Considerations

Summarization failures are silent. If your summarization call fails or returns garbage, the agent proceeds with a broken context and produces plausible-sounding wrong answers. Validate summary output: check that it is non-empty, meets a minimum length, and does not contain model refusals. Retry on failure before falling back to truncation.

Embedding model versioning. If you switch embedding models (e.g., from text-embedding-ada-002 to text-embedding-3-small), past embeddings become incompatible. Re-embed your entire store or maintain versioned embedding columns. Mismatched embeddings produce silently wrong retrieval results, not errors.

Memory poisoning. A malicious or confused user who puts false information into long-term memory will corrupt future sessions. For user-facing applications, consider human-in-the-loop confirmation before persisting facts to long-term store, and allow users to inspect and delete their stored memories.

Cold start for new users. On the first session there is no long-term memory. Design prompts to handle the zero-memory case explicitly rather than silently omitting the memory context section.

Observability. Log the token count breakdown per call (system prompt, memory injection, conversation history, response). Log retrieval hit rates and the similarity scores of returned memories. Log summarization latency. These metrics surface memory bugs that are invisible in normal response quality evaluations.

---

Choosing a Strategy

Start here: does your application need to recall information across sessions?

No: Use a rolling summarization buffer. It handles long single-session conversations gracefully and requires no external infrastructure. Set a summarization threshold between 8-16 messages, use your cheapest capable model for summarization, and keep the last 4-6 turns verbatim.

Yes, but retrieval quality is secondary to cost: Add a vector store with per-session summary embeddings. Store one embedding per completed session, not per message. Retrieve the 3-5 most relevant past sessions and inject their summaries.

Yes, and precision matters: Add a structured key-value store alongside the vector store. Explicit facts go in the KV store; semantic context goes in the vector store. Retrieve both on each turn and merge.

Building a multi-step task agent: The answer is working memory first, everything else second. Explicit state serialization buys more reliability than any memory retrieval strategy because it eliminates the model’s need to infer task state from history.

---

The pattern across all of these strategies is the same: treat context as a managed resource, not an unlimited buffer. Decide explicitly what goes in, how it gets there, and what leaves. The models are capable. The context window is the constraint. Managing it well is the engineering problem.