Building an Agentic RAG System: Query Planning, Multi-Source Retrieval, and Self-Correcting Answer Pipelines

Naive RAG works well in demos. Vector similarity search, top-k retrieval, stuff the chunks into a prompt, get an answer. The problem surfaces in production when the queries stop being simple lookups.

“What was our Q3 revenue for the enterprise segment, and how does that compare to what our largest competitor reported?” That question needs structured data from your analytics database, unstructured content from a document store, and possibly real-time data from an external source. A single vector similarity search returns irrelevant chunks, and the model hallucinates the rest.

This is the gap agentic RAG fills. Instead of a fixed retrieval pipeline, an agent actively reasons about what information it needs, where to get it, and whether the result is actually good enough to use.

Why Naive RAG Breaks on Complex Queries

The failure modes are specific and worth naming before writing any code.

Multi-hop reasoning. “Which of our customers who churned in Q4 had previously escalated a support ticket marked as unresolved?” This requires joining structured churn data with unstructured support ticket content. A single vector search cannot do this. You need a plan: get churned customers from the database, then search support tickets for that customer set, then filter by resolution status.

Ambiguous intent. “Tell me about our pricing changes” could mean recent product announcements, historical pricing evolution, competitive pricing analysis, or the internal rationale behind a specific change. A naive retriever picks the top-k semantically similar chunks and gets a blend of all of them. An agent should resolve the ambiguity before retrieving.

Heterogeneous sources. Real production systems have data in multiple places: structured records in Postgres or BigQuery, documents in a vector store, current state in APIs. Naive RAG only queries one source. Wiring it to multiple sources without a planning layer produces noise, not answers.

Missing context detection. If the relevant information is not in the vector store, naive RAG still returns something. The model fills the gap with plausible-sounding text. An agentic pipeline can detect low retrieval quality and either reformulate the query, try a different source, or explicitly say it does not have the information.

The Architecture

An agentic RAG system has four layers:

1. Query planner: takes a user question and produces a structured retrieval plan
2. Retrieval orchestrator: executes the plan across multiple sources in parallel where possible
3. Answer synthesizer: combines retrieved results into a grounded response with source attribution
4. Self-correction loop: validates the answer against the retrieved sources and iterates if quality is below threshold

Each layer is a discrete component with its own inputs, outputs, and failure modes.

Query Planning with LLMs

The query planner takes an ambiguous natural language question and outputs a typed retrieval plan. The model decides which sources to query, what to ask each one, and in what order.

interface SubQuery {
  id: string;
  source: "vector" | "sql" | "api";
  query: string;
  dependsOn?: string[]; // IDs of sub-queries that must complete first
  filters?: Record<string, unknown>;
}

interface RetrievalPlan {
  originalQuery: string;
  intent: string;
  subQueries: SubQuery[];
  synthesisInstructions: string;
}

async function planQuery(
  userQuery: string,
  availableSources: SourceSchema[],
  llm: LLMClient
): Promise<RetrievalPlan> {
  const systemPrompt = `You are a retrieval planner. Given a user question and a list of available data sources,
produce a structured plan to retrieve the information needed to answer it.

Available sources:
${availableSources.map((s) => `- ${s.id} (${s.type}): ${s.description}`).join("\n")}

Rules:
- Decompose complex questions into independent sub-queries where possible
- Mark dependencies explicitly when a sub-query result is needed to formulate another
- Prefer parallel retrieval over sequential where dependencies allow
- If the question is ambiguous, pick the most likely intent and note it`;

  const response = await llm.complete({
    model: "gpt-4o",
    temperature: 0,
    messages: [
      { role: "system", content: systemPrompt },
      {
        role: "user",
        content: `Plan retrieval for: "${userQuery}"\n\nReturn a JSON RetrievalPlan object.`,
      },
    ],
    responseFormat: { type: "json_object" },
  });

  return JSON.parse(response.content) as RetrievalPlan;
}

The key design decisions here:

The dependsOn field enables sequential execution only where necessary. A query about “customers who churned last quarter and their support history” decomposes into two sub-queries: one SQL query for churned customers, and one vector search for support tickets filtered to those customer IDs. The vector search depends on the SQL result. Without explicit dependency tracking, you either run everything sequentially (too slow) or run everything in parallel (incorrect, because you do not have the filter values yet).

Temperature is set to 0. The planner is not doing creative work. It is parsing intent and mapping to schema. Variance in plan structure makes the orchestrator brittle.

The synthesisInstructions field carries intent forward. The planner knows why each sub-query was included. The synthesizer can use this to weight sources appropriately rather than treating all retrieved content equally.

Multi-Source Retrieval Orchestration

The orchestrator takes the plan and executes it, handling parallelism, timeouts, and partial failures.

interface RetrievedContext {
  subQueryId: string;
  source: string;
  content: string;
  metadata: Record<string, unknown>;
  score?: number; // relevance score where available
  retrievedAt: string;
}

interface OrchestrationResult {
  contexts: RetrievedContext[];
  failedSubQueries: string[];
  totalLatencyMs: number;
}

async function orchestrateRetrieval(
  plan: RetrievalPlan,
  sources: SourceRegistry,
  options: { timeoutMs: number; maxRetries: number }
): Promise<OrchestrationResult> {
  const startTime = Date.now();
  const completed = new Map<string, RetrievedContext>();
  const failed: string[] = [];

  // Topological sort to find execution waves
  const waves = buildExecutionWaves(plan.subQueries);

  for (const wave of waves) {
    const results = await Promise.allSettled(
      wave.map((subQuery) =>
        executeSubQueryWithRetry(subQuery, completed, sources, options)
      )
    );

    for (let i = 0; i < results.length; i++) {
      const result = results[i];
      const subQuery = wave[i];

      if (result.status === "fulfilled") {
        completed.set(subQuery.id, result.value);
      } else {
        failed.push(subQuery.id);
        // Mark dependents as also failed
        markDependentsFailed(subQuery.id, plan.subQueries, failed);
      }
    }
  }

  return {
    contexts: Array.from(completed.values()),
    failedSubQueries: failed,
    totalLatencyMs: Date.now() - startTime,
  };
}

async function executeSubQueryWithRetry(
  subQuery: SubQuery,
  completedQueries: Map<string, RetrievedContext>,
  sources: SourceRegistry,
  options: { timeoutMs: number; maxRetries: number }
): Promise<RetrievedContext> {
  const resolvedQuery = resolveQueryTemplate(subQuery, completedQueries);

  for (let attempt = 0; attempt < options.maxRetries; attempt++) {
    try {
      const result = await Promise.race([
        executeSubQuery(resolvedQuery, sources),
        timeout(options.timeoutMs),
      ]);
      return result;
    } catch (err) {
      if (attempt === options.maxRetries - 1) throw err;
      await sleep(200 * Math.pow(2, attempt)); // exponential backoff
    }
  }

  throw new Error(`Exhausted retries for sub-query ${subQuery.id}`);
}

function buildExecutionWaves(subQueries: SubQuery[]): SubQuery[][] {
  const waves: SubQuery[][] = [];
  const resolved = new Set<string>();
  let remaining = [...subQueries];

  while (remaining.length > 0) {
    const ready = remaining.filter(
      (sq) => !sq.dependsOn || sq.dependsOn.every((dep) => resolved.has(dep))
    );

    if (ready.length === 0) {
      throw new Error("Circular dependency detected in retrieval plan");
    }

    waves.push(ready);
    ready.forEach((sq) => resolved.add(sq.id));
    remaining = remaining.filter((sq) => !resolved.has(sq.id));
  }

  return waves;
}

The execution wave pattern is the core of parallel retrieval. All sub-queries in a wave run concurrently. A new wave starts only when all queries in the previous wave complete (or fail). This gives you maximum parallelism without violating dependency constraints.

Each source type gets its own retriever:

async function executeSubQuery(
  subQuery: SubQuery,
  sources: SourceRegistry
): Promise<RetrievedContext> {
  switch (subQuery.source) {
    case "vector": {
      const store = sources.getVectorStore();
      const results = await store.similaritySearch(subQuery.query, {
        topK: 8,
        filters: subQuery.filters,
        includeScore: true,
      });
      return {
        subQueryId: subQuery.id,
        source: "vector",
        content: results.map((r) => r.pageContent).join("\n\n"),
        metadata: { chunks: results.map((r) => r.metadata) },
        score: Math.min(...results.map((r) => r.score ?? 0)),
        retrievedAt: new Date().toISOString(),
      };
    }

    case "sql": {
      const db = sources.getDatabase();
      // The query here is a natural language question that gets translated to SQL
      const sqlQuery = await translateToSQL(subQuery.query, db.schema);
      const rows = await db.query(sqlQuery);
      return {
        subQueryId: subQuery.id,
        source: "sql",
        content: JSON.stringify(rows, null, 2),
        metadata: { rowCount: rows.length, sqlQuery },
        retrievedAt: new Date().toISOString(),
      };
    }

    case "api": {
      const apiClient = sources.getAPIClient(subQuery.filters?.apiName as string);
      const response = await apiClient.fetch(subQuery.query, subQuery.filters);
      return {
        subQueryId: subQuery.id,
        source: "api",
        content: JSON.stringify(response.data),
        metadata: { endpoint: response.endpoint, cachedAt: response.cachedAt },
        retrievedAt: new Date().toISOString(),
      };
    }

    default:
      throw new Error(`Unknown source type: ${subQuery.source}`);
  }
}

The SQL path deserves attention. You are not giving the LLM planner raw SQL generation authority over your database. The natural language to SQL translation happens in a separate, tightly scoped step with its own schema context and validation layer. Letting a planning LLM write arbitrary SQL against your production database is a security and correctness problem.

Answer Synthesis with Source Attribution

The synthesizer takes all retrieved contexts and produces a grounded answer. Grounding means every claim in the answer is traceable to a specific retrieved source.

interface SourceAttribution {
  claim: string;
  sourceId: string;
  sourceType: string;
  confidence: "high" | "medium" | "low";
}

interface SynthesizedAnswer {
  answer: string;
  attributions: SourceAttribution[];
  gaps: string[]; // questions the retrieved context could not answer
  overallConfidence: number; // 0-1
}

async function synthesizeAnswer(
  originalQuery: string,
  plan: RetrievalPlan,
  orchestrationResult: OrchestrationResult,
  llm: LLMClient
): Promise<SynthesizedAnswer> {
  const contextBlocks = orchestrationResult.contexts.map((ctx) => ({
    id: ctx.subQueryId,
    source: ctx.source,
    content: ctx.content,
    metadata: ctx.metadata,
  }));

  const systemPrompt = `You are an answer synthesizer. Combine the retrieved context to answer the user's question.

Rules:
- Only make claims that are supported by the provided context
- For each factual claim, note which context block supports it
- If context blocks conflict, note the conflict explicitly
- If the context is insufficient to answer part of the question, list the gaps
- Do not hallucinate. "I don't know" is a valid and correct answer.

Synthesis instructions from planner: ${plan.synthesisInstructions}`;

  const contextText = contextBlocks
    .map(
      (block) =>
        `[Context ${block.id} | Source: ${block.source}]\n${block.content}`
    )
    .join("\n\n---\n\n");

  const response = await llm.complete({
    model: "gpt-4o",
    temperature: 0.1,
    messages: [
      { role: "system", content: systemPrompt },
      {
        role: "user",
        content: `Question: ${originalQuery}\n\nContext:\n${contextText}\n\nReturn a JSON SynthesizedAnswer object.`,
      },
    ],
    responseFormat: { type: "json_object" },
  });

  return JSON.parse(response.content) as SynthesizedAnswer;
}

The gaps field is what distinguishes a reliable system from a hallucinating one. Forcing the model to explicitly enumerate what it could not answer from the retrieved context creates a surface for the self-correction loop to work on.

Self-Correction Loop

The correction loop detects low-quality answers and attempts to improve them through targeted re-retrieval.

interface ValidationResult {
  passed: boolean;
  retrievalQualityScore: number; // 0-1
  halluccinationRisk: "low" | "medium" | "high";
  issues: string[];
  suggestedRefinements: SubQuery[];
}

async function validateAnswer(
  answer: SynthesizedAnswer,
  originalQuery: string,
  retrievedContexts: RetrievedContext[],
  llm: LLMClient
): Promise<ValidationResult> {
  // Check 1: retrieval quality from scores
  const vectorContexts = retrievedContexts.filter((c) => c.source === "vector");
  const avgVectorScore =
    vectorContexts.length > 0
      ? vectorContexts.reduce((sum, c) => sum + (c.score ?? 0), 0) /
        vectorContexts.length
      : 1;

  // Check 2: gap coverage
  const hasSignificantGaps = answer.gaps.length > 0;

  // Check 3: confidence distribution in attributions
  const lowConfidenceRatio =
    answer.attributions.filter((a) => a.confidence === "low").length /
    Math.max(answer.attributions.length, 1);

  // Check 4: LLM-based factual consistency check
  const consistencyCheck = await checkFactualConsistency(
    answer,
    retrievedContexts,
    llm
  );

  const retrievalQualityScore =
    avgVectorScore * 0.3 +
    (hasSignificantGaps ? 0 : 0.3) +
    (1 - lowConfidenceRatio) * 0.2 +
    consistencyCheck.score * 0.2;

  const passed = retrievalQualityScore > 0.7 && !consistencyCheck.flagged;

  return {
    passed,
    retrievalQualityScore,
    halluccinationRisk: consistencyCheck.flagged
      ? "high"
      : retrievalQualityScore < 0.5
      ? "medium"
      : "low",
    issues: [
      ...consistencyCheck.issues,
      ...(hasSignificantGaps
        ? [`Unanswered gaps: ${answer.gaps.join(", ")}`]
        : []),
    ],
    suggestedRefinements: consistencyCheck.suggestedRefinements,
  };
}

async function runWithCorrection(
  userQuery: string,
  sources: SourceRegistry,
  llm: LLMClient,
  options: { maxIterations: number; timeoutMs: number }
): Promise<SynthesizedAnswer> {
  let plan = await planQuery(userQuery, sources.getSchemas(), llm);
  let iteration = 0;

  while (iteration < options.maxIterations) {
    const orchestrationResult = await orchestrateRetrieval(plan, sources, {
      timeoutMs: options.timeoutMs,
      maxRetries: 2,
    });

    const answer = await synthesizeAnswer(
      userQuery,
      plan,
      orchestrationResult,
      llm
    );

    const validation = await validateAnswer(
      answer,
      userQuery,
      orchestrationResult.contexts,
      llm
    );

    if (validation.passed) {
      return answer;
    }

    // Refine the plan based on validation feedback
    if (validation.suggestedRefinements.length > 0) {
      plan = {
        ...plan,
        subQueries: [
          ...plan.subQueries,
          ...validation.suggestedRefinements,
        ],
      };
    } else {
      // No suggested refinements means we have retrieved everything we can
      // Return what we have with the quality score noted
      return {
        ...answer,
        overallConfidence: validation.retrievalQualityScore,
      };
    }

    iteration++;
  }

  throw new Error(
    `Could not produce a validated answer after ${options.maxIterations} iterations`
  );
}

The correction loop has a hard iteration cap. Without it, a query that genuinely cannot be answered from available sources will spin indefinitely, burning tokens and time. When the validator has no suggested refinements, you surface what you have. This is the right behavior: a partial answer with an honest confidence score is more useful than either an infinite loop or a hallucinated answer.

Production Concerns

Latency Budgets

Multi-step retrieval compounds latency. A three-wave plan with three LLM calls (planner, synthesizer, validator) and parallel retrieval can take 8-15 seconds. Whether this is acceptable depends on your use case.

For interactive queries, set aggressive timeouts per sub-query (2-3 seconds) and prefer partial results over waiting. Return the answer you have and stream refinements. For batch or async queries, the latency budget is larger and you can afford more iterations.

Track latency at each stage separately: planning, each retrieval wave, synthesis, validation. This tells you where time goes and what to optimize. Usually the bottleneck is the first LLM call (planning) or sequential retrieval waves that could be parallelized differently.

Stage	Typical latency	Optimization lever
Query planning	1-2s	Cache plans for similar queries
Vector retrieval	50-200ms	ANN index tuning, pre-filtering
SQL retrieval	100ms-2s	Query optimization, read replicas
API retrieval	200ms-3s	Response caching, connection pooling
Answer synthesis	2-5s	Smaller context windows, faster models
Validation	1-3s	Rule-based checks first, LLM as fallback

Caching Strategies

Three levels of caching apply here.

Plan caching. Similar queries often produce identical retrieval plans. Embed the query, check cosine similarity against cached plans, and reuse if similarity is above 0.95. Cache the plan object, not just the answer. This saves the most expensive call and keeps retrieval fresh.

Retrieval caching. Cache sub-query results by source and normalized query string. SQL results can be cached for seconds to minutes depending on data volatility. Vector search results for the same query string can be cached for hours. API responses should respect the API’s own caching headers.

Answer caching. Cache final answers only for deterministic queries where you are confident the underlying data has not changed. Use TTLs tied to source volatility, not a single global TTL.

class RetrievalCache {
  constructor(private redis: Redis, private vectorIndex: VectorIndex) {}

  async getPlanForQuery(query: string): Promise<RetrievalPlan | null> {
    const embedding = await embed(query);
    const similar = await this.vectorIndex.search(embedding, {
      namespace: "plan-cache",
      topK: 1,
      threshold: 0.95,
    });

    if (similar.length === 0) return null;

    const cached = await this.redis.get(`plan:${similar[0].id}`);
    return cached ? JSON.parse(cached) : null;
  }

  async cachePlan(query: string, plan: RetrievalPlan): Promise<void> {
    const embedding = await embed(query);
    const id = generateId();
    await this.vectorIndex.upsert(id, embedding, { namespace: "plan-cache" });
    await this.redis.setex(`plan:${id}`, 3600, JSON.stringify(plan));
  }

  async getSubQueryResult(
    subQueryId: string,
    normalizedQuery: string,
    source: string
  ): Promise<RetrievedContext | null> {
    const key = `retrieval:${source}:${hash(normalizedQuery)}`;
    const cached = await this.redis.get(key);
    return cached ? JSON.parse(cached) : null;
  }
}

Observability for Agentic Retrieval Chains

Standard APM traces do not capture enough for agentic pipelines. You need trace data at the level of individual sub-queries: which sources were queried, what was returned, how long each took, which validation checks passed or failed.

Structure your traces as a DAG, not a linear span tree:

interface AgenticTrace {
  traceId: string;
  userQuery: string;
  iterations: {
    iteration: number;
    planGenerated: RetrievalPlan;
    retrievalWaves: {
      wave: number;
      subQueries: {
        id: string;
        source: string;
        query: string;
        latencyMs: number;
        resultSize: number;
        score?: number;
        error?: string;
      }[];
      waveLatencyMs: number;
    }[];
    synthesisLatencyMs: number;
    validationResult: ValidationResult;
    totalIterationLatencyMs: number;
  }[];
  finalAnswer: SynthesizedAnswer;
  totalLatencyMs: number;
  totalLLMCalls: number;
  totalTokensUsed: number;
}

The totalLLMCalls and totalTokensUsed fields are critical for cost attribution. Each iteration of the correction loop consumes multiple LLM calls. A query that iterates three times costs 3x what you expect.

Emit this trace to your observability backend on every request. Alert on:

• Queries exceeding your latency budget p95
• High iteration counts (signals retrieval quality issues or over-ambitious query decomposition)
• High rate of validation failures (signals the retriever and synthesizer are misaligned)
• Cost per query exceeding threshold (a single expensive query can indicate a runaway loop)

Cost Management

Every LLM call in this pipeline has a cost, and the correction loop multiplies it. Three levers control cost:

Model tiering. Use a fast, cheap model (gpt-4o-mini, Claude Haiku) for query planning and validation. Reserve the capable model for synthesis. Planning is classification work; it does not require frontier capability. This alone reduces per-query cost by 40-60% compared to using a single model throughout.

Context window discipline. The synthesizer receives all retrieved contexts. Large context windows are expensive. Truncate retrieved content aggressively before synthesis. The planner can include a maxChars budget per sub-query as part of the plan.

Iteration limits and circuit breakers. Cap correction iterations at 2-3. Beyond that, the marginal improvement from additional retrieval rarely justifies the cost. Log when you hit the cap so you can investigate whether the query type needs a different strategy.

const costConfig = {
  planning: { model: "gpt-4o-mini", maxTokens: 1000 },
  synthesis: { model: "gpt-4o", maxTokens: 2000 },
  validation: { model: "gpt-4o-mini", maxTokens: 800 },
  maxIterations: 2,
  maxContextCharsPerSubQuery: 4000,
  maxTotalContextChars: 12000,
};

Set these values per query type or per feature. A low-stakes internal search tool can afford cheaper settings. A customer-facing answer system warrants the better model for synthesis.

Tradeoffs Table

Approach	Retrieval quality	Latency	Cost	When to use
Naive single-vector RAG	Low on complex queries	200-500ms	Minimal	Simple FAQ, single-source
Multi-source without planning	Medium	500ms-2s	Low	Known query types, static source mix
Agentic with query planning	High	3-8s	Medium	Heterogeneous sources, complex queries
Agentic with self-correction	Highest	8-20s	High	High-stakes answers, low hallucination tolerance
Agentic with caching	High	500ms-3s (warm)	Medium (amortized)	Repeated query patterns, production scale

What You Actually Get

Agentic RAG does not eliminate hallucination. It reduces the conditions that cause it: poor retrieval, mismatched sources, insufficient context. A well-built agentic pipeline catches most hallucinations before they reach the user by detecting when the retrieved context does not actually support the generated claims.

The operational reality: the correction loop adds latency and cost, but it shifts the failure mode from “confident wrong answers” to “uncertain slow answers.” For most production use cases, that trade is worth making. A system that says “I could not find enough information to answer this confidently” is far more trustworthy than one that makes things up at 500ms.

The components worth spending time on before anything else: the retrieval quality score in the validation step, and the gap detection in synthesis. These two signals tell you whether your retrieval strategy is actually working and where to invest next, whether that is indexing improvements, new source integrations, or better query decomposition prompts.