Knowledge Graphs for RAG: When Vector Search Alone Is Not Enough

Vector search is a good retrieval primitive. It handles semantic similarity well, it scales predictably, and its failure modes are well understood. But when your users start asking questions that require connecting facts across documents, understanding entity relationships, or reasoning over sequences of events, cosine similarity stops being enough.

This article covers the specific failure modes of vector-only RAG, explains the knowledge graph concepts engineers need to understand without the academic ceremony, and walks through a concrete hybrid retrieval implementation. If you have already shipped a RAG system and are running into its limits, this is the practical path forward.

Where Vector-Only RAG Breaks

Before adding a knowledge graph to your stack, you need to know which problems it actually solves. Not every RAG system needs one.

Multi-hop reasoning. A user asks: “Which engineers who worked on Project Atlas also contributed to the authentication service refactor?” Answering this requires connecting person-to-project and person-to-codebase relationships across multiple documents. A vector search returns chunks that are semantically similar to the query, but none of those chunks contain the full answer. The answer is assembled by traversing relationships, not by finding a similar passage.

Entity relationship queries. “What are all the dependencies of the payments service?” requires knowing which specific entities exist and how they relate. Vector similarity will retrieve passages that mention the payments service, but the actual dependency graph is structural knowledge. It is not meaningfully represented as floating-point vectors.

Temporal ordering. “What changed in the API contract between Q3 and Q4 last year?” requires understanding that documents have timestamps and that two versions of the same entity exist in a specific sequence. Without temporal structure, a vector store will blend old and new information in whatever ratio happens to land in the top-k.

Contradictory sources. When document A says a configuration default is 30 seconds and document B says it is 60 seconds, a vector store retrieves both chunks and hands them to the model. The model has no way to know which is authoritative. A knowledge graph can represent provenance, recency, and confidence scores on edges, giving the query layer a basis for resolution.

These are real failure modes. If your RAG system is mostly answering self-contained factual questions from a coherent, non-overlapping corpus, you probably do not need a graph. If users are asking relational, temporal, or multi-step questions, you do.

Knowledge Graph Fundamentals

A knowledge graph is a directed labeled graph where:

• Nodes represent entities: people, services, documents, concepts, events.
• Edges represent relationships between entities: DEPENDS_ON, AUTHORED_BY, REPLACED_BY, MENTIONED_IN.
• Properties are key-value pairs attached to either nodes or edges: { name: "payments-service", version: "2.4.1" } on a node, { since: "2025-Q3", confidence: 0.92 } on an edge.

The fundamental unit of information is a triple: (subject, predicate, object). For example: (payments-service, DEPENDS_ON, auth-service). Triple stores (like RDF-based systems) are built around this model. Property graph databases like Neo4j extend it by allowing rich property bags on both nodes and edges, which is usually more practical for engineering use cases.

Cypher is the query language for Neo4j and has become a de facto standard for property graph query. A basic query looks like:

MATCH (s:Service)-[:DEPENDS_ON]->(dep:Service)
WHERE s.name = 'payments-service'
RETURN dep.name, dep.version

You do not need to understand RDF or OWL to use a knowledge graph effectively in a RAG pipeline. Property graphs with Cypher are the pragmatic path for most engineering teams.

Building a Knowledge Graph from Documents

The hard part is not querying the graph. It is building it.

Entity Extraction

You need to identify the named entities in your documents: services, people, configurations, APIs, events. An LLM is the right tool here for unstructured text where you cannot rely on schema.

import OpenAI from "openai";
import { z } from "zod";

const client = new OpenAI();

const EntitySchema = z.object({
  entities: z.array(
    z.object({
      id: z.string(),
      type: z.enum(["Service", "Person", "Concept", "Event", "Configuration"]),
      name: z.string(),
      properties: z.record(z.string()),
    })
  ),
  relations: z.array(
    z.object({
      from: z.string(),
      predicate: z.string(),
      to: z.string(),
      properties: z.record(z.string()),
    })
  ),
});

type ExtractionResult = z.infer<typeof EntitySchema>;

async function extractEntitiesAndRelations(
  text: string,
  documentId: string
): Promise<ExtractionResult> {
  const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [
      {
        role: "system",
        content: `Extract entities and relationships from the provided technical document.
Return a JSON object with:
- entities: array of { id, type, name, properties }
- relations: array of { from, to, predicate, properties }

Entity types: Service, Person, Concept, Event, Configuration
Use snake_case for predicates: DEPENDS_ON, AUTHORED_BY, REPLACED_BY, CONFIGURES, TRIGGERS

Be conservative. Only extract relationships you are confident about.
Assign short stable IDs like "service:payments" or "person:alice-chen".`,
      },
      {
        role: "user",
        content: `Document ID: ${documentId}\n\n${text}`,
      },
    ],
    response_format: { type: "json_object" },
  });

  const raw = JSON.parse(response.choices[0].message.content ?? "{}");
  return EntitySchema.parse(raw);
}

Coreference Resolution

A single entity may be referenced by multiple names across documents. “the payments service”, “payments-svc”, and “the payment processing system” may all refer to the same node. Without resolving this, your graph fragments into isolated clusters.

A practical approach: after extraction, run a deduplication pass that embeds entity names, clusters them by cosine similarity, and prompts an LLM to confirm whether clustered entities are the same thing. Merge them into a canonical node with aliases stored as properties.

async function mergeEntityCandidates(
  candidates: Array<{ id: string; name: string; type: string }>,
  embeddings: number[][]
): Promise<Map<string, string>> {
  // Returns a map of { entityId -> canonicalId }
  const mergeMap = new Map<string, string>();
  const threshold = 0.92;

  for (let i = 0; i < candidates.length; i++) {
    if (mergeMap.has(candidates[i].id)) continue;
    mergeMap.set(candidates[i].id, candidates[i].id);

    for (let j = i + 1; j < candidates.length; j++) {
      if (mergeMap.has(candidates[j].id)) continue;
      if (candidates[i].type !== candidates[j].type) continue;

      const similarity = cosineSimilarity(embeddings[i], embeddings[j]);
      if (similarity > threshold) {
        // High similarity same-type entities: prompt LLM to confirm
        const confirmed = await confirmSameEntity(candidates[i], candidates[j]);
        if (confirmed) {
          mergeMap.set(candidates[j].id, candidates[i].id);
        }
      }
    }
  }

  return mergeMap;
}

function cosineSimilarity(a: number[], b: number[]): number {
  const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dot / (magA * magB);
}

Writing to Neo4j

import neo4j from "neo4j-driver";

const driver = neo4j.driver(
  process.env.NEO4J_URI!,
  neo4j.auth.basic(process.env.NEO4J_USER!, process.env.NEO4J_PASSWORD!)
);

async function upsertGraph(extraction: ExtractionResult, documentId: string) {
  const session = driver.session();
  try {
    await session.executeWrite(async (tx) => {
      // Upsert entities
      for (const entity of extraction.entities) {
        await tx.run(
          `MERGE (n:${entity.type} { id: $id })
           SET n += $properties, n.name = $name, n.updatedAt = datetime()`,
          { id: entity.id, name: entity.name, properties: entity.properties }
        );
      }

      // Upsert relations with document provenance on the edge
      for (const rel of extraction.relations) {
        await tx.run(
          `MATCH (a { id: $from }), (b { id: $to })
           MERGE (a)-[r:${rel.predicate}]->(b)
           SET r += $properties, r.sourceDocument = $documentId, r.updatedAt = datetime()`,
          {
            from: rel.from,
            to: rel.to,
            properties: rel.properties,
            documentId,
          }
        );
      }
    });
  } finally {
    await session.close();
  }
}

Hybrid Retrieval: Vector + Graph

The model that works in practice is not vector-or-graph. It is vector-then-graph.

1. Run the user query through your vector store to find semantically relevant chunks.
2. Extract the entities mentioned in those chunks (or in the original query).
3. Use those entities as entry points into the graph.
4. Traverse relevant relationships to collect structured facts.
5. Combine both result sets in the prompt context.

import { pgvector } from "@/lib/pgvector"; // your vector store client
import { driver } from "@/lib/neo4j";

interface RetrievalContext {
  vectorChunks: Array<{ content: string; score: number; documentId: string }>;
  graphFacts: Array<{ subject: string; predicate: string; object: string; properties: Record<string, string> }>;
}

async function hybridRetrieve(
  query: string,
  queryEmbedding: number[],
  topK = 5
): Promise<RetrievalContext> {
  // Step 1: Vector retrieval
  const vectorChunks = await pgvector.query({
    embedding: queryEmbedding,
    limit: topK,
    minScore: 0.72,
  });

  // Step 2: Extract entity mentions from query + top chunks
  const combinedText = [query, ...vectorChunks.map((c) => c.content)].join("\n");
  const mentionedEntityIds = await extractEntityMentions(combinedText);

  if (mentionedEntityIds.length === 0) {
    return { vectorChunks, graphFacts: [] };
  }

  // Step 3: Graph traversal from entity entry points
  const session = driver.session();
  try {
    const result = await session.run(
      `MATCH (n)-[r]->(m)
       WHERE n.id IN $entityIds OR m.id IN $entityIds
       RETURN n.name AS subject, type(r) AS predicate, m.name AS object,
              properties(r) AS props
       LIMIT 50`,
      { entityIds: mentionedEntityIds }
    );

    const graphFacts = result.records.map((record) => ({
      subject: record.get("subject"),
      predicate: record.get("predicate"),
      object: record.get("object"),
      properties: record.get("props"),
    }));

    return { vectorChunks, graphFacts };
  } finally {
    await session.close();
  }
}

Building the Prompt Context

function buildContext(retrieval: RetrievalContext): string {
  const parts: string[] = [];

  if (retrieval.vectorChunks.length > 0) {
    parts.push("## Relevant Document Passages");
    for (const chunk of retrieval.vectorChunks) {
      parts.push(`Source: ${chunk.documentId}\n${chunk.content}`);
    }
  }

  if (retrieval.graphFacts.length > 0) {
    parts.push("## Structured Relationships");
    for (const fact of retrieval.graphFacts) {
      const props = Object.entries(fact.properties)
        .map(([k, v]) => `${k}=${v}`)
        .join(", ");
      parts.push(
        `${fact.subject} --[${fact.predicate}${props ? ` (${props})` : ""}]--> ${fact.object}`
      );
    }
  }

  return parts.join("\n\n");
}

The Microsoft GraphRAG Pattern

Microsoft’s GraphRAG (released in 2024) takes a different angle. Instead of storing raw chunks in a vector store, it pre-processes the entire corpus into a community hierarchy: the graph is clustered into communities, each community gets a summary generated by an LLM, and those summaries are what gets retrieved.

The query flow has two modes:

• Local search: find the most relevant entities, pull their neighbors and community summaries, answer from that context.
• Global search: use community summaries at the top of the hierarchy to answer broad thematic questions that no single passage would support.

The tradeoff is cost. GraphRAG requires significant upfront LLM calls to generate community summaries, and the graph must be rebuilt when the corpus changes substantially. It is best suited for relatively stable corpora where you need strong thematic coverage (think: an entire product documentation site, a legal corpus, or a company wiki).

For dynamic or frequently updated corpora, the incremental graph construction approach described above is more practical.

Query Planning: When to Hit the Graph

Not every query needs graph traversal. Running Cypher on every request adds latency and is often unnecessary.

A simple classifier decides:

type QueryIntent =
  | "factual_lookup"      // "What is the default timeout for the payments service?"
  | "relational"          // "What depends on the auth service?"
  | "temporal"            // "What changed between v1 and v2?"
  | "thematic";           // "Summarize how we handle error recovery"

async function classifyQueryIntent(query: string): Promise<QueryIntent> {
  const response = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content: `Classify the query intent as one of: factual_lookup, relational, temporal, thematic.
- relational: asks about connections, dependencies, relationships between entities
- temporal: asks about changes over time, versions, sequences
- thematic: broad summarization, overview questions
- factual_lookup: everything else

Return only the classification string.`,
      },
      { role: "user", content: query },
    ],
  });

  return (response.choices[0].message.content?.trim() ?? "factual_lookup") as QueryIntent;
}

async function retrieve(query: string, embedding: number[]) {
  const intent = await classifyQueryIntent(query);

  if (intent === "relational" || intent === "temporal") {
    return hybridRetrieve(query, embedding);
  }

  // Vector-only for factual and thematic
  const chunks = await pgvector.query({ embedding, limit: 5 });
  return { vectorChunks: chunks, graphFacts: [] };
}

This keeps the common case fast and only invokes graph traversal when the query structure warrants it.

Tradeoffs

Dimension	Vector-Only RAG	Hybrid RAG (Vector + Graph)	GraphRAG (Microsoft pattern)
Retrieval quality (factual)	Good	Good	Good
Retrieval quality (relational)	Poor	Strong	Strong
Retrieval quality (thematic)	Moderate	Moderate	Excellent
Ingest cost	Low	Moderate (extraction LLM calls)	High (community summaries)
Query latency	Low (5-50ms)	Moderate (50-200ms)	Moderate to High
Corpus update cost	Low	Moderate (incremental)	High (rebuild summaries)
Operational complexity	Low	Moderate	High
Contradiction handling	None	Provenance on edges	Partial (community aggregation)

Production Considerations

Graph maintenance and drift. Documents change. When a document is updated, you need to retract the triples extracted from its previous version before ingesting the new one. Store sourceDocument on every edge (as shown above) so you can query MATCH ()-[r]-() WHERE r.sourceDocument = $id DELETE r before re-extracting. Without this, you accumulate stale edges that silently corrupt answers.

Extraction quality degrades at scale. LLM entity extraction is not 100% accurate. At 10,000 documents, extraction errors compound. Build an evaluation set of 50-100 documents with manually verified graphs and track extraction precision and recall on each pipeline update. Do not assume accuracy is stable.

Graph schema evolution. New entity types and predicates emerge as your corpus grows. Design your graph schema loosely (avoid hard constraints on node labels) and use a migration script when you need to rename or merge predicate types. Cypher’s APOC library has utilities for bulk predicate renaming.

Cost. Extraction at scale means LLM calls per document chunk. At gpt-4o-mini rates, extracting from a 10,000-document corpus costs roughly $5-15 depending on average document length. Budget for re-extraction on corpus updates. Consider batching chunks from the same document into a single extraction call to reduce per-call overhead and improve cross-sentence relation extraction.

Observability. Log graph traversal depth and result count on every hybrid query. A traversal that returns zero graph facts means either your extraction missed the relevant entities or your query classifier is misfiring. Both are worth alerting on separately.

When Knowledge Graphs Are Overkill

If your corpus is relatively flat (support articles, FAQs, product documentation without deep interdependencies), a knowledge graph adds engineering complexity without meaningful retrieval improvement. Add one when:

• Users regularly ask multi-hop questions that require connecting facts across documents.
• Your corpus has well-defined entities whose relationships are load-bearing for correct answers.
• You need to track provenance or resolve contradictions between sources.
• Temporal reasoning over versioned entities is a first-class requirement.

For everything else, invest in better chunking, hybrid BM25 + vector search, and a reranker. Those improvements will move recall metrics further per hour of engineering time than a graph will.

---

Vector similarity is the right retrieval primitive for most RAG workloads. Knowledge graphs are the right extension when your users’ questions are fundamentally relational and the answer requires traversal, not pattern matching. The two are not competing approaches. Used together, with a query classifier routing to the appropriate retrieval path, they cover a much wider class of questions than either can handle alone.