Graph RAG & Knowledge Graphs

Graph RAG is a retrieval architecture that uses a knowledge graph instead of (or in addition to) a vector database to find relevant information. Where standard RAG measures the semantic similarity between a query embedding and document embeddings, Graph RAG traverses the edges of a graph -- hopping from entity to entity across explicit, labeled relationships -- to retrieve facts that are structurally connected rather than merely topically similar.

A knowledge graph stores three things: nodes (entities such as people, organizations, drugs, statutes, products), edges (typed relationships such as WORKS_AT, ACQUIRED, INTERACTS_WITH, CITES), and properties (attributes on either, such as a date, a confidence score, or a source chunk ID). Retrieval becomes a graph traversal problem: starting from a seed entity mentioned in the question, you follow edges outward to collect connected entities and the document chunks that mention them.

This is the core distinction. Vector RAG asks: “Which chunks are closest to my question in embedding space?” Graph RAG asks: “Which entities are connected to the entities in my question, and how?” The two answer fundamentally different questions, and the most effective production systems combine them.

Vector retrieval has a known, well-documented weakness: it finds documents that look similar to the query but has no concept of how the entities inside those documents relate to one another. This breaks down on multi-hop questions -- questions whose answer requires chaining facts across two, three, or more documents that may not even be semantically similar to the question.

Consider this question:

The answer requires four hops:

1. Identify the tool we use (a product entity in your inventory).
2. Find the startup that built that tool (a BUILT_BY edge).
3. Find the company that acquired that startup (an ACQUIRED edge).
4. Find the CEO of that company (a HAS_CEO edge).

A vector database will return chunks that mention tools, startups, acquisitions, and CEOs -- but it cannot guarantee that the specific tool, startup, acquirer, and CEO are the same connected chain. It returns topical hits, not a traversed path. The LLM is then left to reconstruct the chain from context, which frequently hallucinates or picks the wrong entity when several candidates are present.

A knowledge graph answers this directly. Each hop is a single edge traversal. In Cypher (Neo4j's query language) the whole question is a four-line pattern match that returns the exact person, with the exact chain of evidence, in one query. No guessing, no hallucination, no reliance on the model to stitch fragments together.

The practical consequence is that Graph RAG dramatically improves accuracy on questions that involve entities, relationships, timelines, and chains of inference -- which in domains like medicine, law, finance, and enterprise intelligence is most of the valuable questions.

A knowledge graph is a data structure that represents the world as entities and the relationships between them. Three primitives make up the entire model:

A single fact is expressed as a triple -- a subject node, a predicate (edge type), and an object node -- optionally decorated with properties. The triple [Person:Steve] -[WORKS_AT]-> [Company:Avondale.AI] asserts that Steve works at Avondale.AI. Add the property since: 2024-03-15 and you know when he started. Add source_chunk: 842 and you can trace the claim back to the exact document chunk the extractor read it from.

A small knowledge graph built from a few sentences might look like this:

Five facts, five edges. From Person:Steve you can traverse to every tool the company uses, find where it is hosted, and enumerate every service -- all by following edges. This is what a vector database fundamentally cannot do, because it has no edges.

Your documents are unstructured text. Your knowledge graph is structured triples. The gap between them is bridged by a four-stage pipeline: extract entities, extract relations between them, resolve duplicates into single canonical nodes, and persist the result in a graph database.

NER scans text and identifies spans that name entities, classifying each into a type. Given “Avondale.AI deployed a Neo4j graph at their Phoenix office in March 2024,” a NER model returns: Avondale.AI (ORG), Neo4j (PRODUCT), Phoenix (LOC), March 2024 (DATE).

Three families of extractors dominate on-premise RAG builds:

• spaCy with a trained ner pipe -- fast, deterministic, fully local, no GPU required. Trained on standard types (PERSON, ORG, GPE, DATE, MONEY). Best for high-throughput extraction over large corpora where precision matters more than novel entity types.
• GLiNER -- a zero-shot NER model. You pass candidate labels at inference time (“drug, dosage, ICD-10 code, adverse event”) and it extracts spans matching those types without retraining. Excellent for domain-specific graphs where the entity types are not in spaCy's defaults.
• LLM-based extraction -- prompt a local model (Llama 3, Qwen, Mistral) to return JSON entities from each chunk. Most flexible, slowest, most expensive in compute, but handles novel types and nuanced definitions out of the box. Pair with a schema (Pydantic, JSON Schema) so the model returns structured, validated output.

NER gives you nodes. Relation extraction gives you edges: given a pair of entities in a sentence or window, what typed relationship connects them? Two approaches dominate:

• Rule-based -- dependency-parse the sentence and match patterns. “X was acquired by Y” always yields Y -[ACQUIRED]-> X. Deterministic, auditable, zero compute cost beyond parsing, but brittle and limited to patterns you hand-wrote. Strong baseline for closed domains (legal citations, SEC filings) where phrasing is formulaic.
• LLM-based -- pass the chunk and a schema of allowed edge types to a local model and ask it to return triples as JSON. Captures nuance, novel relationships, and implicit links the rule engine would miss. Slower and costlier, but the only practical option for free-form prose like clinical notes or contracts.

The output of either is a set of triples: (Person:Steve, WORKS_AT, Company:Avondale.AI), (Company:Avondale.AI, OFFERS, Service:RAG), each tagged with the source chunk so the relationship is traceable to its origin.

The same entity appears under many surface forms. “Avondale.AI” in one chunk, “Steve” in another, “S. Sargent” in a signature block, “Mr. Sargent” in a memo. Without resolution, your graph ends up with four nodes for one person, and traversal breaks -- the path from Steve to Avondale.AI exists, but the path from S. Sargent does not, because they are stored as different nodes.

Entity resolution (also called deduplication or record linkage) merges these into a single canonical node. Techniques, in order of sophistication:

• String similarity -- Levenshtein distance, Jaro-Winkler, or cosine over character n-grams. Cheap, catches typos and abbreviations, misses semantic aliasing.
• Embedding similarity -- embed each entity mention and cluster by cosine distance. Catches “the CEO” = “Avondale.AI” when both appear in the same context window.
• Contextual resolution -- pass candidate pairs to an LLM with surrounding context and ask whether they refer to the same entity. Highest accuracy, highest cost.
• Rules + dictionaries -- a lookup of known aliases (“IBM” = “International Business Machines”) for deterministic merges.

The merged node keeps all surface forms as aliases and all edges from all duplicates. Traversal now works regardless of which form the question uses.

The final stage persists the resolved triples in a graph database. Each node and edge becomes a row (or its native equivalent) with its properties, including the all-important source_chunk back-reference. The graph database provides the query language and traversal engine that Graph RAG retrieves against.

For sovereign RAG, the database runs on your hardware. The options are covered in detail in the graph databases section below, but the short version: Neo4j Community Edition (self-hosted, Cypher), ArangoDB (multi-model, self-hosted), Postgres with the Apache AGE extension (no new database -- add graph to your existing Postgres), NebulaGraph (distributed), or even DuckDB with graph extensions for a fully embedded, serverless option.

Neither pure vector RAG nor pure graph RAG is the right answer for most real systems. Vector retrieval is unbeatable at “find me documents about X” -- open-ended, semantic, exploratory. Graph retrieval is unbeatable at “how are X and Y connected?” -- structured, relational, multi-hop. Production systems run both, in a two-phase pipeline that uses each for what it does best.

Phase 1 (vector) finds the starting point. You embed the question, search the vector database, and get back the chunks that are semantically closest -- and the entities those chunks mention. This gives the system the entry points into the graph without requiring the user to name exact entities.

Phase 2 (graph) expands from those seed entities. For each one, traverse N hops outward along typed edges, collecting every connected entity and the source chunks that mention the traversed edges. This brings in documents that are structurally relevant even if they are not semantically similar to the query -- the missing acquisition filing, the contract clause referenced by a precedent, the drug-interaction warning that only appears when you traverse from the prescribed drug to its interaction class.

The two chunk sets are merged, de-duplicated, and re-ranked (by a cross-encoder, by graph-distance weighting, or by an LLM judge), then handed to the LLM. The LLM now sees both the topically relevant passages and the structurally connected ones, and can answer multi-hop questions correctly.

Re-ranking is where the two signals are fused. A common weighting scheme:

The weights are a tunable knob. Entity-heavy domains (legal, medical) skew toward the graph term; exploratory Q&A over a knowledge base skews toward the vector term. There is no universal optimum -- measure recall on your own evaluation set.

In 2024 Microsoft Research released GraphRAG, an open-source framework that introduced a now-influential idea: do not just store entities and edges -- cluster them into communities, summarize each community, and retrieve against the summaries for global questions. It splits retrieval into two distinct modes that answer different question types.

The pipeline has four steps:

Retrieval then splits:

• Global search walks the top-level community summaries, asks each for a partial answer to the question, and map-reduces those into a final synthesized answer. This is how GraphRAG answers “what are the main themes across the whole corpus” -- a question that vector RAG cannot answer at all, because vector retrieval has no concept of corpus-level structure.
• Local search finds the entities in the question, pulls the chunks connected to those entities, and also pulls the community summaries for context -- giving the LLM both the specific facts and the surrounding thematic frame.

The knowledge graph needs a home. For sovereign RAG that home is on your hardware, under your control, with no telemetry phoning home. Six credible options, ranging from industry-standard server databases to embedded engines that need no server at all.

A few practical notes on each:

Neo4j is the safe default. Cypher is readable -- MATCH (p:Person)-[:WORKS_AT]->(c:Company) RETURN p, c is self-explanatory -- and the ecosystem (drivers, visualizers, LLM integrations) is the deepest. The Community Edition covers everything a single-node sovereign RAG deployment needs. The Enterprise Edition's extras (clustering, RBAC, backups) matter at scale, but you can run a long time on Community first.

ArangoDB is the “one database to rule them all” option. It stores graph edges and document nodes in the same engine, plus it has a built-in full-text and (experimentally) vector search. If you want to avoid running separate vector, document, and graph stores, ArangoDB collapses them into one process. The cost is that each individual capability is slightly less polished than a single-purpose database.

Postgres + Apache AGE is the “no new database” option and the one we highlight for teams already running Postgres. AGE (A Graph Extension) adds openCypher support as a Postgres extension -- LOAD 'age'; SET graph_path = 'mygraph'; SELECT * FROM cypher('mygraph', $$ MATCH (p:Person) RETURN p $$) AS (p agtype); You get graph queries, and your existing Postgres backups, replication, monitoring, and pgvector embeddings all keep working. One database, three retrieval modes (SQL, Cypher, pgvector), zero new ops burden.

NebulaGraph is the scale-out pick. When your graph exceeds a single machine's memory -- tens of billions of edges, social-scale or enterprise-wide graphs -- NebulaGraph shards the graph across a cluster. The trade-off is operational complexity: you are now running a distributed system with multiple services (graphd, storaged, metad).

RedisGraph is the speed pick. In-memory, sub-millisecond traversals, Cypher interface. Check the license terms (Redis moved modules to RSALv2 / SSPLv1 in recent versions) against your deployment model. Best when the graph is small, the query volume is high, and you are already running Redis for caching.

DuckDB is the embedded pick. No server process -- your Python application opens a file and queries it. DuckDB does not have a native graph type, but its recursive CTEs perform bounded-depth traversals over an edge table efficiently, and for single-process sovereign RAG (a desktop app, a single-user research tool, an on-device medical assistant) it is hard to beat the operational simplicity. Pair with DuckDB's native vector similarity (via the vss extension) and you have vector + graph in one embedded file with no services to run.

Not every workload needs a graph. The decision is driven by the shape of the questions your users actually ask.

A few heuristics that make the call concrete:

• Skip the graph if your documents are a flat knowledge base with no named entities worth modeling (a product manual, an FAQ, a set of marketing pages). Vectors are simpler and answer every question you have.
• Build the graph if your users ask about people, organizations, products, contracts, cases, or drugs -- and especially if they ask how those things connect, depend on, cite, or affect each other.
• Use the hybrid if you have both question types, or if your graph is sparse enough that pure graph retrieval sometimes returns nothing for a reasonable question that vectors would catch.
• Use Microsoft GraphRAG-style community summarization if users ask corpus-wide thematic questions (“what are the main themes?”) that neither vectors nor local graph traversal can answer.

Medicine is the canonical relationship-heavy domain. A patient is not a document; a patient is a timeline of encounters, each with diagnoses, prescriptions, lab results, procedures, and providers, all of which relate to each other in clinically meaningful ways. A vector database can find notes about a condition. A knowledge graph can answer “given this patient's medication list, which of these proposed drugs is contraindicated?” -- because it can traverse from each prescribed drug to its interaction class and check for overlap.

Three patterns where graph RAG earns its keep in clinical settings:

1. Patient Journeys

A patient journey is a temporal graph: [Patient] -[HAD_ENCOUNTER]-> [Encounter] -[DIAGNOSED_WITH]-> [Condition] -[TREATED_WITH]-> [Medication] -[CAUSED]-> [AdverseEvent]. Traversing this graph reconstructs the full clinical narrative -- every diagnosis, every intervention, every outcome -- in chronological and causal order. A query like “show me every patient who started Drug A, later developed Condition B, and was then switched to Drug C” is a single graph traversal. In a vector store it is effectively impossible.

2. Treatment Pathways & Guidelines

Clinical guidelines are themselves graphs: first-line therapy, second-line if first-line fails or is contraindicated, alternatives for specific comorbidities. Modeling the guideline as a graph and the patient's current state as a subgraph lets the system traverse from “where this patient is now” to “what the guideline recommends next,” with the contraindication checks expressed as edges to avoid.

3. Drug-Drug Interactions

Drug interaction knowledge bases (DrugBank, RxNorm) are natively graph-structured. Each drug has metabolism, targets, and interaction edges to other drugs. The question “is Drug X safe alongside this patient's current regimen?” is a traversal from Drug X through its interaction edges to each drug the patient already takes, checking severity and mechanism. This is the textbook case where graph retrieval is not just better than vector retrieval -- it is the only correct tool.

Law is another relationship-native domain. Cases cite cases, which cite earlier cases, forming a directed acyclic graph of precedent. Statutes reference other statutes. Contracts reference parties, subsidiaries, governing-law jurisdictions, and other contracts. Every one of these is an edge, and answering legal questions correctly requires traversing them, not just finding documents that mention the same keywords.

Case Citations & Precedent Chains

A case opinion cites prior opinions to support its reasoning. Those prior opinions cite still earlier ones. The result is a citation graph that can be many layers deep. “Find every case that relied on Smith v. Jones and was later overturned” is a traversal: start at the Smith node, walk CITED_BY edges outward to collecting cases, then filter those by a OUTCOME: overturned property. Vector retrieval will surface cases that mention Smith in passing; only the graph surfaces the ones whose reasoning depends on Smith and were later reversed.

Statute Cross-References

Statutes routinely incorporate other statutes by reference -- “as defined in Section 14-102,” “subject to the requirements of Chapter 7.” These references are edges. Traversing them reconstructs the full operative text of a provision, pulling in every incorporated definition and requirement, so the LLM sees the complete legal picture rather than an isolated fragment that silently omits a cross-referenced obligation.

Contract Entity Networks

A contract names parties, their subsidiaries, governing jurisdictions, referenced agreements, and termination triggers. Modeling these as a graph lets the system answer “if Subsidiary X is sold, which of our 200 contracts have a change-of-control clause that triggers?” by traversing from the entity through PARTY_TO edges to contracts, then checking each contract node for a HAS_CLAUSE: change_of_control property. This is due diligence at a speed and completeness no manual review can match.

A consolidated comparison of the three retrieval approaches, the tools that support them, and how each maps to a sovereign, on-premise deployment.

The end-to-end architecture of a sovereign, on-premise hybrid Graph RAG system. Everything in the diagram runs on hardware you control. No external API calls. No data leaving the premises.

Two pipelines, one stack. The indexing pipeline runs offline as a batch job -- chunk, embed, extract entities and relations, resolve duplicates, build the graph, optionally run community detection. The retrieval pipeline runs per query -- vector search to find seeds, graph traversal to expand, re-rank, generate. Both pipelines share the same document store, and in the Postgres + pgvector + AGE variant they share a single database process.