Why a Personal Knowledge Graph Now
If you keep notes, save links, and handle a busy inbox, you already have a rich archive of your work and ideas. The problem is that search still feels blunt. You can find a word you typed, but not the connections that matter: who introduced you to a topic, which files tie to a decision, or how a theme has evolved across months.
A personal knowledge graph fixes that. Instead of treating your data as loose files and messages, it treats them as entities (people, projects, tools, places) and relationships (authored, mentioned, depends on, duplicates, scheduled with). Pair that graph with modern language models and you get GraphRAG: a retrieval approach that mixes graph structure and semantic search, so answers are drawn from the right context with traceable sources.
This guide is practical, local-first, and privacy-conscious. You will learn a clear plan for building a home-grade graph that runs on your laptop or NAS, scales to real life, and stays under your control.
What a Personal Knowledge Graph Actually Is
Think of a knowledge graph as a structured memory for your digital life. Each item you capture becomes one or more nodes (entities) with edges (relationships) connecting them. A document links to a project; an email links to a person; a bookmark links to a theme.
Core building blocks
- Entities: People, projects, topics, organizations, documents, tasks, meetings, places, tools, and events.
- Relationships: AuthoredBy, Mentions, RefersTo, DependsOn, DuplicateOf, PartOf, ScheduledAt, LocatedIn.
- Provenance: Where a fact came from (file path, email ID, URL), when it was observed, and which process or model extracted it.
- Confidence: A score indicating how sure you are about any fact. You can adjust thresholds, review flags, and avoid polluting your graph.
With these pieces, you move beyond “Where did I store that file?” toward questions like “Which documents mention supplier costs and were last edited after our January meeting?”
Architecture: From Raw Text to Useful Answers
The full stack is simple enough for a home setup. It looks like this:
- Connectors import from your note app exports (Markdown), bookmarks (HTML or JSON), and email (IMAP or .mbox files).
- Parsers clean and normalize text: strip signatures, collapse quoted replies, normalize headings and lists.
- Naming uses lightweight NLP to extract entities and map aliases to canonical IDs.
- Graph store holds entities, edges, attributes, timestamps, and provenance.
- Embedding store holds vector representations of passages to enable semantic retrieval.
- Query engine (GraphRAG) blends graph traversal and semantic search before drafting answers with a local or small hosted LLM.
Connectors you can implement
Notes: Export Markdown (most apps support it), store a copy in your graph project, and track file hashes for change detection. Keep frontmatter metadata (tags, dates) if present.
Bookmarks: Export to HTML from your browser or use a read-it-later app’s JSON export. Extract title, URL, description, tags, and folder path. Try to capture the first seen timestamp from file metadata.
Email: If you use a local client, parse its store. If you prefer server-side, fetch via IMAP and store only selected folders (e.g., “Projects” and “Receipts”). Keep message-ID, thread-ID, from/to/cc, subject, dates, and a cleaned body.
Parsing and cleaning
Good parsing prevents most headaches later. Remove quoted layers in email replies, avoid indexing signature blocks and disclaimers, normalize weird Unicode, and trim boilerplate like cookie banners in saved web pages. These small steps lift signal and cut wrong extractions.
Entity and relation extraction
Run a local NLP pipeline for named entity recognition and simple relation hints. For many people, spaCy with a medium model is the right starting point. You can later add a small quantized LLM with llama.cpp to improve extraction on tricky sentences, but start basic.
Maintain an alias map: “JS,” “John S.,” and “John Smith” point to one canonical person ID. The same for projects: “Hydra,” “Hydra v2,” “Hydra Rework” become a single project node. Store the alias mapping in the graph so you can refine it over time.
Choosing a graph store
- Property graph (Neo4j, Memgraph): Friendly query language (Cypher), easy to add attributes to nodes and edges, great for home-scale data.
- RDF triple store (Apache Jena): Standards-based, strong for interoperability and linked data, queries in SPARQL.
- SQLite schema (adjacency tables): If you want dead-simple portability and backups, a pair of tables for nodes and edges in SQLite works too.
Pick the one you can back up and visualize. For most people, Neo4j Community or SQLite is a good fit. You can always migrate later using CSV exports.
Embedding store for semantic search
Semantic search finds relevant passages when keywords fail. For local-first setups, use an on-device embedding model (e.g., small sentence transformer) and index vectors with FAISS or hnswlib. If you prefer a simpler route, index plain text with SQLite FTS5 first, then add vectors later.
GraphRAG: the hybrid retrieval loop
Most “RAG” examples use only vectors. GraphRAG adds structure constraints so the model sees the right snippets and avoids making things up. A good query loop looks like this:
- Step 1: Expand the query with a tiny LLM prompt to guess likely entities and relationships (no answering yet).
- Step 2: Traverse the graph to fetch entity cards and linked documents that match the predicted relationships.
- Step 3: Retrieve passages from those documents with semantic search. Add provenance and scores.
- Step 4: Draft an answer with a local LLM, citing sources and offering a “view in graph” link for each citation.
This structure keeps hallucinations in check and returns answers you can verify.
Schema That Won’t Paint You Into a Corner
Lightweight schemas are sustainable. You want flexibility without entropy. Start with a small set of node labels and edge types, then extend carefully.
Minimal viable schema
- Nodes: Person, Project, Topic, Document, Message, Event, Organization, Tool.
- Edges: AUTHORED_BY, MENTIONS, REFERS_TO, PART_OF, DEPENDS_ON, DUPLICATE_OF, SCHEDULED_AT, MEMBER_OF.
- Attributes:
- All nodes: id, title/name, created_at, updated_at, source_uri, source_type.
- Confidence fields: extraction_confidence on edges and nodes added by the pipeline.
- Aliases: alias list on nodes, or ALIAS_OF edges between nodes.
Keep compatibility with both property graphs and RDF by avoiding exotic data types. Timestamps, strings, numbers, and lists are fine almost everywhere.
Version and migrate safely
Graph schemas drift as your life changes. Add a schema_version node with key upgrade notes. When you add a new edge type, write a small migration script that tags old edges with the new type or leaves them alone with a fallback rule. Avoid massive rewrites unless you must.
Quality Control Without the Drama
Small errors compound in graphs. Fix them early with a few habits.
- Human review queues: Any extraction with confidence below a threshold goes to a triage list. You can approve, merge, or dismiss in a minute a day.
- Coreference checks: Periodically list near-duplicate entities by string similarity. Merge them with one click and record the alias that caused confusion.
- Contradiction flags: If two edges claim “MemberOf” for a person with conflicting dates, flag both and ask for human input.
- Provenance-first design: Every fact shows you exactly which sentence in which file created it. No opaque magic.
Privacy and Security That Fit Home Setups
Your graph holds sensitive details. Process on-device by default. If you must use a cloud model, strip content and send only short, non-identifying phrases. A few pointers:
- Encryption at rest: If your graph runs on a laptop, use full-disk encryption. On a NAS, enable volume encryption where available.
- Secret handling: Put IMAP credentials and API keys in OS keychain or a password manager. Never hardcode in scripts.
- Audit logging: Record when connectors run, what they import, and which files they touch. Troubleshooting becomes easy.
- Local-only LLMs: Start with a 3B–7B model for summarization and extraction. Only upgrade sizes if latency or quality demands it.
Queries You Can Actually Ask
Structure unlocks questions keyword search can’t answer. Try these:
- “Who introduced me to the ‘Atlas’ tool, and which projects used it afterward?”
Graph: Find the first Message or Document mentioning “Atlas,” get the Person associated, then list Projects connected by REFERS_TO after that date. - “Collect every argument we’ve used to choose vendor K and summarize the pros and cons.”
Graph: Traverse from Organization K to linked Documents tagged as decision notes; RAG: pull passages and summarize with citations. - “What did we promise in meetings about ‘Hydra’ in Q2?”
Graph: EVENTS tagged as “Meeting” linked to Project=Hydra in Q2; RAG: passage extraction from minutes. - “Which bookmarks did I save the week I drafted the first ‘beta plan’?”
Graph: Time-bound filter around Document=Beta Plan created_at; join to Bookmarks by created_at week.
You do not need fancy UIs to start. A small search box with saved query templates can deliver most of the value.
Performance: Smooth on Modest Machines
On a typical home dataset (tens of thousands of messages, a few thousand notes and bookmarks), performance is tractable:
- Batch imports at night: Run connectors once a day. Index updates in batches to reduce churn.
- Lazy embeddings: Only embed new or changed documents. Use a small model and quantization to keep CPU load low.
- ANN indexes: FAISS or hnswlib accelerate vector search. If you start with FTS5, add ANN later without changing your flow.
- Cache query results: LLM calls often repeat across similar questions. Add a disk-backed cache keyed by prompt hash.
Interfaces People Use
Graph hairballs look cool but are rarely helpful. Prefer list and card views:
- Entity cards that show a concise profile: attributes, top connections, and most-cited sources.
- Timeline pivots: Scrollable lists filtered by date, with a “jump to week” control for fast recall.
- Query templates with simple parameters (person, project, date range) instead of free-form grammar-heavy input.
- Citation popovers in answers: every claim has a one-click reveal for the source passage.
Habit Loops That Keep It Alive
A good knowledge graph is more habit than software. Set small routines:
- Daily triage: One minute to approve or merge low-confidence items.
- Capture friction: One hotkey to save a link with a short note. Tag it with a project on capture to avoid guesswork later.
- Weekly digest: Automated summary: new entities, projects that changed a lot, and open contradictions to fix.
- Review hooks: If a node goes “stale” (no updates for 90 days), schedule a reminder or an archival pass.
Measuring If It’s Working
Keep score to know whether your graph is helping or just growing:
- Question success rate: Keep a list of 20 practical questions. Track how often you get a useful answer in under a minute.
- Precision at top-k: For a few known topics, label the right documents and measure how often they show up in the first 5 results.
- Entity resolution F1: For a small set of people and projects, check alias merges. Aim to improve month by month.
- Answer provenance coverage: Ensure at least 90% of answers include two or more citations.
Seven Days to a Working Graph
Day 1: Set the target
Pick three questions you want to answer every month. These will guide schema and quality choices. Examples: “What changed on Project Hydra last week?” “All notes related to vendor K pricing.”
Day 2: Install the basics
Choose a graph store (Neo4j or SQLite) and spin up a simple UI (Neo4j Browser or a minimal web app). Create tables or labels for core entities and edges.
Day 3: Import notes
Export Markdown, parse headings, frontmatter, and body. Create Document nodes with created_at and updated_at. Add MENTIONS edges from regex-tag matches (e.g., #Hydra).
Day 4: Import bookmarks
Parse the HTML export, create Document nodes with URL, tags, and folder path. Use tags to seed Topic nodes. Link bookmarks to Topics with REFERS_TO.
Day 5: Import email
Pull a limited folder, e.g., Projects. Clean quoted replies, strip signatures, and create Message nodes. Link to Person nodes via from/to/cc, creating people as needed.
Day 6: Add entity extraction
Run spaCy to pull new entities from content. Merge aliases manually for the first pass. Store extraction_confidence with each new item.
Day 7: Wire up GraphRAG
Index text in FTS5 or FAISS. Build a small query loop: expand, traverse, retrieve, answer. Test with your three target questions. Adjust thresholds and templates.
Practical Patterns for GraphRAG Prompts
Expansion prompt
Start with a prompt that asks for entities and relations, not answers. For example: “Given the question, list likely entities (people, projects, topics) and relations (MENTIONS, REFERS_TO, DEPENDS_ON) that would help find evidence. Do not answer.”
Answer prompt
After retrieval, ask the model: “Write a concise answer. Only use the provided passages. Cite source IDs in parentheses. If unsure, say so and list what is missing.”
These constraints keep output faithful and keep the graph as the source of truth.
Edge Cases and How to Handle Them
- Names everywhere: Common names produce false merges. Require a second attribute (email, org, or co-mention) before merging People.
- Email signatures: They pollute extraction. Keep a short blocklist of signature patterns and drop lines that match.
- Non-English text: Use multilingual models for embeddings and NER where needed. Or tag language per document and route to the right model.
- Long PDFs: Chunk by headings or pages. Avoid random-length splits; they reduce retrieval quality.
- Threaded messages: Preserve thread IDs. You can infer a workflow by traversing messages in a thread across dates.
Backups, Portability, and Longevity
Your graph will outlast any one tool. Keep it portable.
- Export regularly: CSV for nodes and edges, plus a zip of raw files and embeddings. Even if you change databases, you keep meaning intact.
- Snapshot embeddings: They are costly to recompute. Keep a versioned store with a model name in metadata.
- Write once, read many: Separate importer scripts from the graph store. If you move from Neo4j to SQLite, your importers barely change.
Where This Can Go Next
Once your base graph works, you can add useful refinements:
- Task edges from your to-do app to link commitments to documents and people.
- Calendar integration so Events anchor work bursts to times and attendees.
- Feedback loop: Answers that got “thumbs up” boost edge weights. Bad answers lower them automatically.
- Small team mode: Merge graphs across two or three people on a shared NAS, with per-node visibility flags and merge requests.
Keep it humble. The best systems are ones you actually review and trust. Your graph should be helpful before it is fancy.
Summary:
- A personal knowledge graph turns your notes, bookmarks, and emails into entities and relationships with provenance and confidence.
- Use a simple architecture: connectors, parsers, NLP extraction, a graph store, an embedding index, and a GraphRAG query loop.
- Start with a minimal schema and evolve it slowly to avoid migrations that break history.
- Quality control matters: review low-confidence items, merge aliases, and show exact sources for every fact.
- Keep privacy by processing on-device, encrypting at rest, and using local LLMs for extraction and answers where possible.
- Measure success by practical question coverage, precision at top-k, entity resolution quality, and citation coverage.
- Implement in a week: import notes, bookmarks, and a focused email folder, then add extraction and GraphRAG.
- Grow carefully with tasks, calendars, and small-team sharing while keeping your graph portable and easy to back up.
