StackMap
Subscribe
Explore / knowledge_graph
rahulnyk

knowledge_graph

Notebook recipe that turns any text corpus into a concept graph with a local Mistral 7B via Ollama — chunk, extract concepts and relations, add proximity edges — for Graph RAG and KG QA.

4,031 627 Jupyter Notebook MITupdated 21 days ago
knowledge-graphsraglocal
View on GitHubDispute this mapping →
Curator's take

Read this to *learn* how a text-to-knowledge-graph pipeline works: it is a single notebook that chunks a PDF, asks a local model for concepts per chunk (concepts, not NER entities — the author's good point), treats co-occurrence as a weighted edge, and visualises with NetworkX/pyvis. Free to run, GPU-free, honest about being simple. NOT a library: no schema, no incremental updates, no store, no retrieval layer — 4k stars are for the tutorial, not for shipping. When you've understood the recipe, move to a tool that validates against a schema and persists.

Mapped by ShipWithAI editors · links verified

Continue your stack

What teams reach for next — and why each earns a place beside knowledge_graph. Ranked by curator confidence.

alternativealternativebuilt withHyper-Extractdocling-graphOllamaknowledge_graph
pairs wellalternativebuilt withpick a node for the why · open it from the panel
Weekly digest
README.md3 min read

Convert any Corpus of Text into a Graph of Knowledge

Knowledge Graph Banner A knowledge graph generated using this code ghpages link of this graph: https://rahulnyk.github.io/knowledge_graph/

What is a knowledge graph?

A knowledge graph, also known as a semantic network, represents a network of real-world entities—i.e. objects, events, situations, or concepts—and illustrates the relationship between them. This information is usually stored in a graph database and visualized as a graph structure, prompting the term knowledge “graph.”

Source: https://www.ibm.com/topics/knowledge-graph

How to create a simple knowledge graph from a body of work?

  1. Clean the text corpus (The body of work).
  2. Extract concepts and entities from the body of work.
  3. Extract relations between the entities.
  4. Convert a graph schema.
  5. Populate nodes (concepts) and edges (relations).
  6. Visualise and Query.

Step 6 is purely optional, but it has certain artistic gratification associated with it. Network graphs are beautiful objects (just look at the banner image above, isn't it beautiful?). Fortunately, there are a good number of Python libraries available for generating graph visualisations.

Why Graph?

Once the Knowledge Graph (KG) is build, we can use it for many purposes. We can run graph algorithms and calculate centralities of any node, to understand how important a concept (node) is to this body of work. We can calculate communities to bunch the concepts together to better analyse the text. We can understand the connectedness between seemingly disconnected concepts.

The best of all, we can achieve Graph Retrieval Augmented Generation (GRAG) and chat with our text in a much more profound way using Graph as a retriever. This is a new and improved version of Retrieval Augmented Generation (RAG) where we use a vectory db as a retriever to chat with our documents.

This project

Here I have created a simple knowledge graph from a PDF document. The process I follow here is very similar to what is outlined in the above sections, with some simplifications.

First I split the entire text into chunks. Then I extract concepts mentioned within each chunk using an LLM. Note that I am not extracting entities using an NER model here. There is a difference between concepts and entities. For example 'Bangalore' is an entity, and 'Pleasant weather in Bangalore' is a concept. In my experience, concepts make more meaningful KG than entities.

I assume that the concepts that are mentioned in the vicinity of each other are related. So every edge in the KG is a text chunk in which the two connected concepts are mentioned.

Once the nodes (concepts) and the edges (text chunks) are calculated, It is easy to create a graph out of them using the libraries mentioned here. All the components I used here are set up locally, so this project can be run very easily on a personal machine. I have adopted a no-GPT approach here to keep things economical. I am using the fantastic Mistral 7B openorca instruct, which crushes this use case wonderfully. The model can be set up locally using Ollama so generating the KG is basically free (No calls to GPT).

To generate a graph this the notebook you have to tweak.

extract_graph.ipynb

The notebook implements the method outlined in the following flowchart.

  1. Split the corpus of text into chunks. Assign a chunk_id to each of these chunks.
  2. For every text chunk extract concepts and their semantic relationships using an LLM. Let’s assign this relation a weightage of W1. There can be multiple relationships between the same pair of concepts. Every such relation is an edge between a pair of concepts.
  3. Consider that the concepts that occur in the same text chunk are also related by their contextual proximity. Let’s assign this relation a weightage of W2.