[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"readme:knowledge_graph":3},"\u003Ch1>Convert any Corpus of Text into a \u003Cem>Graph of Knowledge\u003C\u002Fem>\u003C\u002Fh1>\n\u003Cp>\u003Cimg src=\"https:\u002F\u002Fraw.githubusercontent.com\u002Frahulnyk\u002Fknowledge_graph\u002FHEAD\u002Fassets\u002FKG_banner.png\" alt=\"Knowledge Graph Banner\" \u002F>\n\u003Cem>A knowledge graph generated using this code\u003C\u002Fem> \nghpages link of this graph: \u003Ca href=\"https:\u002F\u002Frahulnyk.github.io\u002Fknowledge_graph\u002F\" rel=\"nofollow ugc noopener\">https:\u002F\u002Frahulnyk.github.io\u002Fknowledge_graph\u002F\u003C\u002Fa>\u003C\u002Fp>\n\u003Ch2>What is a knowledge graph?\u003C\u002Fh2>\n\u003Cp>A knowledge graph, also known as a semantic network, represents a network of real-world entities—i.e. objects, events, situations, or concepts—and illustrates the relationship between them. This information is usually stored in a graph database and visualized as a graph structure, prompting the term knowledge “graph.”\u003C\u002Fp>\n\u003Cp>Source: \u003Ca href=\"https:\u002F\u002Fwww.ibm.com\u002Ftopics\u002Fknowledge-graph\" rel=\"nofollow ugc noopener\">https:\u002F\u002Fwww.ibm.com\u002Ftopics\u002Fknowledge-graph\u003C\u002Fa>\u003C\u002Fp>\n\u003Ch2>How to create a simple knowledge graph from a body of work?\u003C\u002Fh2>\n\u003Col>\n\u003Cli>Clean the text corpus (The body of work).\u003C\u002Fli>\n\u003Cli>Extract concepts and entities from the body of work.\u003C\u002Fli>\n\u003Cli>Extract relations between the entities. \u003C\u002Fli>\n\u003Cli>Convert a graph schema. \u003C\u002Fli>\n\u003Cli>Populate nodes (concepts) and edges (relations).\u003C\u002Fli>\n\u003Cli>Visualise and Query.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>Step 6 is purely optional, but it has certain artistic gratification associated with it. Network graphs are beautiful objects (just look at the banner image above, isn't it beautiful?). Fortunately, there are a good number of Python libraries available for generating graph visualisations. \u003C\u002Fp>\n\u003Ch2>Why Graph?\u003C\u002Fh2>\n\u003Cp>Once the Knowledge Graph (KG) is build, we can use it for many purposes. We can run graph algorithms and calculate centralities of any node, to understand how important a concept (node) is to this body of work. We can calculate communities to bunch the concepts together to better analyse the text. We can understand the connectedness between seemingly disconnected concepts. \u003C\u002Fp>\n\u003Cp>The best of all, we can achieve \u003Cstrong>Graph Retrieval Augmented Generation (GRAG)\u003C\u002Fstrong> and chat with our text in a much more profound way using Graph as a retriever. This is a new and improved version of \u003Cstrong>Retrieval Augmented Generation (RAG)\u003C\u002Fstrong> where we use a vectory db as a retriever to chat with our documents. \u003C\u002Fp>\n\u003Chr \u002F>\n\u003Ch2>This project\u003C\u002Fh2>\n\u003Cp>Here I have created a simple knowledge graph from a PDF document. The process I follow here is very similar to what is outlined in the above sections, with some simplifications.\u003C\u002Fp>\n\u003Cp>First I split the entire text into chunks. Then I extract concepts mentioned within each chunk using an LLM. Note that I am not extracting entities using an NER model here. There is a difference between concepts and entities. For example 'Bangalore' is an entity, and 'Pleasant weather in Bangalore' is a concept. In my experience, concepts make more meaningful KG than entities.\u003C\u002Fp>\n\u003Cp>I assume that the concepts that are mentioned in the vicinity of each other are related. So every edge in the KG is a text chunk in which the two connected concepts are mentioned.\u003C\u002Fp>\n\u003Cp>Once the nodes (concepts) and the edges (text chunks) are calculated, It is easy to create a graph out of them using the libraries mentioned here.\nAll the components I used here are set up locally, so this project can be run very easily on a personal machine. I have adopted a no-GPT approach here to keep things economical. I am using the fantastic Mistral 7B openorca instruct, which crushes this use case wonderfully. The model can be set up locally using Ollama so generating the KG is basically free (No calls to GPT).\u003C\u002Fp>\n\u003Cp>To generate a graph this the notebook you have to tweak. \u003C\u002Fp>\n\u003Cp>\u003Cstrong>\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Frahulnyk\u002Fknowledge_graph\u002Fblob\u002Fmain\u002Fextract_graph.ipynb\" rel=\"nofollow ugc noopener\">extract_graph.ipynb\u003C\u002Fa>\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>The notebook implements the method outlined in the following flowchart. \u003C\u002Fp>\n\u003Cimg src=\"https:\u002F\u002Fraw.githubusercontent.com\u002Frahulnyk\u002Fknowledge_graph\u002FHEAD\u002Fassets\u002FMethod.png\" \u002F>\u003Col>\n\u003Cli>Split the corpus of text into chunks. Assign a chunk_id to each of these chunks.\u003C\u002Fli>\n\u003Cli>For every text chunk extract concepts and their semantic relationships using an LLM. Let’s assign this relation a weightage of W1. There can be multiple relationships between the same pair of concepts. Every such relation is an edge between a pair of concepts.\u003C\u002Fli>\n\u003Cli>Consider that the concepts that occur in the same text chunk are also related by their contextual proximity. Let’s assign this relation a weightage of W2.\u003C\u002Fli>\n\u003C\u002Fol>\n",1788652555405]