Scrapegraph Burr Lancedb

🕷️ Chat with your webpage with scrapegraph, burr and lancedb

🔧 Install dependencies

🔑 Import ScrapeGraph and OpenAI API keys

You can find the Scrapegraph API key here

🚀 Define the extraction and query flow using burr and run it

burr is an open-source orchestrator framework that makes it easy to develop applications that make decisions (chatbots, agents, simulations, etc...). It also features a cool self-hosted UI to trace what's happening in the application.

Check the Github Repo

Our goal is to define a flow (DAG) that:

1. Fetches markdown from webpages (scrapegraph)
2. Chunks the content and stores it in a vector store (lancedb)
3. Allows to query the db and generate an answer using a LLM

We can see all of this as Nodes connected together in a Graph, where the Nodes are the actions we want to perform.

In burr we define actions by simply adding the action decorator to each function and specifying what that function needs to read and write from the graph's state.

All imports

🔍 Define fetch_webpage action to fetch and convert a webpage into markdown

Here we use markdownify to fetch a webpage and convert it into markdown format, which is suitable to LLM ingestion.

You can find more info in the official scrapegraph documentation

📁 Define embed_and_store action to chunk the markdown and store it in a local vector store

Define the data structure to hold the chunks in the lancedb vector store

Utils to create chunks based on the number of tokens

Let's define the action. It creates a webpages local vector store if not present and add the chunks to the chunks table

💬 Define ask_question action to retrieve the most relevant chunks from the vector store and query them with a llm

Fetches the first 3 relevant chunks based on the user query and generate and answer

🤖 Define the burr application graph and run it

🖼️ Visualize the traces with Burr UI

🔗 Resources

• 🚀 Get your API Key: ScrapeGraphAI Dashboard
• 🐙 GitHub: ScrapeGraphAI GitHub
• 💼 LinkedIn: ScrapeGraphAI LinkedIn
• 🐦 Twitter: ScrapeGraphAI Twitter
• 💬 Discord: Join our Discord Community
• ⏩ Burr: Github
• 🛢️ LanceDB: Github

Made with ❤️ by the ScrapeGraphAI Team