Implementing RAG pipelines with MongoDB and Llama3 and Open models From Hugging Face

This notebook is designed to demonstrate how to integrate and utilize Hugging Face's open-source models, specifically Llama3, with MongoDB to implement Retrieval-Augmented Generation (RAG) pipelines for enhanced question answering capabilities.

The process involves preparing a dataset of arXiv papers, transforming their data for effective retrieval, setting up a MongoDB database with vector search capabilities, and using llama3 model for generating answers based on the retrieved documents.

Key Highlights:

• Usage of Hugging Face open-source models and MongoDB for creating RAG pipelines.
• Steps include dataset preparation, database setup, data ingestion, and query processing.
• Detailed guidance on setting up MongoDB collections and vector search indexes.
• Integration with the Llama3 model from Hugging Face for answering complex queries.

Follow the following instruction to set up a MongoDB database and enable vector search:

1. Register a free Atlas account or sign in to your existing Atlas account.

2. Follow the instructions (select Atlas UI as the procedure) to deploy your first cluster, which distributes your data across multiple servers for improved performance and redundancy.

3. For a free Cluser, be sure to select "Shared" option when creating your new cluster. See image below for details

4. Create the database: knowledge_base, and collection research_papers

Import Libraries

Import libaries into development environment

Dataset Loading and Preparation

Load the dataset from HuggingFace.

Only using the first 100 datapoint for demo purposes.

Generate Embeddings

Database and Collection Setup

Complete the steps below if not already carried out previously. Creating a database and collection within MongoDB is made simple with MongoDB Atlas.

1. Register a free Atlas account or sign in to your existing Atlas account.

2. Follow the instructions (select Atlas UI as the procedure) to deploy your first cluster.

3. Create the database: knowledge_base.

4. Within the database knowledge_base, create the following collections: research_papers

5. Create a vector search index named vector_index for the research_papers collection. This index enables the RAG application to retrieve records as additional context to supplement user queries via vector search. Below is the JSON definition of the data collection vector search index.

Below is a snipper of what the vector search index definition should looks like:

	   {
     "fields": [
       {
         "numDimensions": 1024,
         "path": "embedding",
         "similarity": "cosine",
         "type": "vector"
       }
     ]
   }

Data Ingestion

Vector Search

Handling User Queries

Loading and Using Llama3