RAG pipelines with Haystack + Zephyr 7B Beta 🪁

Notebook by Stefano Fiorucci and Tuana Celik

We are going to build a nice Retrieval Augmented Generation pipeline for Rock music, using the 🏗️ Haystack LLM orchestration framework and a good LLM: 💬 Zephyr 7B Beta (fine-tuned version of Mistral 7B V.01 that focuses on helpfulness and outperforms many larger models on the MT-Bench and AlpacaEval benchmarks)

Install dependencies

• wikipedia is needed to download data from Wikipedia
• haystack-ai is the Haystack package
• sentence_transformers is needed for embeddings
• transformers is needed to use open-source LLMs
• accelerate and bitsandbytes are required to use quantized versions of these models (with smaller memory footprint)

Load data from Wikipedia

We are going to download the Wikipedia pages related to some Rock bands, using the python library wikipedia.

These pages are converted into Haystack Documents

The Indexing Pipeline

We will save our final Documents in an InMemoryDocumentStore, a simple database which lives in memory.

Our indexing Pipeline transform the original Documents and save them in the Document Store.

It consists of several components:

• DocumentCleaner: performs a basic cleaning of the Documents
• DocumentSplitter: chunks each Document into smaller pieces (more appropriate for semantic search and RAG)
• SentenceTransformersDocumentEmbedder:
  • represent each Document as a vector (capturing its meaning).
  • we choose a good but not too big model from MTEB leaderboard.
  • Also the metadata title is embedded, because it contains relevant information (metadata_fields_to_embed parameter).
  • We use the GPU for this expensive operation (device parameter).
• DocumentWriter just saves the Documents in the Document Store

Let's draw the indexing pipeline

We finally run the indexing pipeline

Let's inspect the total number of chunked Documents and examine a Document

The RAG Pipeline

HuggingFaceLocalGenerator with zephyr-7b-beta

• To load and manage Open Source LLMs in Haystack, we can use the HuggingFaceLocalGenerator.

• The LLM we choose is Zephyr 7B Beta, a fine-tuned version of Mistral 7B V.01 that focuses on helpfulness and outperforms many larger models on the MT-Bench and AlpacaEval benchmarks; the model was fine-tuned by the Hugging Face team.

• Since we are using a free Colab instance (with limited resources), we load the model using 4-bit quantization (passing the appropriate huggingface_pipeline_kwargs to our Generator). For an introduction to Quantization in Hugging Face Transformers, you can read this simple blog post.

Let's try the model...

Ok, nice!

PromptBuilder

It's a component that renders a prompt from a template string using Jinja2 engine.

Let's setup our prompt builder, with a format like the following (appropriate for Zephyr):

"<|system|>\nSYSTEM MESSAGE</s>\n<|user|>\nUSER MESSAGE</s>\n<|assistant|>\n"

Let's create the RAG pipeline

Our RAG Pipeline finds Documents relevant to the user query and pass them to the LLM to generate a grounded answer.

It consists of several components:

• SentenceTransformersTextEmbedder: represent the query as a vector (capturing its meaning).
• InMemoryEmbeddingRetriever: finds the (top 5) Documents that are most similar to the query vector
• PromptBuilder
• HuggingFaceLocalGenerator

Visualize our pipeline!

We create an utility function that runs the RAG pipeline and nicely prints the answer.

Let's try our RAG pipeline...

More questions to try...