[Gemma 2]RAG LlamaIndex
Copyright 2024 Google LLC.
RAG with Gemma and LlamaIndex
This notebook demonstrates how to integrate Gemma model with LlamaIndex library to build a basic RAG application.
Setup
Select the Colab runtime
To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:
- In the upper-right of the Colab window, select ▾ (Additional connection options).
- Select Change runtime type.
- Under Hardware accelerator, select T4 GPU.
Gemma setup
To complete this tutorial, you'll first need to complete the setup instructions at Gemma setup. The Gemma setup instructions show you how to do the following:
- Get access to Gemma on kaggle.com.
- Select a Colab runtime with sufficient resources to run the Gemma 2B model.
- Generate and configure a Kaggle username and an API key as Colab secrets.
After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment.
Configure your credentials
Add your your Kaggle credentials to the Colab Secrets manager to securely store it.
- Open your Google Colab notebook and click on the 🔑 Secrets tab in the left panel.

- Create new secrets:
KAGGLE_USERNAMEandKAGGLE_KEY - Copy/paste your username into
KAGGLE_USERNAME - Copy/paste your key into
KAGGLE_KEY - Toggle the buttons on the left to allow notebook access to the secrets.
Install dependencies
Run the cell below to install all the required dependencies.
Gemma
About Gemma
Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. They are text-to-text, decoder-only large language models, available in English, with open weights, pre-trained variants, and instruction-tuned variants. Gemma models are well-suited for a variety of text generation tasks, including question answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop or your own cloud infrastructure, democratizing access to state of the art AI models and helping foster innovation for everyone.
LlamaIndex
LlamaIndex is a toolkit for developers to build applications that use large language models (LLMs) with specific data. This data can be private or related to a particular field. With LlamaIndex, developers can create various LLM applications, including question-answering chatbots, document analysis tools, and even autonomous agents. The toolkit offers functions to process data and design workflows that combine data retrieval with instructions for the LLM.
Large language models (LLMs) are powerful but lack your specific data. Retrieval-Augmented Generation (RAG) bridges this gap by incorporating your data for improved performance. RAG works by indexing your data for efficient retrieval based on user queries. The most relevant information, along with the query itself, is then fed to the LLM to generate a response. Understanding RAG is essential for building LLM applications like chatbots and agents.
Setup
To ensure compatibility between Gemma and the LlamaIndex library, you need to creata a simple interface class. The provided code implements basic generation methods, allowing the library to interact with our model effectively.
.gitattributes: 0%| | 0.00/1.48k [00:00<?, ?B/s]
1_Pooling/config.json: 0%| | 0.00/270 [00:00<?, ?B/s]
2_Dense/config.json: 0%| | 0.00/115 [00:00<?, ?B/s]
pytorch_model.bin: 0%| | 0.00/2.36M [00:00<?, ?B/s]
README.md: 0%| | 0.00/66.2k [00:00<?, ?B/s]
config.json: 0%| | 0.00/1.55k [00:00<?, ?B/s]
config_sentence_transformers.json: 0%| | 0.00/122 [00:00<?, ?B/s]
pytorch_model.bin: 0%| | 0.00/439M [00:00<?, ?B/s]
sentence_bert_config.json: 0%| | 0.00/53.0 [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/2.20k [00:00<?, ?B/s]
spiece.model: 0%| | 0.00/792k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/2.42M [00:00<?, ?B/s]
tokenizer_config.json: 0%| | 0.00/2.43k [00:00<?, ?B/s]
modules.json: 0%| | 0.00/461 [00:00<?, ?B/s]
load INSTRUCTOR_Transformer max_seq_length 512
Retrieval-Augmented Generation (RAG)
RAGs improve generative AI outputs through a multi-step process. First, it searches for relevant external data (webpages, databases) using powerful algorithms. This retrieved information is then cleaned and prepped for the LLM. Finally, the prepped data is fed alongside the original query into the LLM. This extra context allows the LLM to understand the topic better, resulting in more precise, informative, and engaging responses.
This notebook demonstrates how to build an RAG application by utilising Paul Graham's essay. The essay is used as a placeholder to illustrate the RAG concepts without introducing complexities of real-world data. It simplifies the process by focusing on a single, well-defined source (the essay) to showcase how data retrieval enhances LLM performance in a RAG system.
Chunking the data
'What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn\'t write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n\nThe first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district\'s 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain\'s lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.'
Building Vector Store
Now you can now build a search engine that finds the best parts of text that answer a user's question.
Found: 3 relevant chunks 1) She liked to paint on big, square canvases, 4 to 5 feet on a sid... 2) The next year, from the summer of 1998 to the summer of 1999, mu... 3) It was a lot of fun working with Robert and Trevor. They're the ...
Those chunks will be inject to the LLM's prompt in order to answer user query.
So when an user ask a question, the following prompt will be send to the LLM:
Text:
<chunk #1>
<chunk #2>
<chunk #3>
According to the text answer the query: <question>
By providing the large language model (LLM) with additional context, it can generate more accurate and informative responses to user queries.
Test it yourself!
ASP stands for **Application Service Provider**. <end_of_turn>
The text mentions that Paul Graham applied to RISD (Rhode Island School of Design) and the Accademia. <end_of_turn>
Run in Google Colab