Retrieval-Augmented Generation: Question Answering based on Custom Dataset with Open-sourced LangChain Library

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Many use cases such as building a chatbot require text (text2text) generation models like BloomZ 7B1, Flan T5 XXL, and Flan T5 UL2 to respond to user questions with insightful answers. The BloomZ 7B1, Flan T5 XXL, and Flan T5 UL2 models have picked up a lot of general knowledge in training, but we often need to ingest and use a large library of more specific information.

In this notebook we will demonstrate how to use BloomZ 7B1, Flan T5 XXL, and Flan T5 UL2 to answer questions using a library of documents as a reference, by using document embeddings and retrieval. The embeddings are generated from GPT-J-6B embedding model.

This notebook serves a template such that you can easily replace the example dataset by your own to build a custom question and asnwering application.

Step 1. Deploy large language model (LLM) and embedding model in SageMaker JumpStart

To better illustrate the idea, let's first deploy all the models that are required to perform the demo. You can choose either deploying all three Flan T5 XL, BloomZ 7B1, and Flan UL2 models as the large language model (LLM) to compare their model performances, or select subset of the models based on your preference. To do that, you need modify the _MODEL_CONFIG_ python dictionary defined as below.

Deploy SageMaker endpoint(s) for large language models and GPT-J 6B embedding model. Please uncomment the entries as below if you want to deploy multiple LLM models to compare their performance.

Step 2. Ask a question to LLM without providing the context

To better illustrate why we need retrieval-augmented generation (RAG) based approach to solve the question and anwering problem. Let's directly ask the model a question and see how they respond.

You can see the generated answer is wrong or doesn't make much sense.

Step 3. Improve the answer to the same question using prompt engineering with insightful context

To better answer the question well, we provide extra contextual information, combine it with a prompt, and send it to model together with the question. Below is an example.

The output from step 3 tells us the chance to get the correct response significantly correlates with the insightful context you send into the LLM.

Now, the question becomes where can I find the insightful context based on the user query? The answer is to use a pre-stored knowledge data base with retrieval augmented generation, as shown in step 4 below.

Step 4. Use RAG based approach with LangChain and SageMaker endpoints to build a simplified question and answering application.

We plan to use document embeddings to fetch the most relevant documents in our document knowledge library and combine them with the prompt that we provide to LLM.

To achieve that, we will do following.

1. Generate embedings for each of document in the knowledge library with SageMaker GPT-J-6B embedding model.
2. Identify top K most relevant documents based on user query.
   • 2.1 For a query of your interest, generate the embedding of the query using the same embedding model.
   • 2.2 Search the indexes of top K most relevant documents in the embedding space using in-memory Faiss search.
   • 2.3 Use the indexes to retrieve the corresponded documents.
3. Combine the retrieved documents with prompt and question and send them into SageMaker LLM.

Note: The retrieved document/text should be large enough to contain enough information to answer a question; but small enough to fit into the LLM prompt -- maximum sequence length of 1024 tokens.

To build a simiplied QA application with LangChain, we need:

1. Wrap up our SageMaker endpoints for embedding model and LLM into langchain.embeddings.SagemakerEndpointEmbeddings and langchain.llms.sagemaker_endpoint.SagemakerEndpoint. That requires a small overwritten of SagemakerEndpointEmbeddings class to make it compatible with SageMaker embedding mdoel.
2. Prepare the dataset to build the knowledge data base.

Wrap up our SageMaker endpoints for embedding model into langchain.embeddings.SagemakerEndpointEmbeddings. That requires a small overwritten of SagemakerEndpointEmbeddings class to make it compatible with SageMaker embedding mdoel.

Next, we wrap up our SageMaker endpoints for LLM into langchain.llms.sagemaker_endpoint.SagemakerEndpoint.

Now, let's download the example data and prepare it for demonstration. We will use Amazon SageMaker FAQs as knowledge library. The data are formatted in a CSV file with two columns Question and Answer. We use the Answer column as the documents of knowledge library, from which relevant documents are retrieved based on a query.

For your purpose, you can replace the example dataset of your own to build a custom question and answering application.

For the case when you have data saved in multiple subsets. The following code will read all files that end with .csv and concatenate them together. Please ensure each csv file has the same format.

Drop the Question column as it is not used in this demonstration.

Use langchain to read the csv data. There are multiple built-in functions in LangChain to read different format of files such as txt, html, and pdf. For details, see LangChain document loaders.

Now, we can build an QA application. LangChain makes it extremly simple with following few lines of code.

Based on the question below, we can achieven the points in Step 4 with just a few lines of code as shown below.

Step 5. Customize the QA application above with different prompt.

Now, we see how simple it is to use LangChain to achieve question and answering application with just few lines of code. Let's break down the above VectorstoreIndexCreator and see what's happening under the hood. Furthermore, we will see how to incorporate a customize prompt rather than using a default prompt with VectorstoreIndexCreator.

Firstly, we generate embedings for each of document in the knowledge library with SageMaker GPT-J-6B embedding model.

Based on the question above, we then identify top K most relevant documents based on user query, where K = 3 in this setup.

Print out the top 3 most relevant docuemnts as below.

Finally, we combine the retrieved documents with prompt and question and send them into SageMaker LLM.

We define a customized prompt as below.

Send the top 3 most relevant docuemnts and question into LLM to get a answer.

Print the final answer from LLM as below, which is accurate.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.