Build a Generative AI application using Elasticsearch and OpenAI

This notebook demonstrates how to:

• Index the OpenAI Wikipedia vector dataset into Elasticsearch
• Build a simple Gen AI application with Streamlit that retrieves context using Elasticsearch and formulate answers using OpenAI

Install packages and import modules

Connect to Elasticsearch

ℹ️ We're using an Elastic Cloud deployment of Elasticsearch for this notebook. If you don't already have an Elastic deployment, you can sign up for a free Elastic Cloud trial.

To connect to Elasticsearch, you need to create a client instance with the Cloud ID and password for your deployment.

Find the Cloud ID for your deployment by going to https://cloud.elastic.co/deployments and selecting your deployment.

Test the connection with Elasticsearch.

Configure OpenAI connection

Our example will use OpenAI to formulate an answer, so please provide a valid OpenAI Api Key here.

You can follow this guide to retrieve your API Key.

Then test the connection with OpenAI and check the model used in this notebook is available.

Download the dataset

In this step we download the OpenAI Wikipedia embeddings dataset, and extract the zip file.

Read CSV file into a Pandas DataFrame

Next we use the Pandas library to read the unzipped CSV file into a DataFrame. This step makes it easier to index the data into Elasticsearch in bulk.

Create index with mapping

Now we need to create an Elasticsearch index with the necessary mappings. This will enable us to index the data into Elasticsearch.

We use the dense_vector field type for the title_vector and content_vector fields. This is a special field type that allows us to store dense vectors in Elasticsearch.

Later, we'll need to target the dense_vector field for kNN search.

Index data into Elasticsearch

The following function generates the required bulk actions that can be passed to Elasticsearch's Bulk API, so we can index multiple documents efficiently in a single request.

For each row in the DataFrame, the function yields a dictionary representing a single document to be indexed.

As the dataframe is large, we will index data in batches of 100. We index the data into Elasticsearch using the Python client's helpers for the bulk API.

Build application with Streamlit

In the following section, you will build a simple interface using streamlit.

This application will display a simple search bar where an user can ask a question. Elasticsearch is used to retrieve the relevant documents (context) matching the question then OpenAI formulate an answer using the context.

Install the dependency to access the application once running.

Create application

Run the application

Run the application and check your IP for the tunneling

Create the tunnel to access it from anywhere

Run the tunnel and use the link below to connect to the tunnel.

Use the IP from the previous step to connect to the application

Success you build your first Gen AI Application.

You can try it by asking question such as "Who is Beethoven?" or "What is football?" and see the answers.

Next steps

Now you know how to quickly put together an interface that allows you to ask questions and get answer from a specific dataset, in this notebook example, wikipedia.

You can adapt this example to use your own dataset, and use the streamlit application as a blueprint for integrating with your own application.