Using Pinecone For Embeddings Search
Using Pinecone for Embeddings Search
This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.
What is a Vector Database
A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.
Why use a Vector Database
Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.
Demo Flow
The demo flow is:
- Setup: Import packages and set any required variables
- Load data: Load a dataset and embed it using OpenAI embeddings
- Pinecone
- Setup: Here we'll set up the Python client for Pinecone. For more details go here
- Index Data: We'll create an index with namespaces for titles and content
- Search Data: We'll test out both namespaces with search queries to confirm it works
Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.
Setup
Import the required libraries and set the embedding model that we'd like to use.
Requirement already satisfied: pinecone-client in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (2.2.2) Requirement already satisfied: requests>=2.19.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.31.0) Requirement already satisfied: pyyaml>=5.4 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (6.0) Requirement already satisfied: loguru>=0.5.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (0.7.0) Requirement already satisfied: typing-extensions>=3.7.4 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (4.5.0) Requirement already satisfied: dnspython>=2.0.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.3.0) Requirement already satisfied: python-dateutil>=2.5.3 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.8.2) Requirement already satisfied: urllib3>=1.21.1 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (1.26.16) Requirement already satisfied: tqdm>=4.64.1 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (4.65.0) Requirement already satisfied: numpy>=1.22.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (1.25.0) Requirement already satisfied: six>=1.5 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from python-dateutil>=2.5.3->pinecone-client) (1.16.0) Requirement already satisfied: charset-normalizer<4,>=2 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (3.1.0) Requirement already satisfied: idna<4,>=2.5 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (3.4) Requirement already satisfied: certifi>=2017.4.17 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (2023.5.7) Requirement already satisfied: wget in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (3.2)
/Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console) from tqdm.autonotebook import tqdm
Load data
In this section we'll load embedded data that we've prepared in this article.
<class 'pandas.core.frame.DataFrame'> RangeIndex: 25000 entries, 0 to 24999 Data columns (total 7 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 id 25000 non-null int64 1 url 25000 non-null object 2 title 25000 non-null object 3 text 25000 non-null object 4 title_vector 25000 non-null object 5 content_vector 25000 non-null object 6 vector_id 25000 non-null object dtypes: int64(1), object(6) memory usage: 1.3+ MB
Pinecone
The next option we'll look at is Pinecone, a managed vector database which offers a cloud-native option.
Before you proceed with this step you'll need to navigate to Pinecone, sign up and then save your API key as an environment variable titled PINECONE_API_KEY.
For section we will:
- Create an index with multiple namespaces for article titles and content
- Store our data in the index with separate searchable "namespaces" for article titles and content
- Fire some similarity search queries to verify our setup is working
Create Index
First we will need to create an index, which we'll call wikipedia-articles. Once we have an index, we can create multiple namespaces, which can make a single index searchable for various use cases. For more details, consult Pinecone documentation.
If you want to batch insert to your index in parallel to increase insertion speed then there is a great guide in the Pinecone documentation on batch inserts in parallel.
['podcasts', 'wikipedia-articles']
Uploading vectors to content namespace..
Uploading vectors to title namespace..
{'dimension': 1536,
, 'index_fullness': 0.1,
, 'namespaces': {'content': {'vector_count': 25000},
, 'title': {'vector_count': 25000}},
, 'total_vector_count': 50000} Search data
Now we'll enter some dummy searches and check we get decent results back
Most similar results to modern art in Europe in "title" namespace: Museum of Modern Art (score = 0.875177085) Western Europe (score = 0.867441177) Renaissance art (score = 0.864156306) Pop art (score = 0.860346854) Northern Europe (score = 0.854658186)
Most similar results to Famous battles in Scottish history in "content" namespace: Battle of Bannockburn (score = 0.869336188) Wars of Scottish Independence (score = 0.861470938) 1651 (score = 0.852588475) First War of Scottish Independence (score = 0.84962213) Robert I of Scotland (score = 0.846214116)