Book Title Search Using Towhee, Milvus and OpenAI

In this notebook we go over how to search for the best matching book titles using Towhee as the data processing pipeline, Milvus as the Vector Database and OpenAI as the embedding system.

Packages

We first begin with importing the required packages. In this example, the only non-builtin packages are towhee and pymilvus, with each being the client pacakges for their respective services. These packages can be installed using pip install pymilvus towhee.

Parameters

Here we can find the main parameters that need to be modified for running with your own accounts. Beside each is a description of what it is.

Milvus

This segment deals with Milvus and setting up the database for this use case. Within Milvus we need to setup a collection and index the collection. For more information on how to install and run Milvus, look here.

Insert Data

Once we have the collection setup we need to start inserting our data. This is done by creating a pipeline using Towhee. Within this pipeline there are two steps, embedding the text that is inputted, and inserting that data into Milvus.

Search the Data

With the collection setup and all our embedded data inserted, we can begin searching the data. This is done by creating a pipeline similar to the inserting pipeline, but in this one instead of inserting the data, we search the data. The search phrase embedded and its vector is searched across all the stored vectors to find the closest matches. These matches are the most semantically similar titles.