Movie Recommender System

In this example you will create a movie recommender system.

The system will extract feature vectors from metadata about films using SentenceTransformers, import those vectors into Milvus, with the metadata. When a user submits information about movies they're interested in, you'll search Milvus for similar films and provide searchers movie info from Redis using the results.

Requirements

• Python 3.x.
• Docker
• A system with at least 32GB of RAM, or a Zilliz cloud account

Data

In this project, you'll use The Movies Dataset from Kaggle. This dataset contains metadata on more than 45k movies.

The dataset has several files, but you'll only need movies_metadata.csv, the main Movies Metadata file. You can use this notebook as a starting point and modify it to take advantage of the rest of this dataset.

Requirements

First, install the Python packages needed for this project.

Download dataset

Now you'll download the dataset. You'll use the Kaggle API to retrieve the data.

Set your login information below, or download kaggle.json to a location where the API will find it.

Download the data and unzip it to the dataset directory.

Milvus Server

You're going to create vector embeddings from the movie's descriptions. So, you need a way to store, index, and search on those embeddngs. That's where Milvus comes in.

This is a relatively large dataset, at least for a server running on a personal computer. So, you may want to use a Zilliz Cloud instance to store these vectors.

But, if you want to stay with a local instance, you can download a docker compose configuration and run that.

Here's how to get the compose file, and start the server.

But, if you want to use cloud, sign up for an account here.

Youhave more than 45k movies, with 24 columns of metadata.

List the columns.

There's no need to store all these columns in Milvus. Trim them down to the metatdata we want to store with the vectors and remove any items that are missing critical fields.

Now, it's time to connect to Milvus so you can start uploading data.

Here's the code for connecting to a cloud instance. Replace the URI and TOKEN with the correct values for your instance.

You can find them in your Zilliz dashboard:

So, with the meta data stored in Redis, it's time to calculate the embeddings and add them to Milvus.

First, you need a collection to store them in. Create a simple one that stores the title and embeddings for in the Movies field, while also allowing dynamic fields. You'll use the dynamic fields for metadata.

Then, you'll index the embedding field to make searches more efficent.

Now, you need a function to create the embeddings.

The primary artifact for movie information is the overview, but including the genre and release date in complete sentences may help with search accuracy.

Create a transformer and call it from a simple function:

• extract the id field
• creates an embed from the overview, genre and release date
• inserts the vector into Milvus.

You'll reuse the build_genres function below for searching.

Now, you can create the embeddings. This dataset is too large to send to Milvus in a single insert statement, but sending them one at a time would create unnecessary network traffic and add too much time. So, this code uses batches. You can play with the batch size to suit your individual needs and preferences.

Now you can search for movies that match viewer criteria. To do this, you need a few more functions.

First, you need a transformer to convert the user's search string to an embedding. For this, embed_search takes their criteria and passed it to the same transformer you used to populate Milvus.

By setting the title and overview fields in the return set, you can simply print the result set for the user.

Finally, search_for_movies performs the actual vector search, using the other two functions for support.

So, put this search to work!

This search is looking for 1990s comedies with Vampires. The first hit is exactly that, but as the vector distance increases you can see that the films move further away from what you're looking for.

You can play around with different search criteria.