Recommendation Using Embeddings
Recommendation using embeddings and nearest neighbor search
Recommendations are widespread across the web.
- 'Bought that item? Try these similar items.'
- 'Enjoy that book? Try these similar titles.'
- 'Not the help page you were looking for? Try these similar pages.'
This notebook demonstrates how to use embeddings to find similar items to recommend. In particular, we use AG's corpus of news articles as our dataset.
Our model will answer the question: given an article, what other articles are most similar to it?
2. Load data
Next, let's load the AG news data and see what it looks like.
Let's take a look at those same examples, but not truncated by ellipses.
Title: World Briefings Description: BRITAIN: BLAIR WARNS OF CLIMATE THREAT Prime Minister Tony Blair urged the international community to consider global warming a dire threat and agree on a plan of action to curb the quot;alarming quot; growth of greenhouse gases. Label: World Title: Nvidia Puts a Firewall on a Motherboard (PC World) Description: PC World - Upcoming chip set will include built-in security features for your PC. Label: Sci/Tech Title: Olympic joy in Greek, Chinese press Description: Newspapers in Greece reflect a mixture of exhilaration that the Athens Olympics proved successful, and relief that they passed off without any major setback. Label: Sports Title: U2 Can iPod with Pictures Description: SAN JOSE, Calif. -- Apple Computer (Quote, Chart) unveiled a batch of new iPods, iTunes software and promos designed to keep it atop the heap of digital music players. Label: Sci/Tech Title: The Dream Factory Description: Any product, any shape, any size -- manufactured on your desktop! The future is the fabricator. By Bruce Sterling from Wired magazine. Label: Sci/Tech
3. Build cache to save embeddings
Before getting embeddings for these articles, let's set up a cache to save the embeddings we generate. In general, it's a good idea to save your embeddings so you can re-use them later. If you don't save them, you'll pay again each time you compute them again.
The cache is a dictionary that maps tuples of (text, model) to an embedding, which is a list of floats. The cache is saved as a Python pickle file.
Let's check that it works by getting an embedding.
Example string: BRITAIN: BLAIR WARNS OF CLIMATE THREAT Prime Minister Tony Blair urged the international community to consider global warming a dire threat and agree on a plan of action to curb the quot;alarming quot; growth of greenhouse gases. Example embedding: [0.0545826330780983, -0.00428084097802639, 0.04785159230232239, 0.01587914116680622, -0.03640881925821304, 0.0143799539655447, -0.014267769642174244, -0.015175441280007362, -0.002344391541555524, 0.011075624264776707]...
4. Recommend similar articles based on embeddings
To find similar articles, let's follow a three-step plan:
- Get the similarity embeddings of all the article descriptions
- Calculate the distance between a source title and all other articles
- Print out the other articles closest to the source title
5. Example recommendations
Let's look for articles similar to first one, which was about Tony Blair.
Source string: BRITAIN: BLAIR WARNS OF CLIMATE THREAT Prime Minister Tony Blair urged the international community to consider global warming a dire threat and agree on a plan of action to curb the quot;alarming quot; growth of greenhouse gases.
--- Recommendation #1 (nearest neighbor 1 of 5) ---
String: The anguish of hostage Kenneth Bigley in Iraq hangs over Prime Minister Tony Blair today as he faces the twin test of a local election and a debate by his Labour Party about the divisive war.
Distance: 0.514
--- Recommendation #2 (nearest neighbor 2 of 5) ---
String: THE re-election of British Prime Minister Tony Blair would be seen as an endorsement of the military action in Iraq, Prime Minister John Howard said today.
Distance: 0.516
--- Recommendation #3 (nearest neighbor 3 of 5) ---
String: Israel is prepared to back a Middle East conference convened by Tony Blair early next year despite having expressed fears that the British plans were over-ambitious and designed
Distance: 0.546
--- Recommendation #4 (nearest neighbor 4 of 5) ---
String: Allowing dozens of casinos to be built in the UK would bring investment and thousands of jobs, Tony Blair says.
Distance: 0.568
--- Recommendation #5 (nearest neighbor 5 of 5) ---
String: AFP - A battle group of British troops rolled out of southern Iraq on a US-requested mission to deadlier areas near Baghdad, in a major political gamble for British Prime Minister Tony Blair.
Distance: 0.579
Pretty good! 4 of the 5 recommendations explicitly mention Tony Blair and the fifth is an article from London about climate change, topics that might be often associated with Tony Blair.
Let's see how our recommender does on the second example article about NVIDIA's new chipset with more security.
Source string: PC World - Upcoming chip set will include built-in security features for your PC.
--- Recommendation #1 (nearest neighbor 1 of 5) ---
String: PC World - Updated antivirus software for businesses adds intrusion prevention features.
Distance: 0.422
--- Recommendation #2 (nearest neighbor 2 of 5) ---
String: PC World - Symantec, McAfee hope raising virus-definition fees will move users to\ suites.
Distance: 0.518
--- Recommendation #3 (nearest neighbor 3 of 5) ---
String: originally offered on notebook PCs -- to its Opteron 32- and 64-bit x86 processors for server applications. The technology will help servers to run
Distance: 0.522
--- Recommendation #4 (nearest neighbor 4 of 5) ---
String: PC World - Send your video throughout your house--wirelessly--with new gateways and media adapters.
Distance: 0.532
--- Recommendation #5 (nearest neighbor 5 of 5) ---
String: Chips that help a computer's main microprocessors perform specific types of math problems are becoming a big business once again.\
Distance: 0.532
From the printed distances, you can see that the #1 recommendation is much closer than all the others (0.11 vs 0.14+). And the #1 recommendation looks very similar to the starting article - it's another article from PC World about increasing computer security. Pretty good!
Appendix: Using embeddings in more sophisticated recommenders
A more sophisticated way to build a recommender system is to train a machine learning model that takes in tens or hundreds of signals, such as item popularity or user click data. Even in this system, embeddings can be a very useful signal into the recommender, especially for items that are being 'cold started' with no user data yet (e.g., a brand new product added to the catalog without any clicks yet).
Appendix: Using embeddings to visualize similar articles
To get a sense of what our nearest neighbor recommender is doing, let's visualize the article embeddings. Although we can't plot the 2048 dimensions of each embedding vector, we can use techniques like t-SNE or PCA to compress the embeddings down into 2 or 3 dimensions, which we can chart.
Before visualizing the nearest neighbors, let's visualize all of the article descriptions using t-SNE. Note that t-SNE is not deterministic, meaning that results may vary from run to run.
As you can see in the chart above, even the highly compressed embeddings do a good job of clustering article descriptions by category. And it's worth emphasizing: this clustering is done with no knowledge of the labels themselves!
Also, if you look closely at the most egregious outliers, they are often due to mislabeling rather than poor embedding. For example, the majority of the blue World points in the green Sports cluster appear to be Sports stories.
Next, let's recolor the points by whether they are a source article, its nearest neighbors, or other.
Looking at the 2D chart above, we can see that the articles about Tony Blair are somewhat close together inside of the World news cluster. Interestingly, although the 5 nearest neighbors (red) were closest in high dimensional space, they are not the closest points in this compressed 2D space. Compressing the embeddings down to 2 dimensions discards much of their information, and the nearest neighbors in the 2D space don't seem to be as relevant as those in the full embedding space.
For the chipset security example, the 4 closest nearest neighbors in the full embedding space remain nearest neighbors in this compressed 2D visualization. The fifth is displayed as more distant, despite being closer in the full embedding space.
Should you want to, you can also make an interactive 3D plot of the embeddings with the function chart_from_components_3D. (Doing so will require recomputing the t-SNE components with n_components=3.)