Export
Visualizing Movie Dataset Embeddings
Introduction
In this notebook, we will be embedding a subset of the Movies Dataset and visualizing the generated vector space representations of the movies using a technique called Uniform Manifold Approximation and Projection (UMAP).
We will color code the movies using genres and see if similar movie genres cluster together in vector space and apart from movies of different movie genres.
Install relevant libraries
[11]
Requirement already satisfied: together in /usr/local/lib/python3.10/dist-packages (1.3.3) Requirement already satisfied: aiohttp<4.0.0,>=3.9.3 in /usr/local/lib/python3.10/dist-packages (from together) (3.10.10) Requirement already satisfied: click<9.0.0,>=8.1.7 in /usr/local/lib/python3.10/dist-packages (from together) (8.1.7) Requirement already satisfied: eval-type-backport<0.3.0,>=0.1.3 in /usr/local/lib/python3.10/dist-packages (from together) (0.2.0) Requirement already satisfied: filelock<4.0.0,>=3.13.1 in /usr/local/lib/python3.10/dist-packages (from together) (3.16.1) Requirement already satisfied: numpy>=1.23.5 in /usr/local/lib/python3.10/dist-packages (from together) (1.26.4) Requirement already satisfied: pillow<11.0.0,>=10.3.0 in /usr/local/lib/python3.10/dist-packages (from together) (10.4.0) Requirement already satisfied: pyarrow>=10.0.1 in /usr/local/lib/python3.10/dist-packages (from together) (16.1.0) Requirement already satisfied: pydantic<3.0.0,>=2.6.3 in /usr/local/lib/python3.10/dist-packages (from together) (2.9.2) Requirement already satisfied: requests<3.0.0,>=2.31.0 in /usr/local/lib/python3.10/dist-packages (from together) (2.32.3) Requirement already satisfied: rich<14.0.0,>=13.8.1 in /usr/local/lib/python3.10/dist-packages (from together) (13.9.2) Requirement already satisfied: tabulate<0.10.0,>=0.9.0 in /usr/local/lib/python3.10/dist-packages (from together) (0.9.0) Requirement already satisfied: tqdm<5.0.0,>=4.66.2 in /usr/local/lib/python3.10/dist-packages (from together) (4.66.5) Requirement already satisfied: typer<0.13,>=0.9 in /usr/local/lib/python3.10/dist-packages (from together) (0.12.5) Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (2.4.3) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (1.3.1) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (24.2.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (1.4.1) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (6.1.0) Requirement already satisfied: yarl<2.0,>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (1.15.2) Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.9.3->together) (4.0.3) Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.6.3->together) (0.7.0) Requirement already satisfied: pydantic-core==2.23.4 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.6.3->together) (2.23.4) Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.6.3->together) (4.12.2) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.31.0->together) (3.4.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.31.0->together) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.31.0->together) (2.2.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.31.0->together) (2024.8.30) Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich<14.0.0,>=13.8.1->together) (3.0.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich<14.0.0,>=13.8.1->together) (2.18.0) Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<0.13,>=0.9->together) (1.5.4) Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich<14.0.0,>=13.8.1->together) (0.1.2) Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from yarl<2.0,>=1.12.0->aiohttp<4.0.0,>=3.9.3->together) (0.2.0) Collecting umap-learn[plot] Downloading umap_learn-0.5.6-py3-none-any.whl.metadata (21 kB) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (1.26.4) Requirement already satisfied: scipy>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (1.13.1) Requirement already satisfied: scikit-learn>=0.22 in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (1.5.2) Requirement already satisfied: numba>=0.51.2 in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (0.60.0) Collecting pynndescent>=0.5 (from umap-learn[plot]) Downloading pynndescent-0.5.13-py3-none-any.whl.metadata (6.8 kB) Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (4.66.5) Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (2.2.2) Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (3.7.1) Collecting datashader (from umap-learn[plot]) Downloading datashader-0.16.3-py2.py3-none-any.whl.metadata (12 kB) Requirement already satisfied: bokeh in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (3.4.3) Requirement already satisfied: holoviews in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (1.19.1) Requirement already satisfied: colorcet in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (3.1.0) Requirement already satisfied: seaborn in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (0.13.2) Requirement already satisfied: scikit-image in /usr/local/lib/python3.10/dist-packages (from umap-learn[plot]) (0.24.0) Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba>=0.51.2->umap-learn[plot]) (0.43.0) Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.10/dist-packages (from pynndescent>=0.5->umap-learn[plot]) (1.4.2) Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.22->umap-learn[plot]) (3.5.0) Requirement already satisfied: Jinja2>=2.9 in /usr/local/lib/python3.10/dist-packages (from bokeh->umap-learn[plot]) (3.1.4) Requirement already satisfied: contourpy>=1.2 in /usr/local/lib/python3.10/dist-packages (from bokeh->umap-learn[plot]) (1.3.0) Requirement already satisfied: packaging>=16.8 in /usr/local/lib/python3.10/dist-packages (from bokeh->umap-learn[plot]) (24.1) Requirement already satisfied: pillow>=7.1.0 in /usr/local/lib/python3.10/dist-packages (from bokeh->umap-learn[plot]) (10.4.0) Requirement already satisfied: PyYAML>=3.10 in /usr/local/lib/python3.10/dist-packages (from bokeh->umap-learn[plot]) (6.0.2) Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.10/dist-packages (from bokeh->umap-learn[plot]) (6.3.3) Requirement already satisfied: xyzservices>=2021.09.1 in /usr/local/lib/python3.10/dist-packages (from bokeh->umap-learn[plot]) (2024.9.0) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->umap-learn[plot]) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->umap-learn[plot]) (2024.2) Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->umap-learn[plot]) (2024.2) Requirement already satisfied: dask in /usr/local/lib/python3.10/dist-packages (from datashader->umap-learn[plot]) (2024.8.2) Requirement already satisfied: multipledispatch in /usr/local/lib/python3.10/dist-packages (from datashader->umap-learn[plot]) (1.0.0) Requirement already satisfied: param in /usr/local/lib/python3.10/dist-packages (from datashader->umap-learn[plot]) (2.1.1) Collecting pyct (from datashader->umap-learn[plot]) Downloading pyct-0.5.0-py2.py3-none-any.whl.metadata (7.4 kB) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from datashader->umap-learn[plot]) (2.32.3) Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from datashader->umap-learn[plot]) (0.12.1) Requirement already satisfied: xarray in /usr/local/lib/python3.10/dist-packages (from datashader->umap-learn[plot]) (2024.9.0) Requirement already satisfied: panel>=1.0 in /usr/local/lib/python3.10/dist-packages (from holoviews->umap-learn[plot]) (1.4.5) Requirement already satisfied: pyviz-comms>=2.1 in /usr/local/lib/python3.10/dist-packages (from holoviews->umap-learn[plot]) (3.0.3) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->umap-learn[plot]) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->umap-learn[plot]) (4.54.1) Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->umap-learn[plot]) (1.4.7) Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->umap-learn[plot]) (3.2.0) Requirement already satisfied: networkx>=2.8 in /usr/local/lib/python3.10/dist-packages (from scikit-image->umap-learn[plot]) (3.4.1) Requirement already satisfied: imageio>=2.33 in /usr/local/lib/python3.10/dist-packages (from scikit-image->umap-learn[plot]) (2.35.1) Requirement already satisfied: tifffile>=2022.8.12 in /usr/local/lib/python3.10/dist-packages (from scikit-image->umap-learn[plot]) (2024.9.20) Requirement already satisfied: lazy-loader>=0.4 in /usr/local/lib/python3.10/dist-packages (from scikit-image->umap-learn[plot]) (0.4) Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from Jinja2>=2.9->bokeh->umap-learn[plot]) (3.0.1) Requirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from panel>=1.0->holoviews->umap-learn[plot]) (3.7) Requirement already satisfied: markdown-it-py in /usr/local/lib/python3.10/dist-packages (from panel>=1.0->holoviews->umap-learn[plot]) (3.0.0) Requirement already satisfied: linkify-it-py in /usr/local/lib/python3.10/dist-packages (from panel>=1.0->holoviews->umap-learn[plot]) (2.0.3) Requirement already satisfied: mdit-py-plugins in /usr/local/lib/python3.10/dist-packages (from panel>=1.0->holoviews->umap-learn[plot]) (0.4.2) Requirement already satisfied: bleach in /usr/local/lib/python3.10/dist-packages (from panel>=1.0->holoviews->umap-learn[plot]) (6.1.0) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from panel>=1.0->holoviews->umap-learn[plot]) (4.12.2) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->umap-learn[plot]) (1.16.0) Requirement already satisfied: click>=8.1 in /usr/local/lib/python3.10/dist-packages (from dask->datashader->umap-learn[plot]) (8.1.7) Requirement already satisfied: cloudpickle>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from dask->datashader->umap-learn[plot]) (3.1.0) Requirement already satisfied: fsspec>=2021.09.0 in /usr/local/lib/python3.10/dist-packages (from dask->datashader->umap-learn[plot]) (2024.6.1) Requirement already satisfied: partd>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from dask->datashader->umap-learn[plot]) (1.4.2) Requirement already satisfied: importlib-metadata>=4.13.0 in /usr/local/lib/python3.10/dist-packages (from dask->datashader->umap-learn[plot]) (8.5.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->datashader->umap-learn[plot]) (3.4.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->datashader->umap-learn[plot]) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->datashader->umap-learn[plot]) (2.2.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->datashader->umap-learn[plot]) (2024.8.30) Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=4.13.0->dask->datashader->umap-learn[plot]) (3.20.2) Requirement already satisfied: locket in /usr/local/lib/python3.10/dist-packages (from partd>=1.4.0->dask->datashader->umap-learn[plot]) (1.0.0) Requirement already satisfied: webencodings in /usr/local/lib/python3.10/dist-packages (from bleach->panel>=1.0->holoviews->umap-learn[plot]) (0.5.1) Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.10/dist-packages (from linkify-it-py->panel>=1.0->holoviews->umap-learn[plot]) (1.0.3) Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py->panel>=1.0->holoviews->umap-learn[plot]) (0.1.2) Downloading pynndescent-0.5.13-py3-none-any.whl (56 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.9/56.9 kB 4.0 MB/s eta 0:00:00 Downloading datashader-0.16.3-py2.py3-none-any.whl (18.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.3/18.3 MB 70.8 MB/s eta 0:00:00 Downloading umap_learn-0.5.6-py3-none-any.whl (85 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.7/85.7 kB 6.5 MB/s eta 0:00:00 Downloading pyct-0.5.0-py2.py3-none-any.whl (15 kB) Installing collected packages: pyct, pynndescent, umap-learn, datashader Successfully installed datashader-0.16.3 pyct-0.5.0 pynndescent-0.5.13 umap-learn-0.5.6
Download and View the Dataset
[ ]
[2]
[{'title': 'Minions',
, 'overview': 'Minions Stuart, Kevin and Bob are recruited by Scarlet Overkill, a super-villain who, alongside her inventor husband Herb, hatches a plot to take over the world.',
, 'director': 'Kyle Balda',
, 'genres': 'Family Animation Adventure Comedy',
, 'tagline': 'Before Gru, they had a history of bad bosses'},
, {'title': 'Interstellar',
, 'overview': 'Interstellar chronicles the adventures of a group of explorers who make use of a newly discovered wormhole to surpass the limitations on human space travel and conquer the vast distances involved in an interstellar voyage.',
, 'director': 'Christopher Nolan',
, 'genres': 'Adventure Drama Science Fiction',
, 'tagline': 'Mankind was born on Earth. It was never meant to die here.'},
, {'title': 'Deadpool',
, 'overview': 'Deadpool tells the origin story of former Special Forces operative turned mercenary Wade Wilson, who after being subjected to a rogue experiment that leaves him with accelerated healing powers, adopts the alter ego Deadpool. Armed with his new abilities and a dark, twisted sense of humor, Deadpool hunts down the man who nearly destroyed his life.',
, 'director': 'Tim Miller',
, 'genres': 'Action Adventure Comedy',
, 'tagline': 'Witness the beginning of a happy ending'}] We've got popular movie titles along with information about their genres, plot overviews and more.
Generate Embeddings
We will use the Together AI embedding API to generate vector representations for each movie title + overview + tagline.
[3]
[8]
[5]
['Minions Minions Stuart, Kevin and Bob are recruited by Scarlet Overkill, a super-villain who, alongside her inventor husband Herb, hatches a plot to take over the world. Before Gru, they had a history of bad bosses', , 'Interstellar Interstellar chronicles the adventures of a group of explorers who make use of a newly discovered wormhole to surpass the limitations on human space travel and conquer the vast distances involved in an interstellar voyage. Mankind was born on Earth. It was never meant to die here.', , 'Deadpool Deadpool tells the origin story of former Special Forces operative turned mercenary Wade Wilson, who after being subjected to a rogue experiment that leaves him with accelerated healing powers, adopts the alter ego Deadpool. Armed with his new abilities and a dark, twisted sense of humor, Deadpool hunts down the man who nearly destroyed his life. Witness the beginning of a happy ending', , 'Guardians of the Galaxy Light years from Earth, 26 years after being abducted, Peter Quill finds himself the prime target of a manhunt after discovering an orb wanted by Ronan the Accuser. All heroes start somewhere.', , "Mad Max: Fury Road An apocalyptic story set in the furthest reaches of our planet, in a stark desert landscape where humanity is broken, and most everyone is crazed fighting for the necessities of life. Within this world exist two rebels on the run who just might be able to restore order. There's Max, a man of action and a man of few words, who seeks peace of mind following the loss of his wife and child in the aftermath of the chaos. And Furiosa, a woman of action and a woman who believes her path to survival may be achieved if she can make it across the desert back to her childhood homeland. What a Lovely Day.", , 'Jurassic World Twenty-two years after the events of Jurassic Park, Isla Nublar now features a fully functioning dinosaur theme park, Jurassic World, as originally envisioned by John Hammond. The park is open.', , "Pirates of the Caribbean: The Curse of the Black Pearl Jack Sparrow, a freewheeling 17th-century pirate who roams the Caribbean Sea, butts heads with a rival pirate bent on pillaging the village of Port Royal. When the governor's daughter is kidnapped, Sparrow decides to help the girl's love save her. But their seafaring mission is hardly simple. Prepare to be blown out of the water.", , 'Dawn of the Planet of the Apes A group of scientists in San Francisco struggle to stay alive in the aftermath of a plague that is wiping out humanity, while Caesar tries to maintain dominance over his community of intelligent apes. One last chance for peace.', , 'The Hunger Games: Mockingjay - Part 1 Katniss Everdeen reluctantly becomes the symbol of a mass rebellion against the autocratic Capitol. Fire burns brighter in the darkness', , 'Big Hero 6 The special bond that develops between plus-sized inflatable robot Baymax, and prodigy Hiro Hamada, who team up with a group of friends to form a band of high-tech heroes. From the creators of Wreck-it Ralph and Frozen']
[6]
[9]
(1000, 768)
[38]
{'Action',
, 'Adventure',
, 'Animation',
, 'Comedy',
, 'Crime',
, 'Drama',
, 'Family',
, 'Fantasy',
, 'History',
, 'Horror',
, 'Music',
, 'Mystery',
, 'Romance',
, 'Science',
, 'Thriller',
, 'War',
, 'Western'} Visualize Embeddings using UMAP
UMAP is a technique that uses local structure of the high dimensional vectors to reduce them down to a very low dimensional space. In our case below we will be projecting each 768 dimensional vector down to 2 dimensions and then visualizing it.
[37]
/usr/local/lib/python3.10/dist-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")
Here we can see that there is some clustering with respect to the movie genres, however there is not a clear pattern - this is probably because the genres for each movie can overlap and each movie actually has multiple genres.
Learn more about how to use embedding models in the docs here!