Blog Post Marengo Elasticsearch Bedrock
openai-chatgptlangchain-pythonchatgptgenaielasticsearchelasticopenaiAItwelvelabs-bedrock-marengochatlogvectordatabasePythonsearchgenaistacksupporting-blog-contentvectorelasticsearch-labslangchainapplications
Export
TwelveLabs Marengo Video Embedding Model + Bedrock + Elasticsearch
In this video we'll create a small app to search video embeddings from TwelveLabs' Marengo model to search trailers from recent summer box office hits. We'll use the AWS Bedrock integration for TwelveLabs so that our data never has to be persisted outside of our own S3 buckets.
To run this notebook.
-
You'll need an S3 bucket that can be written to by your AWS id.
-
You will need the host URL and an API key for your Elasticsearch, either deployed locally or in Elastic cloud. This notebook assumes you are running on Elasticearch serverless or 9+ but should be adapatable back to Elasticsearch 8.16 if you add a version to the
pip install -
You'll need a
.envfile with the following content. Alternatively you can hard code in your keys and configurations below.
DATA_DIR = "./data"
AWS_ACCESS_KEY_ID = "your_access_key_id"
AWS_SECRET_ACCESS_KEY = "your_secret_access_key"
S3_BUCKET_NAME = "your_bucket_name"
ELASTICSEARCH_API_KEY = "your_elasticsearch_api_key"
ELASTICSEARCH_ENDPOINT = "your_elasticsearch_endpoint_including_port_number"
-
Additionally, you'll need to enable the Marengo model in Bedrock for your account.
-
Note, if you are behind a VPN or running on a cloud hosted notebook (like Google Colab) you'll likely be blocked from grabbing source data with yt-dlp
[1]
[2]
True
[3]
[4]
[5]
Video already exists. Skipping video download. Metadata already exists. Loading metadata from file... Video already exists. Skipping video download. Metadata already exists. Loading metadata from file... Video already exists. Skipping video download. Metadata already exists. Loading metadata from file... Video already exists. Skipping video download. Metadata already exists. Loading metadata from file... Video already exists. Skipping video download. Metadata already exists. Loading metadata from file...
[6]
Titan's response: Here are the top 10 highest-grossing movies of all time, according to Box Office Mojo: 1. Avatar (2009) - $2.84 billion 2. Avengers: Endgame (2019) - $2.79 billion 3. Titanic (1997) - $2.19 billion 4. Star Wars: The Force Awakens (2015) - $2.07 billion 5. Avengers: Infinity War (2018) - $2.05 billion 6. Spider-Man: No Way Home (2021) - $1.92 billion 7. Jurassic World (2015) - $1.67 billion 8. The Lion King (2019) - $1.66 billion 9. The Avengers (2012) - $1.52 billion 10. Furious 7 (2015) - $1.51 billion It's worth noting that these numbers are adjusted for inflation and are based on worldwide box office receipts.
[7]
AWS Account ID: REDACTED ✅ Successfully connected to S3 bucket: REDACTED
[8]
Uploading youtube_VWqJifMMgZE to S3... Successfully uploaded youtube_VWqJifMMgZE to S3 Uploading youtube_Ox8ZLF6cGM0 to S3... Successfully uploaded youtube_Ox8ZLF6cGM0 to S3 Uploading youtube_jan5CFWs9ic to S3... Successfully uploaded youtube_jan5CFWs9ic to S3 Uploading youtube_qpoBjOg5RHU to S3... Successfully uploaded youtube_qpoBjOg5RHU to S3 Uploading youtube_22w7z_lT6YM to S3... Successfully uploaded youtube_22w7z_lT6YM to S3
[9]
Processing videos: 0%| | 0/5 [00:00<?, ?it/s]
embeddings/videos/VWqJifMMgZE No existing embedding found for youtube_VWqJifMMgZE. Video embedding task started: arn:aws:bedrock:us-east-1:461485115270:async-invoke/q41nyz3rg2ff
Processing videos: 20%|██ | 1/5 [00:22<01:31, 22.82s/it]
embeddings/videos/Ox8ZLF6cGM0 No existing embedding found for youtube_Ox8ZLF6cGM0. Video embedding task started: arn:aws:bedrock:us-east-1:461485115270:async-invoke/e6ujzy5ws8oi
Processing videos: 40%|████ | 2/5 [01:32<02:30, 50.19s/it]
embeddings/videos/jan5CFWs9ic No existing embedding found for youtube_jan5CFWs9ic. Video embedding task started: arn:aws:bedrock:us-east-1:461485115270:async-invoke/rzm787e3grrv
Processing videos: 60%|██████ | 3/5 [01:55<01:15, 37.82s/it]
embeddings/videos/qpoBjOg5RHU No existing embedding found for youtube_qpoBjOg5RHU. Video embedding task started: arn:aws:bedrock:us-east-1:461485115270:async-invoke/r4nscxlmcb1n
Processing videos: 80%|████████ | 4/5 [02:18<00:31, 31.94s/it]
embeddings/videos/22w7z_lT6YM No existing embedding found for youtube_22w7z_lT6YM. Video embedding task started: arn:aws:bedrock:us-east-1:461485115270:async-invoke/uwpeiuny19pf
Processing videos: 100%|██████████| 5/5 [02:41<00:00, 32.31s/it]
[10]
0 embedding: len 1024 embeddingOption: visual-text startSec: 0.0 endSec: 6.199999809265137 1 embedding: len 1024 embeddingOption: visual-text startSec: 6.199999809265137 endSec: 10.399999618530273 2 embedding: len 1024 embeddingOption: visual-text startSec: 10.399999618530273 endSec: 17.299999237060547
[11]
✅ Successfully connected to Elasticsearch: serverless
[12]
[13]
Deleting Index 'twelvelabs-movie-trailer-flat' and then sleeping for 2 seconds Index 'twelvelabs-movie-trailer-flat' created successfully Deleting Index 'twelvelabs-movie-trailer-hnsw' and then sleeping for 2 seconds Index 'twelvelabs-movie-trailer-hnsw' created successfully Deleting Index 'twelvelabs-movie-trailer-int8_hnsw' and then sleeping for 2 seconds Index 'twelvelabs-movie-trailer-int8_hnsw' created successfully Deleting Index 'twelvelabs-movie-trailer-bbq_hnsw' and then sleeping for 2 seconds Index 'twelvelabs-movie-trailer-bbq_hnsw' created successfully Deleting Index 'twelvelabs-movie-trailer-bbq_flat' and then sleeping for 2 seconds Index 'twelvelabs-movie-trailer-bbq_flat' created successfully Indexing 155 documents into twelvelabs-movie-trailer-flat... Successfully indexed 155 documents into twelvelabs-movie-trailer-flat Completed indexing documents into twelvelabs-movie-trailer-flat Indexing 155 documents into twelvelabs-movie-trailer-hnsw... Successfully indexed 155 documents into twelvelabs-movie-trailer-hnsw Completed indexing documents into twelvelabs-movie-trailer-hnsw Indexing 155 documents into twelvelabs-movie-trailer-int8_hnsw... Successfully indexed 155 documents into twelvelabs-movie-trailer-int8_hnsw Completed indexing documents into twelvelabs-movie-trailer-int8_hnsw Indexing 155 documents into twelvelabs-movie-trailer-bbq_hnsw... Successfully indexed 155 documents into twelvelabs-movie-trailer-bbq_hnsw Completed indexing documents into twelvelabs-movie-trailer-bbq_hnsw Indexing 155 documents into twelvelabs-movie-trailer-bbq_flat... Successfully indexed 155 documents into twelvelabs-movie-trailer-bbq_flat Completed indexing documents into twelvelabs-movie-trailer-bbq_flat
[14]
{'took': 8, 'timed_out': False, '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 10, 'relation': 'eq'}, 'max_score': 0.6417193, 'hits': [{'_index': 'twelvelabs-movie-trailer-flat', '_id': 'kjVNPpkBpEb67Rey0h3W', '_score': 0.6417193, 'fields': {'title': ['Jurassic World Rebirth | Official Trailer'], 'start_sec': [134.5], 'video_id': ['jan5CFWs9ic']}}, {'_index': 'twelvelabs-movie-trailer-flat', '_id': 'wDVNPpkBpEb67Rey1B3x', '_score': 0.6409597, 'fields': {'title': ['How To Train Your Dragon | Official Trailer'], 'start_sec': [59.0], 'video_id': ['22w7z_lT6YM']}}, {'_index': 'twelvelabs-movie-trailer-flat', '_id': 'uzVNPpkBpEb67Rey1B3x', '_score': 0.64061135, 'fields': {'title': ['How To Train Your Dragon | Official Trailer'], 'start_sec': [34.6], 'video_id': ['22w7z_lT6YM']}}, {'_index': 'twelvelabs-movie-trailer-flat', '_id': 'vjVNPpkBpEb67Rey1B3x', '_score': 0.63972485, 'fields': {'title': ['How To Train Your Dragon | Official Trailer'], 'start_sec': [50.399998], 'video_id': ['22w7z_lT6YM']}}, {'_index': 'twelvelabs-movie-trailer-flat', '_id': 'vTVNPpkBpEb67Rey1B3x', '_score': 0.6388538, 'fields': {'title': ['How To Train Your Dragon | Official Trailer'], 'start_sec': [46.3], 'video_id': ['22w7z_lT6YM']}}, {'_index': 'twelvelabs-movie-trailer-flat', '_id': 'ezVNPpkBpEb67Rey0h3W', '_score': 0.6381582, 'fields': {'title': ['Jurassic World Rebirth | Official Trailer'], 'start_sec': [18.8], 'video_id': ['jan5CFWs9ic']}}, {'_index': 'twelvelabs-movie-trailer-flat', '_id': 'vzVNPpkBpEb67Rey1B3x', '_score': 0.63632464, 'fields': {'title': ['How To Train Your Dragon | Official Trailer'], 'start_sec': [54.399998], 'video_id': ['22w7z_lT6YM']}}, {'_index': 'twelvelabs-movie-trailer-flat', '_id': 'zDVNPpkBpEb67Rey1B3x', '_score': 0.63516366, 'fields': {'title': ['How To Train Your Dragon | Official Trailer'], 'start_sec': [116.7], 'video_id': ['22w7z_lT6YM']}}, {'_index': 'twelvelabs-movie-trailer-flat', '_id': 'tDVNPpkBpEb67Rey1B3x', '_score': 0.6347714, 'fields': {'title': ['How To Train Your Dragon | Official Trailer'], 'start_sec': [0.0], 'video_id': ['22w7z_lT6YM']}}, {'_index': 'twelvelabs-movie-trailer-flat', '_id': 'vDVNPpkBpEb67Rey1B3x', '_score': 0.6343807, 'fields': {'title': ['How To Train Your Dragon | Official Trailer'], 'start_sec': [39.1], 'video_id': ['22w7z_lT6YM']}}]}}
[ ]
Screenshot of UI when done:
