Notebooks
L
LanceDB Main

Main

agentsllmsvector-databaselancedbgptopenaiAImultimodal-aimachine-learningembeddingsfine-tuningexamplesdeep-learninggpt-4-visionllama-indexragmultimodal_jina_clipv2multimodallangchainlancedb-recipes
alph-notebooks/lancedb-recipes / main.ipynb

Building Search engine using Jina CLIP v2: Multilingual Multimodal Embeddings for Text and Images & lancedb

Jina-CLIP v2, a 0.9B multimodal embedding model with multilingual support of 89 languages, high image resolution at 512x512, and Matryoshka representations.

In this tutorial we are using jina2 using huggigface libraries

Install libraries

[1]
Collecting lancedb
  Downloading lancedb-0.16.0-cp38-abi3-manylinux_2_28_x86_64.whl.metadata (4.8 kB)
Collecting deprecation (from lancedb)
  Downloading deprecation-2.1.0-py2.py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: nest-asyncio~=1.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (1.6.0)
Collecting pylance==0.19.2 (from lancedb)
  Downloading pylance-0.19.2-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (7.4 kB)
Requirement already satisfied: tqdm>=4.27.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (4.66.6)
Requirement already satisfied: pydantic>=1.10 in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.9.2)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from lancedb) (24.2)
Collecting overrides>=0.7 (from lancedb)
  Downloading overrides-7.7.0-py3-none-any.whl.metadata (5.8 kB)
Requirement already satisfied: pyarrow>=12 in /usr/local/lib/python3.10/dist-packages (from pylance==0.19.2->lancedb) (17.0.0)
Requirement already satisfied: numpy>=1.22 in /usr/local/lib/python3.10/dist-packages (from pylance==0.19.2->lancedb) (1.26.4)
Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (0.7.0)
Requirement already satisfied: pydantic-core==2.23.4 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (2.23.4)
Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (4.12.2)
Downloading lancedb-0.16.0-cp38-abi3-manylinux_2_28_x86_64.whl (27.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 27.4/27.4 MB 11.6 MB/s eta 0:00:00
Downloading pylance-0.19.2-cp39-abi3-manylinux_2_28_x86_64.whl (30.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 30.5/30.5 MB 10.5 MB/s eta 0:00:00
Downloading overrides-7.7.0-py3-none-any.whl (17 kB)
Downloading deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: overrides, deprecation, pylance, lancedb
Successfully installed deprecation-2.1.0 lancedb-0.16.0 overrides-7.7.0 pylance-0.19.2

In this tutorial we are using jina2 using huggigface libraries

The Jina CLIP v2 model can be loaded in various ways. In this instance, we're utilizing the SentenceTransformer-based approach. While this method may be slower due to CPU usage, faster embedding generation can be achieved by leveraging the CLIP 2 API. https://lancedb.github.io/lancedb/embeddings/available_embedding_models/multimodal_embedding_functions/jina_multimodal_embedding/

Below is sample example of using jina-clip-v2

we can do text based, Image based & nultilingual search

[2]
[3]
/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: 
The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
  warnings.warn(

modules.json:   0%|          | 0.00/273 [00:00<?, ?B/s]
config_sentence_transformers.json:   0%|          | 0.00/281 [00:00<?, ?B/s]
WARNING:sentence_transformers.SentenceTransformer:You try to use a model that was created with version 3.3.0, however, your version is 3.2.1. This might cause unexpected behavior or errors. In that case, try to update to the latest version.




README.md:   0%|          | 0.00/17.5k [00:00<?, ?B/s]
custom_st.py:   0%|          | 0.00/12.0k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-clip-v2:
- custom_st.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

config.json:   0%|          | 0.00/2.03k [00:00<?, ?B/s]
configuration_clip.py:   0%|          | 0.00/12.6k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-clip-implementation:
- configuration_clip.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

modeling_clip.py:   0%|          | 0.00/25.5k [00:00<?, ?B/s]
eva_model.py:   0%|          | 0.00/26.4k [00:00<?, ?B/s]
rope_embeddings.py:   0%|          | 0.00/5.71k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-clip-implementation:
- rope_embeddings.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-clip-implementation:
- eva_model.py
- rope_embeddings.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

hf_model.py:   0%|          | 0.00/12.9k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-clip-implementation:
- hf_model.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

transform.py:   0%|          | 0.00/12.7k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-clip-implementation:
- transform.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-clip-implementation:
- modeling_clip.py
- eva_model.py
- hf_model.py
- transform.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

model.safetensors:   0%|          | 0.00/1.73G [00:00<?, ?B/s]
/root/.cache/huggingface/modules/transformers_modules/jinaai/jina-clip-implementation/3742048faee267e47f6cd862e45fd7c240cbd205/modeling_clip.py:137: UserWarning: Flash attention requires CUDA, disabling
  warnings.warn('Flash attention requires CUDA, disabling')
/root/.cache/huggingface/modules/transformers_modules/jinaai/jina-clip-implementation/3742048faee267e47f6cd862e45fd7c240cbd205/modeling_clip.py:172: UserWarning: xFormers requires CUDA, disabling
  warnings.warn('xFormers requires CUDA, disabling')

config.json:   0%|          | 0.00/1.80k [00:00<?, ?B/s]
configuration_xlm_roberta.py:   0%|          | 0.00/6.54k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- configuration_xlm_roberta.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

modeling_lora.py:   0%|          | 0.00/15.4k [00:00<?, ?B/s]
modeling_xlm_roberta.py:   0%|          | 0.00/49.9k [00:00<?, ?B/s]
xlm_padding.py:   0%|          | 0.00/10.0k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- xlm_padding.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

embedding.py:   0%|          | 0.00/3.88k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- embedding.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

mlp.py:   0%|          | 0.00/7.62k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- mlp.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

block.py:   0%|          | 0.00/17.8k [00:00<?, ?B/s]
stochastic_depth.py:   0%|          | 0.00/3.76k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- stochastic_depth.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

mha.py:   0%|          | 0.00/34.4k [00:00<?, ?B/s]
rotary.py:   0%|          | 0.00/24.5k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- rotary.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- mha.py
- rotary.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- block.py
- stochastic_depth.py
- mha.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- modeling_xlm_roberta.py
- xlm_padding.py
- embedding.py
- mlp.py
- block.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- modeling_lora.py
- modeling_xlm_roberta.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

tokenizer_config.json:   0%|          | 0.00/1.15k [00:00<?, ?B/s]
tokenizer.json:   0%|          | 0.00/17.1M [00:00<?, ?B/s]
special_tokens_map.json:   0%|          | 0.00/964 [00:00<?, ?B/s]
tokenizer_config.json:   0%|          | 0.00/1.15k [00:00<?, ?B/s]
tokenizer.json:   0%|          | 0.00/17.1M [00:00<?, ?B/s]
special_tokens_map.json:   0%|          | 0.00/964 [00:00<?, ?B/s]
preprocessor_config.json:   0%|          | 0.00/584 [00:00<?, ?B/s]
processing_clip.py:   0%|          | 0.00/2.67k [00:00<?, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-clip-implementation:
- processing_clip.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.

Results from text query:
    label                                          image_uri  \
0     cat  http://farm1.staticflickr.com/53/167798175_7c7...   
1  बिल्ली  http://farm1.staticflickr.com/134/332220238_da...   
2       狗  http://farm5.staticflickr.com/4092/5017326486_...   

                                              vector  _distance  
0  [-0.014676968, 0.08036212, 0.005830338, 0.0749...   1.366044  
1  [-0.014399904, 0.10926569, -0.0125547685, 0.08...   1.403262  
2  [0.028017132, 0.122899786, -0.010693597, 0.034...   1.538371

Query example via Image

we are doing image based search now for the above example

[4]
Raw query_image_embedding shape: (1, 512)
Raw query_image_embedding type: <class 'numpy.ndarray'>
Final query_image_embedding shape: 512
Final query_image_embedding type: <class 'list'>
Image Query Results:
    label                                          image_uri  \
0   horse  http://farm6.staticflickr.com/5142/5835678453_...   
1   horse  http://farm9.staticflickr.com/8216/8434969557_...   
2  बिल्ली  http://farm1.staticflickr.com/134/332220238_da...   

                                              vector     _distance  
0  [-0.0058654496, 0.102436356, 0.07907705, 0.106...  7.908797e-14  
1  [0.02997741, 0.049638063, 0.006789788, 0.11391...  4.896866e-01  
2  [-0.014399904, 0.10926569, -0.0125547685, 0.08...  7.422640e-01

This is how you can get results based on text & images & multilingual query

Lets build Food recommandation based on jina v2 multimodal embedding & lancdb

Download full food-101 dataset from here Kaggle

For this notebook demo I used only 2 classes data

Organizing Data into Folders and Preparing Labels and Paths for Embedding Steps

Folder Structure:

  • new_food_2
    • Rice
    • Vegetable-Fruit Dataset
[5]
Mounted at /content/drive

Creating a CSV File to Store Image Paths and Their Corresponding Labels

[6]
CSV file created at /content/drive/MyDrive/small_food/image_labels.csv with 98 entries.
[7]

Now based on image & labels we are doing embedding & saving it in lancdb

[8]
WARNING:sentence_transformers.SentenceTransformer:You try to use a model that was created with version 3.3.0, however, your version is 3.2.1. This might cause unexpected behavior or errors. In that case, try to update to the latest version.



/root/.cache/huggingface/modules/transformers_modules/jinaai/jina-clip-implementation/3742048faee267e47f6cd862e45fd7c240cbd205/modeling_clip.py:137: UserWarning: Flash attention requires CUDA, disabling
  warnings.warn('Flash attention requires CUDA, disabling')
/root/.cache/huggingface/modules/transformers_modules/jinaai/jina-clip-implementation/3742048faee267e47f6cd862e45fd7c240cbd205/modeling_clip.py:172: UserWarning: xFormers requires CUDA, disabling
  warnings.warn('xFormers requires CUDA, disabling')

Processing 1/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/10.jpg
Processing 2/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/11.jpg
Processing 3/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/0.jpg
Processing 4/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/1.jpg
Processing 5/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/13.jpg
Processing 6/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/15.jpg
Processing 7/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/17.jpg
Processing 8/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/12.jpg
Processing 9/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/16.jpg
Processing 10/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/14.jpg
Processing 11/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/20.jpg
Processing 12/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/2.jpg
Processing 13/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/18.jpg
Processing 14/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/19.jpg
Processing 15/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/21.jpg
Processing 16/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/5.jpg
Processing 17/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/33.jpg
Processing 18/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/46.jpg
Processing 19/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/41.jpg
Processing 20/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/27.jpg
Processing 21/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/49.jpg
Processing 22/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/44.jpg
Processing 23/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/4.jpg
Processing 24/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/23.jpg
Processing 25/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/24.jpg
Processing 26/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/42.jpg
Processing 27/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/36.jpg
Processing 28/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/34.jpg
Processing 29/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/31.jpg
Processing 30/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/45.jpg
Processing 31/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/6.jpg
Processing 32/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/50.jpg
Processing 33/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/28.jpg
Processing 34/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/25.jpg
Processing 35/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/3.jpg
Processing 36/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/38.jpg
Processing 37/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/39.jpg
Processing 38/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/48.jpg
Processing 39/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/35.jpg
Processing 40/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/37.jpg
Processing 41/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/43.jpg
Processing 42/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/47.jpg
Processing 43/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/29.jpg
Processing 44/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/32.jpg
Processing 45/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/30.jpg
Processing 46/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/40.jpg
Processing 47/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/26.jpg
Processing 48/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/22.jpg
Processing 49/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/8.jpg
Processing 50/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/9.jpg
Processing 51/98: /content/drive/MyDrive/small_food/new_food_2/Vegetable-Fruit/7.jpg
Processing 52/98: /content/drive/MyDrive/small_food/new_food_2/Rice/3.jpg
Processing 53/98: /content/drive/MyDrive/small_food/new_food_2/Rice/19.jpg
Processing 54/98: /content/drive/MyDrive/small_food/new_food_2/Rice/16.jpg
Processing 55/98: /content/drive/MyDrive/small_food/new_food_2/Rice/13.jpg
Processing 56/98: /content/drive/MyDrive/small_food/new_food_2/Rice/10.jpg
Processing 57/98: /content/drive/MyDrive/small_food/new_food_2/Rice/34.jpg
Processing 58/98: /content/drive/MyDrive/small_food/new_food_2/Rice/6.jpg
Processing 59/98: /content/drive/MyDrive/small_food/new_food_2/Rice/43.jpg
Processing 60/98: /content/drive/MyDrive/small_food/new_food_2/Rice/27.jpg
Processing 61/98: /content/drive/MyDrive/small_food/new_food_2/Rice/32.jpg
Processing 62/98: /content/drive/MyDrive/small_food/new_food_2/Rice/2.jpg
Processing 63/98: /content/drive/MyDrive/small_food/new_food_2/Rice/20.jpg
Processing 64/98: /content/drive/MyDrive/small_food/new_food_2/Rice/33.jpg
Processing 65/98: /content/drive/MyDrive/small_food/new_food_2/Rice/17.jpg
Processing 66/98: /content/drive/MyDrive/small_food/new_food_2/Rice/9.jpg
Processing 67/98: /content/drive/MyDrive/small_food/new_food_2/Rice/1.jpg
Processing 68/98: /content/drive/MyDrive/small_food/new_food_2/Rice/37.jpg
Processing 69/98: /content/drive/MyDrive/small_food/new_food_2/Rice/25.jpg
Processing 70/98: /content/drive/MyDrive/small_food/new_food_2/Rice/48.jpg
Processing 71/98: /content/drive/MyDrive/small_food/new_food_2/Rice/30.jpg
Processing 72/98: /content/drive/MyDrive/small_food/new_food_2/Rice/14.jpg
Processing 73/98: /content/drive/MyDrive/small_food/new_food_2/Rice/7.jpg
Processing 74/98: /content/drive/MyDrive/small_food/new_food_2/Rice/41.jpg
Processing 75/98: /content/drive/MyDrive/small_food/new_food_2/Rice/0.jpg
Processing 76/98: /content/drive/MyDrive/small_food/new_food_2/Rice/4.jpg
Processing 77/98: /content/drive/MyDrive/small_food/new_food_2/Rice/5.jpg
Processing 78/98: /content/drive/MyDrive/small_food/new_food_2/Rice/46.jpg
Processing 79/98: /content/drive/MyDrive/small_food/new_food_2/Rice/45.jpg
Processing 80/98: /content/drive/MyDrive/small_food/new_food_2/Rice/15.jpg
Processing 81/98: /content/drive/MyDrive/small_food/new_food_2/Rice/8.jpg
Processing 82/98: /content/drive/MyDrive/small_food/new_food_2/Rice/40.jpg
Processing 83/98: /content/drive/MyDrive/small_food/new_food_2/Rice/21.jpg
Processing 84/98: /content/drive/MyDrive/small_food/new_food_2/Rice/24.jpg
Processing 85/98: /content/drive/MyDrive/small_food/new_food_2/Rice/11.jpg
Processing 86/98: /content/drive/MyDrive/small_food/new_food_2/Rice/50.jpg
Processing 87/98: /content/drive/MyDrive/small_food/new_food_2/Rice/29.jpg
Processing 88/98: /content/drive/MyDrive/small_food/new_food_2/Rice/18.jpg
Processing 89/98: /content/drive/MyDrive/small_food/new_food_2/Rice/12.jpg
Processing 90/98: /content/drive/MyDrive/small_food/new_food_2/Rice/26.jpg
Processing 91/98: /content/drive/MyDrive/small_food/new_food_2/Rice/35.jpg
Processing 92/98: /content/drive/MyDrive/small_food/new_food_2/Rice/28.jpg
Processing 93/98: /content/drive/MyDrive/small_food/new_food_2/Rice/47.jpg
Processing 94/98: /content/drive/MyDrive/small_food/new_food_2/Rice/31.jpg
Processing 95/98: /content/drive/MyDrive/small_food/new_food_2/Rice/22.jpg
Processing 96/98: /content/drive/MyDrive/small_food/new_food_2/Rice/23.jpg
Processing 97/98: /content/drive/MyDrive/small_food/new_food_2/Rice/38.jpg
Processing 98/98: /content/drive/MyDrive/small_food/new_food_2/Rice/39.jpg

Text based query search

[9]
Results from text query:
             label                                          image_uri  \
0  Vegetable-Fruit  /content/drive/MyDrive/small_food/new_food_2/V...   
1  Vegetable-Fruit  /content/drive/MyDrive/small_food/new_food_2/V...   
2  Vegetable-Fruit  /content/drive/MyDrive/small_food/new_food_2/V...   

                                              vector  _distance  
0  [0.053401865, 0.06045878, 0.027279321, 0.09057...   1.451658  
1  [0.020733627, 0.069162644, 0.029952297, 0.0533...   1.501224  
2  [0.014536029, 0.073719315, 0.0315481, 0.080828...   1.516304
[10]
Results from text query:
  label                                          image_uri  \
0  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   
1  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   
2  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   
3  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   
4  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   

                                              vector  _distance  
0  [-0.03897679, 0.17268911, 0.026397461, 0.05065...   1.246697  
1  [0.01770382, 0.17614241, 0.021768652, 0.094345...   1.352594  
2  [0.013835855, 0.110533215, 0.060185626, 0.1150...   1.354973  
3  [0.018703124, 0.12488442, 0.03489212, 0.107657...   1.357902  
4  [0.038268246, 0.16299663, 0.004079318, 0.07822...   1.357914

Image based search

[13]
Query Image:
Output
Raw query_image_embedding shape: (1, 512)
Raw query_image_embedding type: <class 'numpy.ndarray'>
Image Query Results:
             label                                          image_uri  \
0  Vegetable-Fruit  /content/drive/MyDrive/small_food/new_food_2/V...   
1  Vegetable-Fruit  /content/drive/MyDrive/small_food/new_food_2/V...   
2  Vegetable-Fruit  /content/drive/MyDrive/small_food/new_food_2/V...   
3  Vegetable-Fruit  /content/drive/MyDrive/small_food/new_food_2/V...   
4  Vegetable-Fruit  /content/drive/MyDrive/small_food/new_food_2/V...   

                                              vector  _distance  
0  [0.03398791, 0.095794275, -0.018574353, 0.0337...   0.000000  
1  [0.032099266, 0.040508382, 0.014722838, 0.0354...   0.112821  
2  [0.02648662, 0.04464159, 0.002022647, 0.073952...   0.128473  
3  [0.040251896, 0.060741726, 0.039796557, 0.0159...   0.138648  
4  [0.04798884, 0.10667701, 0.039873254, 0.057920...   0.147284  

Retrieved Images:
Label: Vegetable-Fruit, Distance: 0.0
Output
Label: Vegetable-Fruit, Distance: 0.11282147467136383
Output
Label: Vegetable-Fruit, Distance: 0.1284734308719635
Output
Label: Vegetable-Fruit, Distance: 0.13864830136299133
Output
Label: Vegetable-Fruit, Distance: 0.14728376269340515
Output
[12]
Query Image:
Output
Raw query_image_embedding shape: (1, 512)
Raw query_image_embedding type: <class 'numpy.ndarray'>
Image Query Results:
  label                                          image_uri  \
0  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   
1  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   
2  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   
3  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   
4  Rice  /content/drive/MyDrive/small_food/new_food_2/R...   

                                              vector  _distance  
0  [0.10215698, 0.03488325, 0.07692675, 0.1340418...   0.000000  
1  [0.07078372, 0.1694398, 0.055898074, 0.1191742...   0.303376  
2  [0.056605097, 0.12720048, 0.031542275, 0.13055...   0.322529  
3  [0.019074202, 0.07827992, 0.06916776, 0.127158...   0.347323  
4  [0.022701448, 0.16595668, 0.038484853, 0.07074...   0.357251  

Retrieved Images:
Label: Rice, Distance: 0.0
Output
Label: Rice, Distance: 0.30337584018707275
Output
Label: Rice, Distance: 0.32252901792526245
Output
Label: Rice, Distance: 0.3473229706287384
Output
Label: Rice, Distance: 0.3572513163089752
Output

Please check our other coverd topics related to multimodal-embeeding