Notebooks
L
LanceDB Main

Main

archived_examplesagentsllmsvector-databaselancedbgptopenaiAImultimodal-aimachine-learningembeddingsfine-tuningzero-shot-image-classificationexamplesdeep-learninggpt-4-visionllama-indexragmultimodallangchainlancedb-recipes
alph-notebooks/lancedb-recipes / main.ipynb
[2]
Collecting datasets
  Downloading datasets-2.20.0-py3-none-any.whl (547 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 547.8/547.8 kB 4.7 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets) (3.15.4)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (1.25.2)
Collecting pyarrow>=15.0.0 (from datasets)
  Downloading pyarrow-16.1.0-cp310-cp310-manylinux_2_28_x86_64.whl (40.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 MB 12.4 MB/s eta 0:00:00
Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets) (0.6)
Collecting dill<0.3.9,>=0.3.0 (from datasets)
  Downloading dill-0.3.8-py3-none-any.whl (116 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 11.5 MB/s eta 0:00:00
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (2.0.3)
Collecting requests>=2.32.2 (from datasets)
  Downloading requests-2.32.3-py3-none-any.whl (64 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 7.1 MB/s eta 0:00:00
Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.10/dist-packages (from datasets) (4.66.4)
Collecting xxhash (from datasets)
  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 17.0 MB/s eta 0:00:00
Collecting multiprocess (from datasets)
  Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 15.4 MB/s eta 0:00:00
Requirement already satisfied: fsspec[http]<=2024.5.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2023.6.0)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.9.5)
Requirement already satisfied: huggingface-hub>=0.21.2 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.23.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets) (24.1)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (6.0.1)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.2->datasets) (4.12.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (2024.6.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2023.4)
Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)
Installing collected packages: xxhash, requests, pyarrow, dill, multiprocess, datasets
  Attempting uninstall: requests
    Found existing installation: requests 2.31.0
    Uninstalling requests-2.31.0:
      Successfully uninstalled requests-2.31.0
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 14.0.2
    Uninstalling pyarrow-14.0.2:
      Successfully uninstalled pyarrow-14.0.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 16.1.0 which is incompatible.
google-colab 1.0.0 requires requests==2.31.0, but you have requests 2.32.3 which is incompatible.
ibis-framework 8.0.0 requires pyarrow<16,>=2, but you have pyarrow 16.1.0 which is incompatible.
Successfully installed datasets-2.20.0 dill-0.3.8 multiprocess-0.70.16 pyarrow-16.1.0 requests-2.32.3 xxhash-3.4.1
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.41.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.15.4)
Requirement already satisfied: huggingface-hub<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.23.4)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.25.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.1)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.5.15)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.3)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.19.1)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.3)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.4)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.0->transformers) (2023.6.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.0->transformers) (4.12.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.6.2)
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.3.0+cu121)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.15.4)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch) (4.12.2)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch) (1.12.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2023.6.0)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch)
  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch)
  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch)
  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch)
  Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch)
  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch)
  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
Collecting nvidia-curand-cu12==10.3.2.106 (from torch)
  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch)
  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch)
  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
Collecting nvidia-nccl-cu12==2.20.5 (from torch)
  Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch)
  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
Requirement already satisfied: triton==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torch) (2.3.0)
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch)
  Downloading nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl (21.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.3/21.3 MB 55.9 MB/s eta 0:00:00
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (2.1.5)
Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch) (1.3.0)
Installing collected packages: nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, nvidia-cusparse-cu12, nvidia-cudnn-cu12, nvidia-cusolver-cu12
Successfully installed nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.20.5 nvidia-nvjitlink-cu12-12.5.82 nvidia-nvtx-cu12-12.1.105
Collecting lancedb
  Downloading lancedb-0.9.0-cp38-abi3-manylinux_2_28_x86_64.whl (20.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.9/20.9 MB 49.5 MB/s eta 0:00:00
Collecting deprecation (from lancedb)
  Downloading deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting pylance==0.13.0 (from lancedb)
  Downloading pylance-0.13.0-cp39-abi3-manylinux_2_28_x86_64.whl (25.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.5/25.5 MB 47.6 MB/s eta 0:00:00
Collecting ratelimiter~=1.0 (from lancedb)
  Downloading ratelimiter-1.2.0.post0-py3-none-any.whl (6.6 kB)
Requirement already satisfied: requests>=2.31.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.32.3)
Collecting retry>=0.9.2 (from lancedb)
  Downloading retry-0.9.2-py2.py3-none-any.whl (8.0 kB)
Requirement already satisfied: tqdm>=4.27.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (4.66.4)
Requirement already satisfied: pydantic>=1.10 in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.8.0)
Requirement already satisfied: attrs>=21.3.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (23.2.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from lancedb) (24.1)
Requirement already satisfied: cachetools in /usr/local/lib/python3.10/dist-packages (from lancedb) (5.3.3)
Collecting overrides>=0.7 (from lancedb)
  Downloading overrides-7.7.0-py3-none-any.whl (17 kB)
Collecting pyarrow<15.0.1,>=12 (from pylance==0.13.0->lancedb)
  Downloading pyarrow-15.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (38.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.3/38.3 MB 12.3 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.22 in /usr/local/lib/python3.10/dist-packages (from pylance==0.13.0->lancedb) (1.25.2)
Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (0.7.0)
Requirement already satisfied: pydantic-core==2.20.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (2.20.0)
Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (4.12.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb) (2024.6.2)
Requirement already satisfied: decorator>=3.4.2 in /usr/local/lib/python3.10/dist-packages (from retry>=0.9.2->lancedb) (4.4.2)
Collecting py<2.0.0,>=1.4.26 (from retry>=0.9.2->lancedb)
  Downloading py-1.11.0-py2.py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.7/98.7 kB 8.8 MB/s eta 0:00:00
Installing collected packages: ratelimiter, pyarrow, py, overrides, deprecation, retry, pylance, lancedb
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 16.1.0
    Uninstalling pyarrow-16.1.0:
      Successfully uninstalled pyarrow-16.1.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 15.0.0 which is incompatible.
Successfully installed deprecation-2.1.0 lancedb-0.9.0 overrides-7.7.0 py-1.11.0 pyarrow-15.0.0 pylance-0.13.0 ratelimiter-1.2.0.post0 retry-0.9.2
[3]
/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: 
The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
  warnings.warn(

Downloading readme:   0%|          | 0.00/9.98k [00:00<?, ?B/s]
Downloading data:   0%|          | 0.00/119M [00:00<?, ?B/s]
Downloading data:   0%|          | 0.00/23.8M [00:00<?, ?B/s]
Generating train split:   0%|          | 0/50000 [00:00<?, ? examples/s]
Generating test split:   0%|          | 0/10000 [00:00<?, ? examples/s]
Dataset({
,    features: ['img', 'fine_label', 'coarse_label'],
,    num_rows: 10000
,})
[ ]
[5]
100

['apple',
, 'aquarium_fish',
, 'baby',
, 'bear',
, 'beaver',
, 'bed',
, 'bee',
, 'beetle',
, 'bicycle',
, 'bottle',
, 'bowl',
, 'boy',
, 'bridge',
, 'bus',
, 'butterfly',
, 'camel',
, 'can',
, 'castle',
, 'caterpillar',
, 'cattle',
, 'chair',
, 'chimpanzee',
, 'clock',
, 'cloud',
, 'cockroach',
, 'couch',
, 'cra',
, 'crocodile',
, 'cup',
, 'dinosaur',
, 'dolphin',
, 'elephant',
, 'flatfish',
, 'forest',
, 'fox',
, 'girl',
, 'hamster',
, 'house',
, 'kangaroo',
, 'keyboard',
, 'lamp',
, 'lawn_mower',
, 'leopard',
, 'lion',
, 'lizard',
, 'lobster',
, 'man',
, 'maple_tree',
, 'motorcycle',
, 'mountain',
, 'mouse',
, 'mushroom',
, 'oak_tree',
, 'orange',
, 'orchid',
, 'otter',
, 'palm_tree',
, 'pear',
, 'pickup_truck',
, 'pine_tree',
, 'plain',
, 'plate',
, 'poppy',
, 'porcupine',
, 'possum',
, 'rabbit',
, 'raccoon',
, 'ray',
, 'road',
, 'rocket',
, 'rose',
, 'sea',
, 'seal',
, 'shark',
, 'shrew',
, 'skunk',
, 'skyscraper',
, 'snail',
, 'snake',
, 'spider',
, 'squirrel',
, 'streetcar',
, 'sunflower',
, 'sweet_pepper',
, 'table',
, 'tank',
, 'telephone',
, 'television',
, 'tiger',
, 'tractor',
, 'train',
, 'trout',
, 'tulip',
, 'turtle',
, 'wardrobe',
, 'whale',
, 'willow_tree',
, 'wolf',
, 'woman',
, 'worm']
[6]
['This is an image for apple',
, 'This is an image for aquarium_fish',
, 'This is an image for baby',
, 'This is an image for bear',
, 'This is an image for beaver',
, 'This is an image for bed',
, 'This is an image for bee',
, 'This is an image for beetle',
, 'This is an image for bicycle',
, 'This is an image for bottle',
, 'This is an image for bowl',
, 'This is an image for boy',
, 'This is an image for bridge',
, 'This is an image for bus',
, 'This is an image for butterfly',
, 'This is an image for camel',
, 'This is an image for can',
, 'This is an image for castle',
, 'This is an image for caterpillar',
, 'This is an image for cattle',
, 'This is an image for chair',
, 'This is an image for chimpanzee',
, 'This is an image for clock',
, 'This is an image for cloud',
, 'This is an image for cockroach',
, 'This is an image for couch',
, 'This is an image for cra',
, 'This is an image for crocodile',
, 'This is an image for cup',
, 'This is an image for dinosaur',
, 'This is an image for dolphin',
, 'This is an image for elephant',
, 'This is an image for flatfish',
, 'This is an image for forest',
, 'This is an image for fox',
, 'This is an image for girl',
, 'This is an image for hamster',
, 'This is an image for house',
, 'This is an image for kangaroo',
, 'This is an image for keyboard',
, 'This is an image for lamp',
, 'This is an image for lawn_mower',
, 'This is an image for leopard',
, 'This is an image for lion',
, 'This is an image for lizard',
, 'This is an image for lobster',
, 'This is an image for man',
, 'This is an image for maple_tree',
, 'This is an image for motorcycle',
, 'This is an image for mountain',
, 'This is an image for mouse',
, 'This is an image for mushroom',
, 'This is an image for oak_tree',
, 'This is an image for orange',
, 'This is an image for orchid',
, 'This is an image for otter',
, 'This is an image for palm_tree',
, 'This is an image for pear',
, 'This is an image for pickup_truck',
, 'This is an image for pine_tree',
, 'This is an image for plain',
, 'This is an image for plate',
, 'This is an image for poppy',
, 'This is an image for porcupine',
, 'This is an image for possum',
, 'This is an image for rabbit',
, 'This is an image for raccoon',
, 'This is an image for ray',
, 'This is an image for road',
, 'This is an image for rocket',
, 'This is an image for rose',
, 'This is an image for sea',
, 'This is an image for seal',
, 'This is an image for shark',
, 'This is an image for shrew',
, 'This is an image for skunk',
, 'This is an image for skyscraper',
, 'This is an image for snail',
, 'This is an image for snake',
, 'This is an image for spider',
, 'This is an image for squirrel',
, 'This is an image for streetcar',
, 'This is an image for sunflower',
, 'This is an image for sweet_pepper',
, 'This is an image for table',
, 'This is an image for tank',
, 'This is an image for telephone',
, 'This is an image for television',
, 'This is an image for tiger',
, 'This is an image for tractor',
, 'This is an image for train',
, 'This is an image for trout',
, 'This is an image for tulip',
, 'This is an image for turtle',
, 'This is an image for wardrobe',
, 'This is an image for whale',
, 'This is an image for willow_tree',
, 'This is an image for wolf',
, 'This is an image for woman',
, 'This is an image for worm']
[7]
preprocessor_config.json:   0%|          | 0.00/316 [00:00<?, ?B/s]
tokenizer_config.json:   0%|          | 0.00/905 [00:00<?, ?B/s]
vocab.json:   0%|          | 0.00/961k [00:00<?, ?B/s]
merges.txt:   0%|          | 0.00/525k [00:00<?, ?B/s]
tokenizer.json:   0%|          | 0.00/2.22M [00:00<?, ?B/s]
special_tokens_map.json:   0%|          | 0.00/389 [00:00<?, ?B/s]
config.json:   0%|          | 0.00/4.52k [00:00<?, ?B/s]
model.safetensors:   0%|          | 0.00/1.71G [00:00<?, ?B/s]
[8]
'cpu'
[9]
Token ID : tensor([49406,   589,   533,   550,  2867,   556,  3055, 49407, 49407, 49407]), Text : <|startoftext|>this is an image for apple <|endoftext|><|endoftext|><|endoftext|>
Token ID : tensor([49406,   589,   533,   550,  2867,   556, 16814,   318,  2759, 49407]), Text : <|startoftext|>this is an image for aquarium _ fish <|endoftext|>
Token ID : tensor([49406,   589,   533,   550,  2867,   556,  1794, 49407, 49407, 49407]), Text : <|startoftext|>this is an image for baby <|endoftext|><|endoftext|><|endoftext|>
Token ID : tensor([49406,   589,   533,   550,  2867,   556,  4298, 49407, 49407, 49407]), Text : <|startoftext|>this is an image for bear <|endoftext|><|endoftext|><|endoftext|>
Token ID : tensor([49406,   589,   533,   550,  2867,   556, 22874, 49407, 49407, 49407]), Text : <|startoftext|>this is an image for beaver <|endoftext|><|endoftext|><|endoftext|>

Creating the embeddings!

When you're working with zero-shot image classification using CLIP (Contrastive Language-Image Pre-Training), you're essentially leveraging the ability of the CLIP model to understand both images and text by mapping them into a shared embedding space. This shared space allows the model to compute the similarity between text and image embeddings, making it ideal for zero-shot tasks.

Given that you're using CLIP for zero-shot classification, it's important to use the embeddings generated by the CLIP model for both images and text. The reason for this is that CLIP has been specifically trained to align these two modalities (text and images) in the same embedding space. Using different embeddings would break this alignment and undermine the model's ability to perform zero-shot classification effectively.

However, if you want to explore other embeddings, you could consider using different pre-trained models for other tasks or experiments, but for the zero-shot classification task with CLIP, sticking with the CLIP-generated embeddings is essential.

[10]
{'input_ids': tensor([[49406,   589,   533,   550,  2867,   556,  3055, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 16814,   318,  2759, 49407],
        [49406,   589,   533,   550,  2867,   556,  1794, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  4298, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 22874, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  2722, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  5028, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 16534, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 11652, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  5392, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  3814, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  1876, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  2465, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  2840, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  9738, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 21914, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,   753, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  3540, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 27111, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 13644, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  4269, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 10543,  1072, 14080, 49407],
        [49406,   589,   533,   550,  2867,   556,  6716, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  3887, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,   622,   916, 31073, 49407],
        [49406,   589,   533,   550,  2867,   556, 12724, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 18362, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 24757, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  1937, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 15095, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 16464, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 10299, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  8986,  2759, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  4167, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  3240, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  1611, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 33313, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  1212, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 25513, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 13017, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 10725, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 11024,   318, 30895, 49407],
        [49406,   589,   533,   550,  2867,   556, 15931, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  5567, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 17221, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 13793, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,   786, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 10570,   318,  2677, 49407],
        [49406,   589,   533,   550,  2867,   556, 10297, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  3965, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  9301, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 13011, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  7221,   318,  2677, 49407],
        [49406,   589,   533,   550,  2867,   556,  4287, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 18678, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 22456, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  8612,   318,  2677, 49407],
        [49406,   589,   533,   550,  2867,   556, 18820, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 15382,   318,  4629, 49407],
        [49406,   589,   533,   550,  2867,   556,  7374,   318,  2677, 49407],
        [49406,   589,   533,   550,  2867,   556, 10709, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  5135, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 15447, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,   817,  5059,   715, 49407],
        [49406,   589,   533,   550,  2867,   556, 38575, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 10274, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 29516, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  3077, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  1759, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  8383, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  3568, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  2102, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 10159, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  7980, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 12101,   342, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 42194, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  3075, 11187,  1284, 49407],
        [49406,   589,   533,   550,  2867,   556, 23132, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  8798, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  7622, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 14004, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 34268, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 21559, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  2418,   318,  8253, 49407],
        [49406,   589,   533,   550,  2867,   556,  2175, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  6172, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 17243, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  8608, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  6531, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 14607, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  3231, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 14853, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 28389, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 10912, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 15020, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 11650, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 15665,   318,  2677, 49407],
        [49406,   589,   533,   550,  2867,   556,  5916, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556,  2308, 49407, 49407, 49407],
        [49406,   589,   533,   550,  2867,   556, 10945, 49407, 49407, 49407]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0]])}
[11]
(100, 768)

We got the 768 dimensional vector for each of our 10 text classes sentences.

Now there needs to be the normalizing the embeddings to make sure we get the better results when we are doing the dot product, and we can do this by dividing each of the vectors by the square root of the sum of the squares of each of the vectors.

[12]
(-15.682584, 13.341022)
[13]
(-0.3624336, 0.39170283)
[14]
Output
[15]
torch.Size([1, 3, 224, 224])
[16]
tensor([[[[ 1.7844,  1.7844,  1.7844,  ..., -1.2083, -1.2083, -1.2083],
,          [ 1.7990,  1.7990,  1.7990,  ..., -1.2229, -1.2229, -1.2083],
,          [ 1.7990,  1.7990,  1.7990,  ..., -1.2229, -1.2229, -1.2229],
,          ...,
,          [ 0.6603,  0.6603,  0.6603,  ...,  0.4997,  0.5143,  0.5143],
,          [ 0.6603,  0.6603,  0.6603,  ...,  0.5143,  0.5289,  0.5289],
,          [ 0.6603,  0.6603,  0.6603,  ...,  0.5289,  0.5435,  0.5435]],
,
,         [[ 2.0149,  2.0149,  2.0149,  ..., -0.9267, -0.9267, -0.9267],
,          [ 2.0149,  2.0149,  2.0149,  ..., -0.9267, -0.9267, -0.9267],
,          [ 2.0149,  2.0149,  2.0149,  ..., -0.9417, -0.9417, -0.9417],
,          ...,
,          [ 0.5741,  0.5741,  0.5741,  ...,  0.5891,  0.6041,  0.6041],
,          [ 0.5741,  0.5741,  0.5741,  ...,  0.5891,  0.6041,  0.6041],
,          [ 0.5741,  0.5741,  0.5741,  ...,  0.5891,  0.6041,  0.6041]],
,
,         [[ 2.1032,  2.1032,  2.1032,  ..., -1.0252, -1.0252, -1.0252],
,          [ 2.1032,  2.1032,  2.1032,  ..., -1.0252, -1.0252, -1.0252],
,          [ 2.1032,  2.1032,  2.1032,  ..., -1.0252, -1.0252, -1.0252],
,          ...,
,          [ 1.1789,  1.1789,  1.1789,  ...,  0.8092,  0.8092,  0.8092],
,          [ 1.1789,  1.1789,  1.1789,  ...,  0.8234,  0.8234,  0.8234],
,          [ 1.1789,  1.1789,  1.1789,  ...,  0.8377,  0.8377,  0.8377]]]])
[17]
torch.Size([1, 768])
[18]
           label                                             vector  \
0            bus  [0.022191629, 0.15996626, 0.13757694, 0.058719...   
1          train  [-0.025220517, 0.11286947, 0.12789312, 0.01722...   
2           road  [-0.052865148, 0.13100702, 0.16935337, -0.0592...   
3  aquarium_fish  [-0.007917204, 0.15597954, -0.0052733854, 0.07...   
4   pickup_truck  [0.08430549, 0.090862826, 0.07901725, -0.23153...   
5       mountain  [-0.026367433, 0.12044583, 0.050345775, 0.0273...   
6      telephone  [0.17053358, 0.18755479, 0.10471857, 0.0254959...   
7        bicycle  [0.033512242, -0.05071287, 0.088322446, 0.1234...   
8          plain  [0.0394026, 0.13996188, 0.11271955, -0.0268946...   
9          whale  [0.10277067, 0.050650656, 0.010288065, -0.0908...   

    _distance  
0  439.172211  
1  446.832275  
2  449.517883  
3  449.519348  
4  449.607666  
5  449.909271  
6  450.536194  
7  450.587311  
8  450.887573  
9  450.944214
[ ]