Notebooks
G
Google Gemini
[PaliGemma 1]Zero Shot Object Detection In Images

[PaliGemma 1]Zero Shot Object Detection In Images

recurrentgemmagemmapaligemmagemma-cookbookcodegemma
Copyright 2024 Google LLC.
[ ]

This notebook is created by Nitin Tiwari.

Social links:

Zero-shot Object Detection in images

This notebook guides you to perform zero-shot object detection on images using PaliGemma and draw the inferences using OpenCV and PIL.

Get access to PaliGemma

Before using PaliGemma for the first time, you must request access to the model through Hugging Face by completing the following steps:

  1. Log in to Hugging Face, or create a new Hugging Face account if you don't already have one.
  2. Go to the PaliGemma model card to get access to the model.
  3. Complete the consent form and accept the terms and conditions.

To generate a Hugging Face token, open your Settings page in Hugging Face, choose Access Tokens option in the left pane and click New token. In the next window that appears, give a name to your token and choose the type as Write to get the write access.

Then, in Colab, select Secrets (🔑) in the left pane and add your Hugging Face token. Store your Hugging Face token under the name HF_TOKEN.

Select the runtime

To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the PaliGemma model. In this case, you can use a T4 GPU:

  1. In the upper-right of the Colab window, click the ▾ (Additional connection options) dropdown menu.
  2. Select Change runtime type.
  3. Under Hardware accelerator, select T4 GPU.

Step 1: Install libraries

[ ]
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 119.8/119.8 MB 2.1 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 309.4/309.4 kB 38.3 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 251.6/251.6 kB 35.2 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.3/21.3 MB 22.7 MB/s eta 0:00:00

Step 2: Set environment variables for Hugging Face token

[ ]

Step 3: Load pre-trained PaliGemma base model

[ ]
config.json:   0%|          | 0.00/1.03k [00:00<?, ?B/s]
model.safetensors.index.json:   0%|          | 0.00/62.6k [00:00<?, ?B/s]
Downloading shards:   0%|          | 0/3 [00:00<?, ?it/s]
model-00001-of-00003.safetensors:   0%|          | 0.00/4.95G [00:00<?, ?B/s]
model-00002-of-00003.safetensors:   0%|          | 0.00/5.00G [00:00<?, ?B/s]
model-00003-of-00003.safetensors:   0%|          | 0.00/1.74G [00:00<?, ?B/s]
`config.hidden_act` is ignored, you should use `config.hidden_activation` instead.
Gemma's activation function will be set to `gelu_pytorch_tanh`. Please, use
`config.hidden_activation` if you want to override this behaviour.
See https://github.com/huggingface/transformers/pull/29402 for more details.
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
generation_config.json:   0%|          | 0.00/137 [00:00<?, ?B/s]
preprocessor_config.json:   0%|          | 0.00/699 [00:00<?, ?B/s]
tokenizer_config.json:   0%|          | 0.00/40.0k [00:00<?, ?B/s]
tokenizer.model:   0%|          | 0.00/4.26M [00:00<?, ?B/s]
tokenizer.json:   0%|          | 0.00/17.5M [00:00<?, ?B/s]
added_tokens.json:   0%|          | 0.00/24.0 [00:00<?, ?B/s]
special_tokens_map.json:   0%|          | 0.00/607 [00:00<?, ?B/s]

Step 4: Function to draw inference on images

[ ]

Step 5: Configure the input image and text prompt

[51]
input_image
prompt
[52]
Output

Step 6: Pass the input image and text prompt to PaliGemma

[53]
[54]

Step 7: Parse the PaliGemma output

This section parses the response from PaliGemma model and draws the bounding box on image.

[55]

Step 8: Display the final output

[56]
Output