[PaliGemma 1]Zero Shot Object Detection In Images
Copyright 2024 Google LLC.
Zero-shot Object Detection in images
This notebook guides you to perform zero-shot object detection on images using PaliGemma and draw the inferences using OpenCV and PIL.
Get access to PaliGemma
Before using PaliGemma for the first time, you must request access to the model through Hugging Face by completing the following steps:
- Log in to Hugging Face, or create a new Hugging Face account if you don't already have one.
- Go to the PaliGemma model card to get access to the model.
- Complete the consent form and accept the terms and conditions.
To generate a Hugging Face token, open your Settings page in Hugging Face, choose Access Tokens option in the left pane and click New token. In the next window that appears, give a name to your token and choose the type as Write to get the write access.
Then, in Colab, select Secrets (🔑) in the left pane and add your Hugging Face token. Store your Hugging Face token under the name HF_TOKEN.
Select the runtime
To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the PaliGemma model. In this case, you can use a T4 GPU:
- In the upper-right of the Colab window, click the ▾ (Additional connection options) dropdown menu.
- Select Change runtime type.
- Under Hardware accelerator, select T4 GPU.
Step 1: Install libraries
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 119.8/119.8 MB 2.1 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 309.4/309.4 kB 38.3 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 251.6/251.6 kB 35.2 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.3/21.3 MB 22.7 MB/s eta 0:00:00
Step 2: Set environment variables for Hugging Face token
Step 3: Load pre-trained PaliGemma base model
config.json: 0%| | 0.00/1.03k [00:00<?, ?B/s]
model.safetensors.index.json: 0%| | 0.00/62.6k [00:00<?, ?B/s]
Downloading shards: 0%| | 0/3 [00:00<?, ?it/s]
model-00001-of-00003.safetensors: 0%| | 0.00/4.95G [00:00<?, ?B/s]
model-00002-of-00003.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00003-of-00003.safetensors: 0%| | 0.00/1.74G [00:00<?, ?B/s]
`config.hidden_act` is ignored, you should use `config.hidden_activation` instead. Gemma's activation function will be set to `gelu_pytorch_tanh`. Please, use `config.hidden_activation` if you want to override this behaviour. See https://github.com/huggingface/transformers/pull/29402 for more details.
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
generation_config.json: 0%| | 0.00/137 [00:00<?, ?B/s]
preprocessor_config.json: 0%| | 0.00/699 [00:00<?, ?B/s]
tokenizer_config.json: 0%| | 0.00/40.0k [00:00<?, ?B/s]
tokenizer.model: 0%| | 0.00/4.26M [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/17.5M [00:00<?, ?B/s]
added_tokens.json: 0%| | 0.00/24.0 [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/607 [00:00<?, ?B/s]
Step 4: Function to draw inference on images
Step 5: Configure the input image and text prompt
Step 6: Pass the input image and text prompt to PaliGemma
Step 7: Parse the PaliGemma output
This section parses the response from PaliGemma model and draws the bounding box on image.
Run in Google Colab