[PaliGemma 2]Using With Transformersjs
Copyright 2024 Google LLC.
PaliGemma2 - Run with Transformers.js
Author: Sitam Meur
- GitHub: github.com/sitamgithub-MSIT
- X: @sitammeur
Description: This notebook demonstrates how you can run inference on PaliGemma2 model using Node.js and Transformers.js. Transformers.js lets you run Hugging Face's transformer models directly in browser, offering a JavaScript API similar to Python's. It supports NLP, computer vision, audio, and multimodal tasks using ONNX Runtime and allows easy conversion of PyTorch, TensorFlow, and JAX models.
Setup
Select the Colab runtime
To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the PaliGemma 2 model. In this case, you can use CPU runtime:
- In the upper-right of the Colab window, select ▾ (Additional connection options).
- Select Change runtime type.
- Under Hardware accelerator, select CPU.
Installation
Let's get started with installing the dependencies.
Create Node.js project
Create a new Node.js project and install the required transformers package via NPM.
Transformers.js Inference
Now, let's run inference on the PaliGemma2 model using Transformers.js. First, load the model and processor and then prepare inputs (Text query + Image) to run inference and get the output as desired image caption. For reference, you can check the model's page on the Hugging Face model hub under ONNX models section here.
It's an image of a cat sitting on a bag, now let's see what the model predicts.
Run Application
Conclusion
Congratulations! You have successfully run inference on PaliGemma2 model using Transformers.js via Node.js environment. You can now integrate this into your projects.
Run in Google Colab