Copyright 2025 Google LLC.

Gemini API: Using Gemini API to tag and caption images

You will use the Gemini model's vision capabilities and the embedding model to add tags and captions to images of pieces of clothing.

These descriptions can be used alongside embeddings to allow you to search for specific pieces of clothing using natural language, or other images.

Setup

Configure your API key

To run the following cell, your API key must be stored in a Colab Secret named GOOGLE_API_KEY. If you don't already have an API key, or you're not sure how to create a Colab Secret, see Authentication for an example.

Downloading dataset

First, you need to download a dataset with images. It contains images of various clothing that you can use to test the model.

Unzip the data in clothes-dataset.zip and place them in a folder in your Colab environment.

Generating keywords

You can use the LLM to extract relevant keywords from the images.

Here is a helper function for calling Gemini API with images. Sleep is for ensuring that the quota is not exceeded. Refer to our princing page for current quotas.

First, define the list of possible keywords.

Go ahead and define a prompt that will help define keywords that describe clothing. In the following prompt, few-shot prompting is used to prime the LLM with examples of how these keywords should be generated and which are valid.

Generate keywords for each of the images.

Keyword correction and deduplication

Unfortunately, despite providing a list of possible keywords, the model, at least in theory, can return an invalid keyword. It may be a duplicate e.g. "denim" for "jeans", or be completely unrelated to any keyword from the list.

To address these issues, you can use embeddings to map the keywords to predefined ones and remove unrelated ones.

For demonstration purposes, define a function that assesses the similarity between two embedding vectors. In this case, you will use cosine similarity, but other measures such as dot product work too.

Next, define a function that allows you to replace a keyword with the most similar word in the keyword dataframe that you have previously created.

Note that the threshold is decided arbitrarily, it may require tweaking depending on use case and dataset.

Here is an example of how these keywords can be mapped to a keyword with the closest meaning.

You can now either leave words that do not fit our predefined categories or delete them. In this scenario, all words without a suitable replacement will be omitted.

Generating captions

Searching for specific clothes

Preparing out dataset

First, you need to generate caption and keywords for every image. Then, you will use embeddings, which will be used later to compare the images in the search dataset with other descriptions and images.

Also, the ast.literal_eval() helper function allows you to evaluate an object passed in and get the literal object. For instance, if you passed in a string "[1, 2, 3]", the ast.literal_eval() function would return it as a list [1, 2, 3]. For more information on this function, here is the documentation.

You will use only the first 8 images, so the rest can be used for testing.

Finding clothes using natural language

Finding similar clothes using images

Summary

You have used Gemini API's Python SDK to tag and caption images of clothing. Using embedding models, you were able to search a database of images for clothing matching our description, or similar to the provided clothing item.