Vertexai Gemini Examples
Haystack 💙 Google Gemini
by Tuana Celik: Twitter, LinkedIn, Tilde Thurium: Twitter, LinkedIn and Silvano Cerza: LinkedIn
This is a notebook showing how you can use Gemini + Vertex AI with Haystack.
To use Gemini models on the Gemini Developer API with Haystack, check out our documentation.
Gemini is Google's newest model. You can read more about its capabilities here.
Install dependencies
As a prerequisite, you need to have a Google Cloud Project set up that has access to Vertex AI and Gemini.
Useful resources:
Following that, you'll only need to authenticate yourself in this Colab.
First thing first we need to install our dependencies including Google Gen AI integration:
Let's login using Application Default Credentials (ADCs). For more info see the official documentation.
Remember to set the project_id variable to a valid project ID that you have enough authorization to use for Gemini.
We're going to use this one throughout the example!
To find your project ID you can find it in the GCP resource manager or locally by running gcloud projects list in your terminal. For more info on the gcloud CLI see the official documentation.
Use gemini-2.5-flash
Answer Questions
Now that we setup everything we can create an instance of our GoogleGenAIChatGenerator. This component supports both Gemini and Vertex AI. For this demo, we will set api="vertex", and pass our project_id as vertex_ai_project.
Let's start by asking something simple.
This component expects a list of ChatMessage as input to the run() method. You can pass text or function calls through the messages.
The most interesting thing I know, and one of the most profound mysteries in all of science, is that **about 95% of the universe is made of something we cannot see or directly detect: dark energy and dark matter.** Imagine if 95% of the world around you was completely invisible and unknown, yet it fundamentally shaped everything you *could* see. That's our current situation with the cosmos. * **Dark Matter** makes up about 27% of the universe. We know it exists because of its gravitational effects – it holds galaxies together, prevents clusters from flying apart, and influenced the large-scale structure of the early universe. But it doesn't absorb, reflect, or emit light, making it "dark." We don't know what particles it's made of. * **Dark Energy** makes up about 68% of the universe. It's an even bigger enigma. We infer its existence because it's responsible for the accelerated expansion of the universe. It's essentially pushing the cosmos apart, overcoming the attractive force of gravity. Its nature is one of the biggest unsolved problems in physics. This means that all the stars, planets, galaxies, gas, and dust – everything we can observe with telescopes – makes up only about 5% of the universe's total mass-energy content. The vast majority of reality is utterly mysterious, and understanding it is one of the greatest scientific quests of our time. It dictates the fate of the cosmos itself.
Answer Questions about Images
Let's try something a bit different! gemini-2.5-flash can also work with images, let's see if we can have it answer questions about some robots 👇
We're going to download some images for this example. 🤖
These are iconic robots from popular culture: 1. **C-3PO:** A refined protocol droid, fluent in countless languages, known for his golden appearance and nervous demeanor. 2. **T-800 Endoskeleton:** A formidable, relentless combat machine, skeletal and chilling, from a dystopian future. 3. **R2-D2:** A courageous and resourceful astromech, full of personality, who communicates in beeps and whistles.
Function Calling with gemini-2.5-flash
With gemini-2.5-flash, we can also use function calling!
So let's see how we can do that 👇
Let's see if we can build a system that can run a get_current_weather function, based on a question asked in natural language.
First we create our function definition and tool (learn more about Tools in the docs).
For demonstration purposes, we're simply creating a get_current_weather function that returns an object which will always tell us it's 'Sunny, and 21.8 degrees'... If it's Celsius, that's a good day! ☀️
[ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[TextContent(text=''), ToolCall(tool_name='get_current_weather', arguments={'unit': 'celsius', 'location': 'Berlin'}, id=None)], _name=None, _meta={'model': 'gemini-2.5-flash', 'finish_reason': 'stop', 'usage': {'prompt_tokens': 53, 'completion_tokens': 10, 'total_tokens': 126}})]
Look at that! We go a message with some interesting information now. We can use that information to call a real function locally.
Let's do exactly that and pass the result back to Gemini.
[ChatMessage(_role=<ChatRole.TOOL: 'tool'>, _content=[ToolCallResult(result="{'weather': 'sunny', 'temperature': 21.8, 'unit': 'celsius'}", origin=ToolCall(tool_name='get_current_weather', arguments={'unit': 'celsius', 'location': 'Berlin'}, id=None), error=False)], _name=None, _meta={})]
The temperature in Berlin is 21.8°C and it's sunny.
Seems like the weather is nice and sunny, remember to put on your sunglasses. 😎
Build a full Retrieval-Augmented Generation Pipeline with gemini-2.5-flash
As a final exercise, let's add the GoogleGenAIChatGenerator to a full RAG pipeline. In the example below, we are building a RAG pipeline that does question answering on the web, using gemini-2.5-flash
Let's try asking Gemini to tell us about Haystack and how to use it.
In Haystack, pipelines are structured as graphs. Specifically, Haystack 1.x pipelines were based on Directed Acyclic Graphs (DAGs). In Haystack 2.0, the "A" (acyclic) is being removed from DAG, meaning pipelines can now branch out, join, and cycle back to other components, allowing for more complex graph structures that can retry or loop.