Gemini2 0 Multi Modality With Mongodb Atlas Vector Store
agentsartificial-intelligencellmsmongodb-genai-showcasenotebooksgenerative-airag
alph-notebooks/mongodb-genai-showcase / Gemini2_0_multi_modality_with_mongodb_atlas_vector_store.ipynb
Export
Gemini 2.0 - Multimodal live API and MongoDB Atlas Vector store as tools
Inspired and built on top of the following Google example notebook.
This notebook provides examples of how to use tools with the multimodal live API with Gemini 2.0 and MongoDB Atlas with langchain integration as tools.
The tutorial build an agentic multimodal agent in websocket realtime API to fetch and store MongoDB context documents. It uses Function Calling tools. The earlier Gemini models supported versions of these tools. The biggest change with Gemini 2 (in the Live API) is that, basically, all the tools are handled by Code Execution. With that change, you can use multiple tools in a single API call.
This tutorial assumes you are familiar with the Live API, as described in the this tutorial.
Setup
Install SDK
The new Google Gen AI SDK provides programmatic access to Gemini 2.0 (and previous models) using both the Google AI for Developers and Vertex AI APIs. With a few exceptions, code that runs on one platform will run on both. This means that you can prototype an application using the Developer API and then migrate the application to Vertex AI without rewriting your code.
More details about this new SDK on the documentation or in the Getting started notebook.
[1]
Setup your API key
To run the following cell, your API key must be stored it in a Colab Secret named GOOGLE_API_KEY. If you don't already have an API key, or you're not sure how to create a Colab Secret, see Authentication for an example.
[4]
Input your Google API Key··········
Initialize SDK client
The client will pickup your API key from the environment variable.
To use the live API you need to set the client version to v1alpha.
[5]
Select a model
Multimodal Live API are a new capability introduced with the Gemini 2.0 model. It won't work with previous generation models.
[6]
Imports
[7]
Utilities
You're going to use the Live API's audio output, the easiest way hear it in Colab is to write the PCM data out as a WAV file:
[8]
Use a logger so it's easier to switch on/off debugging messages.
[9]
Get started
Most of the Live API setup will be similar to the starter tutorial. Since this tutorial doesn't focus on the realtime interactivity of the API, the code has been simplified: This code uses the Live API, but it only sends a single text prompt, and listens for a single turn of replies.
You can set modality="AUDIO" on any of the examples to get the spoken version of the output.
[56]
Since this tutorial demonstrates several tools, you'll need more code to handle the different types of objects it returns.
For example:
- The
code_executiontool can returnexecutable_codeandcode_execution_resultparts. - The
google_searchtool may attach agrounding_metadataobject.
[11]
- Finally, with the
function_declarationstool, the API may returntool_callobjects. In our case we will have 2 MongoDB tools atlas_search_tool: Search employee records using Atlas Vector search for semantic similaritycreate_team: A tool that writes a record with a team name and a people array with assigned names as the array strings.
[36]
Try running it for a first time with no tools:
[13]
before client invoke []
.......
Atlas function setup and calls
MongoDB Vector Database and Connection Setup
MongoDB acts as both an operational and a vector database for the RAG system. MongoDB Atlas specifically provides a database solution that efficiently stores, queries and retrieves vector embeddings.
Creating a database and collection within MongoDB is made simple with MongoDB Atlas.
- First, register for a MongoDB Atlas account. For existing users, sign into MongoDB Atlas.
- Follow the instructions. Select Atlas UI as the procedure to deploy your first cluster.
[14]
Collecting pymongo Downloading pymongo-4.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (22 kB) Collecting langchain-google-genai Downloading langchain_google_genai-2.0.7-py3-none-any.whl.metadata (3.6 kB) Requirement already satisfied: langchain-core in /usr/local/lib/python3.10/dist-packages (0.3.25) Collecting langchain-mongodb Downloading langchain_mongodb-0.3.0-py3-none-any.whl.metadata (2.3 kB) Collecting langchain-community Downloading langchain_community-0.3.13-py3-none-any.whl.metadata (2.9 kB) Collecting dnspython<3.0.0,>=1.16.0 (from pymongo) Downloading dnspython-2.7.0-py3-none-any.whl.metadata (5.8 kB) Collecting filetype<2.0.0,>=1.2.0 (from langchain-google-genai) Downloading filetype-1.2.0-py2.py3-none-any.whl.metadata (6.5 kB) Requirement already satisfied: google-generativeai<0.9.0,>=0.8.0 in /usr/local/lib/python3.10/dist-packages (from langchain-google-genai) (0.8.3) Requirement already satisfied: pydantic<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain-google-genai) (2.10.3) Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain-core) (6.0.2) Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain-core) (1.33) Requirement already satisfied: langsmith<0.3,>=0.1.125 in /usr/local/lib/python3.10/dist-packages (from langchain-core) (0.2.3) Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core) (24.2) Requirement already satisfied: tenacity!=8.4.0,<10.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain-core) (9.0.0) Requirement already satisfied: typing-extensions>=4.7 in /usr/local/lib/python3.10/dist-packages (from langchain-core) (4.12.2) Requirement already satisfied: langchain<0.4,>=0.3 in /usr/local/lib/python3.10/dist-packages (from langchain-mongodb) (0.3.12) Requirement already satisfied: langchain-text-splitters<0.4,>=0.3 in /usr/local/lib/python3.10/dist-packages (from langchain-mongodb) (0.3.3) Collecting motor<4.0,>=3.5 (from langchain-mongodb) Downloading motor-3.6.0-py3-none-any.whl.metadata (21 kB) Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain-mongodb) (1.26.4) Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (2.0.36) Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (3.11.10) Collecting dataclasses-json<0.7,>=0.5.7 (from langchain-community) Downloading dataclasses_json-0.6.7-py3-none-any.whl.metadata (25 kB) Collecting httpx-sse<0.5.0,>=0.4.0 (from langchain-community) Downloading httpx_sse-0.4.0-py3-none-any.whl.metadata (9.0 kB) Collecting langchain<0.4,>=0.3 (from langchain-mongodb) Downloading langchain-0.3.13-py3-none-any.whl.metadata (7.1 kB) Collecting langchain-core Downloading langchain_core-0.3.28-py3-none-any.whl.metadata (6.3 kB) Collecting pydantic-settings<3.0.0,>=2.4.0 (from langchain-community) Downloading pydantic_settings-2.7.0-py3-none-any.whl.metadata (3.5 kB) Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (2.32.3) Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (2.4.4) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.3.2) Requirement already satisfied: async-timeout<6.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (4.0.3) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (24.3.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.5.0) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (6.1.0) Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (0.2.1) Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.18.3) Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain-community) Downloading marshmallow-3.23.2-py3-none-any.whl.metadata (7.1 kB) Collecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain-community) Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) Requirement already satisfied: google-ai-generativelanguage==0.6.10 in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (0.6.10) Requirement already satisfied: google-api-core in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (2.19.2) Requirement already satisfied: google-api-python-client in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (2.155.0) Requirement already satisfied: google-auth>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (2.27.0) Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (4.25.5) Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (4.67.1) Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.3 in /usr/local/lib/python3.10/dist-packages (from google-ai-generativelanguage==0.6.10->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (1.25.0) Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core) (3.0.0) Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.3,>=0.1.125->langchain-core) (0.28.1) Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.3,>=0.1.125->langchain-core) (3.10.12) Requirement already satisfied: requests-toolbelt<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.3,>=0.1.125->langchain-core) (1.0.0) Collecting pymongo Downloading pymongo-4.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (22 kB) Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=2->langchain-google-genai) (0.7.0) Requirement already satisfied: pydantic-core==2.27.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=2->langchain-google-genai) (2.27.1) Collecting python-dotenv>=0.21.0 (from pydantic-settings<3.0.0,>=2.4.0->langchain-community) Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (3.4.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (2.2.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (2024.12.14) Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain-community) (3.1.1) Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (1.66.0) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth>=2.15.0->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (5.5.0) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth>=2.15.0->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (0.4.1) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth>=2.15.0->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (4.9) Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->langsmith<0.3,>=0.1.125->langchain-core) (3.7.1) Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->langsmith<0.3,>=0.1.125->langchain-core) (1.0.7) Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->langsmith<0.3,>=0.1.125->langchain-core) (0.14.0) Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain-community) Downloading mypy_extensions-1.0.0-py3-none-any.whl.metadata (1.1 kB) Requirement already satisfied: httplib2<1.dev0,>=0.19.0 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (0.22.0) Requirement already satisfied: google-auth-httplib2<1.0.0,>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (0.2.0) Requirement already satisfied: uritemplate<5,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (4.1.1) Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.1->google-ai-generativelanguage==0.6.10->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (1.68.1) Requirement already satisfied: grpcio-status<2.0.dev0,>=1.33.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.1->google-ai-generativelanguage==0.6.10->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (1.62.3) Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /usr/local/lib/python3.10/dist-packages (from httplib2<1.dev0,>=0.19.0->google-api-python-client->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (3.2.0) Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth>=2.15.0->google-generativeai<0.9.0,>=0.8.0->langchain-google-genai) (0.6.1) Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio->httpx<1,>=0.23.0->langsmith<0.3,>=0.1.125->langchain-core) (1.3.1) Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx<1,>=0.23.0->langsmith<0.3,>=0.1.125->langchain-core) (1.2.2) Downloading langchain_google_genai-2.0.7-py3-none-any.whl (41 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.3/41.3 kB 3.7 MB/s eta 0:00:00 Downloading langchain_mongodb-0.3.0-py3-none-any.whl (30 kB) Downloading langchain_community-0.3.13-py3-none-any.whl (2.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 27.2 MB/s eta 0:00:00 Downloading langchain_core-0.3.28-py3-none-any.whl (411 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 411.6/411.6 kB 31.8 MB/s eta 0:00:00 Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) Downloading dnspython-2.7.0-py3-none-any.whl (313 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 313.6/313.6 kB 19.6 MB/s eta 0:00:00 Downloading filetype-1.2.0-py2.py3-none-any.whl (19 kB) Downloading httpx_sse-0.4.0-py3-none-any.whl (7.8 kB) Downloading langchain-0.3.13-py3-none-any.whl (1.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 42.3 MB/s eta 0:00:00 Downloading motor-3.6.0-py3-none-any.whl (74 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.8/74.8 kB 7.8 MB/s eta 0:00:00 Downloading pymongo-4.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 60.2 MB/s eta 0:00:00 Downloading pydantic_settings-2.7.0-py3-none-any.whl (29 kB) Downloading marshmallow-3.23.2-py3-none-any.whl (49 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.3/49.3 kB 4.9 MB/s eta 0:00:00 Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB) Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB) Installing collected packages: filetype, python-dotenv, mypy-extensions, marshmallow, httpx-sse, dnspython, typing-inspect, pymongo, pydantic-settings, motor, dataclasses-json, langchain-core, langchain, langchain-mongodb, langchain-google-genai, langchain-community Attempting uninstall: langchain-core Found existing installation: langchain-core 0.3.25 Uninstalling langchain-core-0.3.25: Successfully uninstalled langchain-core-0.3.25 Attempting uninstall: langchain Found existing installation: langchain 0.3.12 Uninstalling langchain-0.3.12: Successfully uninstalled langchain-0.3.12 Successfully installed dataclasses-json-0.6.7 dnspython-2.7.0 filetype-1.2.0 httpx-sse-0.4.0 langchain-0.3.13 langchain-community-0.3.13 langchain-core-0.3.28 langchain-google-genai-2.0.7 langchain-mongodb-0.3.0 marshmallow-3.23.2 motor-3.6.0 mypy-extensions-1.0.0 pydantic-settings-2.7.0 pymongo-4.9.2 python-dotenv-1.0.1 typing-inspect-0.9.0
Prepare MongoDB Atlas vector store
Run the following code to create the Atlas Vector Search index and insert some vectorised employee records for our database.
[15]
Input your MongoDB Atlas URI:·········· New search index named vector_index is building. Polling to check if the index is ready. This may take up to a minute. vector_index is ready for querying.
[ ]
[16]
InsertManyResult(['54634', '54633', '54636', '54635', '54637', '54638'], acknowledged=True)
MongoDB Atlas Vector Search with Gemini 2.0
A vector similarity search implementation that leverages MongoDB Atlas Vector Search and Google's Gemini 2.0 embeddings to perform semantic document searches, returning the k-most similar documents based on query embedding comparison.
[29]
Additionally, including a function to create new teams with specified members as a document inside the Atlas database.
[47]
Lets create the tool defenitions
[61]
We will first search for "females" similarity search in our Employee database using the "AUDIO" modality response to recieve a voice based response.
[64]
before client invoke [{'function_declarations': [{'name': 'create_team', 'description': 'Creates a new team in the teams collection.', 'parameters': {'type': 'object', 'properties': {'team_data': {'type': 'object', 'description': 'A dictionary containing the team details.', 'properties': {'name': {'type': 'string', 'description': 'team name'}, 'people': {'type': 'array', 'description': 'A list of people in the team.', 'items': {'type': 'string', 'description': 'A person in the team.'}}}, 'required': ['name', 'people']}}, 'required': ['team_data']}}, {'name': 'atlas_search_tool', 'description': ' Perform a vector similarity search for employees using MongoDB Atlas Vector', 'parameters': {'type': 'object', 'properties': {'query': {'type': 'string', 'description': 'The search query.'}, 'k': {'type': 'integer', 'description': 'The number of results to return.'}}, 'required': ['query']}}]}]
Before tool call function_calls=[FunctionCall(id='function-call-7239458625166350317', args={'query': 'Human Resources'}, name='atlas_search_tool')]
>>> function_responses=[FunctionResponse(id='function-call-7239458625166350317', name='atlas_search_tool', response={'result': [(Document(metadata={'_id': '54637'}, page_content='Employee number 54637, name John Johnson, department HR, location Miami, salary 110000'), 0.841124415397644), (Document(metadata={'_id': '54634'}, page_content='Employee number 54634, name Jane Doe, department Marketing, location Los Angeles, salary 120000'), 0.8330270051956177), (Document(metadata={'_id': '54636'}, page_content='Employee number 54636, name Jane Smith, department Finance, location Chicago, salary 130000'), 0.8256025910377502), (Document(metadata={'_id': '54638'}, page_content='Employee number 54638, name Jane Johnson, department Operations, location Seattle, salary 140000'), 0.8211219310760498), (Document(metadata={'_id': '54633'}, page_content='Employee number 54633, name John Doe, department Sales, location New York, salary 100000'), 0.8175163269042969)]})]
.............................. Now, lets use the TEXT modality to perform a complex task for finding and creating a team from only the marketing employees.
[65]
before client invoke [{'code_execution': {}}, {'function_declarations': [{'name': 'create_team', 'description': 'Creates a new team in the teams collection.', 'parameters': {'type': 'object', 'properties': {'team_data': {'type': 'object', 'description': 'A dictionary containing the team details.', 'properties': {'name': {'type': 'string', 'description': 'team name'}, 'people': {'type': 'array', 'description': 'A list of people in the team.', 'items': {'type': 'string', 'description': 'A person in the team.'}}}, 'required': ['name', 'people']}}, 'required': ['team_data']}}, {'name': 'atlas_search_tool', 'description': ' Perform a vector similarity search for employees using MongoDB Atlas Vector', 'parameters': {'type': 'object', 'properties': {'query': {'type': 'string', 'description': 'The search query.'}, 'k': {'type': 'integer', 'description': 'The number of results to return.'}}, 'required': ['query']}}]}]
Before tool call function_calls=[FunctionCall(id='function-call-17685634460885543621', args={'query': 'marketing'}, name='atlas_search_tool')]
>>> function_responses=[FunctionResponse(id='function-call-17685634460885543621', name='atlas_search_tool', response={'result': [(Document(metadata={'_id': '54634'}, page_content='Employee number 54634, name Jane Doe, department Marketing, location Los Angeles, salary 120000'), 0.8123770356178284), (Document(metadata={'_id': '54633'}, page_content='Employee number 54633, name John Doe, department Sales, location New York, salary 100000'), 0.7818812131881714), (Document(metadata={'_id': '54636'}, page_content='Employee number 54636, name Jane Smith, department Finance, location Chicago, salary 130000'), 0.769501805305481), (Document(metadata={'_id': '54637'}, page_content='Employee number 54637, name John Johnson, department HR, location Miami, salary 110000'), 0.7627123594284058), (Document(metadata={'_id': '54635'}, page_content='Employee number 54635, name John Smith, department Engineering, location San Francisco, salary 150000'), 0.7596621513366699)]})]
Before tool call function_calls=[FunctionCall(id='function-call-9056147716109755032', args={'team_data': {'name': 'Marketing Working group', 'people': ['Jane Doe']}}, name='create_team')]
>>> function_responses=[FunctionResponse(id='function-call-9056147716109755032', name='create_team', response={'result': 'Team created successfully with ID: 676acb7c759477c2fbaf03f5'})]
The function calling feature of the API Can handle a wide variety of functions. Support in the SDK is still under construction. So keep this simple just send a minimal function definition: Just the function's name.
Note that in the live API function calls are independent of the chat turns. The conversation can continue while a function call is being processed.