Tool Calling Google Api
Google Calendar Assistant with with Llama 3.2 3B Tool Calling
This notebook showcases how to go about building a digital assistant to schedule meetings with the Llama 3.2 3B model. The core concepts used to implement this are Prompt Engineering and Tool Calling. This demo shows how Llama can be used to interact with 3rd party apps like Google Contacts & Google Calendar and schedule a meeting requested by the user. Even though we are using prompt engineering to achieve this, the approach described doesn't degrade the model's ability to answer general queries. This approach can extended to perform other tasks in a similar manner without affecting the quality of other tasks
Approach
Instead of using a complex system prompt with multiple conditions & expecting Llama to perform various tasks accurately out of the box, the approach here is to treat this as a 2 step process
- Determine user intent - Task classification
- Take action for the specific task using Tool Calling
In the diagram shown below,
- system prompt 1 determines the classfication of the query
- In steps 2 & 3, we classify the task being requested.
- system prompt 2 is chosen based on the classification result
- Steps 4 & 5 implement the classified task.
- For the sake of demo, we show 2 classes: General & Meeting
Both these tasks have a specific prompt. We use the same model with different system prompts depending on the classification result. Additionally, this demo also showcases how Llama can be used to do double tool calling with 1 prompt. In the case of Meeting, Llama returns 2 function calls in Step 5
<function=google_contact>{{"name": "John Constantine"}}</function>
<function=google_calendar>{{"date": "Mar 31 ", "time": "5:30 pm", "attendees": "John Constantine"}}</function>
Actions based on tool calling output
- The google_contact function call returned by the model is used to call People API to look up the email address of the person of interest
- The email address from the previous step is used to call Calendar API along with the other information in the google_calendar toolcall output returned by the model
The end result is that a google meeting is scheduled with the person of interest at the date & time specified
Load Llama 3.2-3B-Instruct model
This demo also intends to show that tool calling can be done effectively even with the 3B model
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Define functions to access People (google_contact) and Google Calendar (google_calendar) API
Note!!!! Accessing Google APIs require you to first get credentials (This is a one time process)
Store your credentials in credentials.json Please refer to the instructions here to get the credentials.
I followed the the steps in service account to get my credentials.
Accessing Google APIs also require you to get authentication token in addition to the credentials. The functions defined below google_contact and google_calendar also include the logic to generate & refresh the authentication token.
To get the authentication token for using the People & Google Calendar API, we need a runnable browser available on the machine where you execute these functions for the FIRST TIME only. You will need to authenticate using your browser for the first time. Executing these functions will generate a token_contacts.json & token_calendar.json
For subsequent calls, you don't need a runnable browser
Install required libraries
Function to process tool calling output and call Google APIs
Functions for calling Llama for the 2 tasks: general & setup meeting
Demo with a general query
To help explain the final output, we also print the system prompts being used in both the calls to Llama
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation. Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
----------------------------------------------------- FIRST PASS ----------------------------------------------------- Classify the given prompt A message can be classified as one of the following categories: meeting, general. Examples: - meeting: "Can you schedule a meeting with Ankith on Dec 1 at 1 pm." - general: "Tell me about San Francisco" Reminder: - ONLY return the classification label and NOTHING else - DO NOT give any explanation or examples Question: Tell me about Paris Answer: general ----------------------------------------------------- SECOND PASS ----------------------------------------------------- You are a helpful assistant. Answer the query in 100 words Question: Tell me about Paris Answer: Paris, the City of Light, is the capital of France, known for its stunning architecture, art, fashion, and romance. The city is home to iconic landmarks like the Eiffel Tower, Notre-Dame Cathedral, and the Louvre Museum, which houses the Mona Lisa. The city's charming streets, cafes, and gardens offer a unique blend of history, culture, and beauty. Visitors can stroll along the Seine River, explore the Latin Quarter, and indulge in French cuisine, wine
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation. Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
----------------------------------------------------- FIRST PASS ----------------------------------------------------- Classify the given prompt A message can be classified as one of the following categories: meeting, general. Examples: - meeting: "Can you schedule a meeting with Ankith on Dec 1 at 1 pm." - general: "Tell me about San Francisco" Reminder: - ONLY return the classification label and NOTHING else - DO NOT give any explanation or examples Question: Can you describe how an internal combustion engine works Answer: general ----------------------------------------------------- SECOND PASS ----------------------------------------------------- You are a helpful assistant. Answer the query in 100 words Question: Can you describe how an internal combustion engine works Answer: An internal combustion engine is a type of heat engine that generates power by burning fuel, typically gasoline or diesel, inside a combustion chamber within the engine. The engine consists of several major components, including cylinders, pistons, crankshafts, and valves. Here's a simplified overview of the process: 1. Air and fuel are drawn into the cylinders through the intake valves. 2. The mixture is compressed by the pistons, creating a high-pressure mixture. 3. The spark plug ignites the
Demo with a set up meeting query, which uses tool calling & calls Google APIs
To help explain the final output, we also print the system prompts being used in both the calls to Llama
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation. Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
-----------------------------------------------------
FIRST PASS
-----------------------------------------------------
Classify the given prompt
A message can be classified as one of the following categories: meeting, general.
Examples:
- meeting: "Can you schedule a meeting with Ankith on Dec 1 at 1 pm."
- general: "Tell me about San Francisco"
Reminder:
- ONLY return the classification label and NOTHING else
- DO NOT give any explanation or examples
Question: Schedule a meeting with John Constantine on Mar 31 at 5:30 pm
Answer:
meeting
-----------------------------------------------------
SECOND PASS
-----------------------------------------------------
You are a helpful assistant.
Here are the tools you have access to: google_contact and google_calendar which you can use for ONLY setting up meetings:
Use the function 'google_contact' to: look up a name and get the email address
{
"name": "google_contact",
"description": "Get email address",
"parameters": {
"name": {
"param_type": "string",
"description": "name",
"required": true
}
}
}
Use the function 'google_calendar' to: schedule a meeting
{
"name": "google_calendar",
"description": "Schedule a meeting",
"parameters": {
"date": {
"param_type": "string",
"description": "date",
"required": true
},
"time"": {
"param_type": "string",
"description": "time",
"required": true
},
"attendees"": {
"param_type": "string",
"description": "name",
"required": true
}
}
}
Identify the name in the query, lookup the name using 'google_contact' to find the email address of the contact
and then use 'google_calndar' to schedule a meeeting with this person.
DO NOT reply with any other reasoning or exaplanation
ONLY reply in the following format with no prefix or suffix:
<function=function_name_being_used>{{"example_name": "example_value"}}</function>
Reminder:
- ONLY return the function call and don't reply anything else
- DO NOT give any explanation
- I REPEAT ONLY return function calls
- Function calls MUST follow the specified format, start with <function= and end with </function>
- Required parameters MUST be specified
- First return google_contact and then google_calendar
Question:
Classify the given prompt
A message can be classified as one of the following categories: meeting, general.
Examples:
- meeting: "Can you schedule a meeting with Ankith on Dec 1 at 1 pm."
- general: "Tell me about San Francisco"
Reminder:
- ONLY return the classification label and NOTHING else
- DO NOT give any explanation or examples
Question: Schedule a meeting with John Constantine on Mar 31 at 5:30 pm
Answer:
<function=google_contact>{{"name": "John Constantine"}}</function>
<function=google_calendar>{{"date": "Mar 31", "time": "5:30 pm", "attendees": "John Constantine"}}</function>
-----------------------------------------------------
Event created with John Constantine with email constantine@example.com on Mar 31 at 5:30 pm:
https://www.google.com/calendar/event?eid=MG1rMnBsbzRvaGYwdXUzc2tydnZvNGZyNWMgYW5raXRodGVzdHRvb2xAbQ
Result
In your Google Calendar, you should see an invite generated as shown below
