Using GPT4 Vision With Function Calling
How to use GPT-4o Vision with Function Calling
The GPT-4o, available as gpt-4o-2024-11-20 as of Novemeber 2024, now enables function calling with vision capabilities, better reasoning and a knowledge cutoff date of Oct 2023. Using images with function calling will unlock multimodal use cases and the ability to use reasoning, allowing you to go beyond OCR and image descriptions.
We will go through two examples to demonstrate the use of function calling with GPT-4o with Vision:
- Simulating a customer service assistant for delivery exception support
- Analyzing an organizational chart to extract employee information
Installation and Setup
Matplotlib is building the font cache; this may take a moment.
1. Simulating a customer service assistant for delivery exception support
We will simulate a customer service assistant for a delivery service that is equipped to analyze images of packages. The assistant will perform the following actions based on the image analysis:
- If a package appears damaged in the image, automatically process a refund according to policy.
- If the package looks wet, initiate a replacement.
- If the package appears normal and not damaged, escalate to an agent.
Let's look at the sample images of packages that the customer service assistant will analyze to determine the appropriate action. We will encode the images as base64 strings for processing by the model.
Encoded image: wet_package.jpg Encoded image: damaged_package.jpg Encoded image: normal_package.jpg
We have successfully encoded the sample images as base64 strings and displayed them. The customer service assistant will analyze these images to determine the appropriate action based on the package condition.
Let's now define the functions/tools for order processing, such as escalating an order to an agent, refunding an order, and replacing an order. We will create placeholder functions to simulate the processing of these actions based on the identified tools. We will be using Pydantic models to define the structure of the data for order actions.
Simulating user messages and processing the package images
We will simulate user messages containing the package images and process the images using the GPT-4o with Vision model. The model will identify the appropriate tool call based on the image analysis and the predefined actions for damaged, wet, or normal packages. We will then process the identified action based on the order ID and display the results.
Processing delivery exception support for different package images... ===================== Simulating user message 1 ===================== - Tool call: refund_order for provided img: damaged_package - Parameters: rationale='The package appears damaged as it is visibly crushed and deformed.' image_description='A package that is visibly crushed and deformed, with torn and wrinkled packaging material.' action='refund_order' message=None >> Action result: Order 12345 has been refunded successfully. ===================== Simulating user message 2 ===================== - Tool call: escalate_to_agent for provided img: normal_package - Parameters: rationale='The package appears normal and undamaged in the image.' image_description='A cardboard box placed on a wooden floor, showing no visible signs of damage or wetness.' action='escalate_to_agent' message='The package appears normal and undamaged. Please review further.' >> Action result: Order 12345 has been escalated to an agent with message: `The package appears normal and undamaged. Please review further.` ===================== Simulating user message 3 ===================== - Tool call: replace_order for provided img: wet_package - Parameters: rationale='The package appears wet, which may compromise its contents.' image_description="A cardboard box labeled 'Fragile' with visible wet spots on its surface." action='replace_order' message=None >> Action result: Order 12345 has been replaced with a new order.
2. Analyzing an organizational chart to extract employee information
For the second example, we will analyze an organizational chart image to extract employee information, such as employee names, roles, managers, and manager roles. We will use GPT-4o with Vision to process the organizational chart image and extract structured data about the employees in the organization. Indeed, function calling lets us go beyond OCR to actually deduce and translate hierarchical relationships within the chart.
We will start with a sample organizational chart in PDF format that we want to analyze and convert the first page of the PDF to a JPEG image for analysis.
The organizational chart image has been successfully extracted from the PDF file and displayed. Let's now define a function to analyze the organizational chart image using the new GPT4o with Vision. The function will extract information about the employees, their roles, and their managers from the image. We will use function/tool calling to specify the input parameters for the organizational structure, such as the employee name, role, and manager's name and role. We will use Pydantic models to define the structure of the data.
Now, we will define a function to parse the response from GPT-4o with vision and extract the employee data. We will tabulate the extracted data for easy visualization. Please note that the accuracy of the extracted data may vary based on the complexity and clarity of the input image.
The extracted data from the organizational chart has been successfully parsed and displayed in a DataFrame. This approach allows us to leverage GPT-4o with Vision capabilities to extract structured information from images, such as organizational charts and diagrams, and process the data for further analysis. By using function calling, we can extend the functionality of multimodal models to perform specific tasks or call external functions.