Multimodal Models from NVIDIA AI Catalog and LangChain Agent

Prerequisites

To run this notebook, you need to follow the steps from here and generate an API key from NVIDIA API Catalog.

This notebook covers the following custom plug-in components:

• LLM using meta/llama-3.1-405b-instruct

• A NVIDIA AI Catalog Deplot as one of the tool

• A NVIDIA AI Catalog Fuyu as one of the tool

• Gradio as the simply User Interface where we will upload a few images

At the end of the day, as below illustrated, we would like to have a UI which allow user to upload image of their choice and have the agent choose tools to do visual reasoning.

Note: As one can see, since we are using NVIDIA AI Catalog as an API, there is no further requirement in the prerequisites about GPUs as compute hardware

Step 1 - Export the NVIDIA_API_KEY

You can supply the NVIDIA_API_KEY directly in this notebook when you run the cell below

Step 2 - Wrap the Fuyu API call into a function and verify by supplying an image to get a respond

Fetch a test image of a pair of white sneakers and verify the function works

Step 3 - Wrap the Deplot API call into a function and verify by supplying an image to get a respond

Step 4 - Construct the agent via LCEL parallel chain

Let's review the below conceptual flow on how the lcel_agent_chain is constructed :

• We will use meta/llama-3.1-405b-instruct model as main LLM for the agent
• We will use with_structured_output to format user input query and form format_chain
• We will use bind_tools to bind ImageCaptionTool and TabularPlotTool as tools to our llm and form tool_chain
• Write an output parser that combine all the 3 branches and construct out agent via LCEL lcel_agent_chain

Initiate meta/llama-3.1-405b-instruct as the main LLM

Make format_chain via with_structured_output for formatting user input query

Make tool_chain via bind_tools to bind ImageCaptionTool and TabularPlotTool as tools to our llm

Conbined the 2 chains to form an **lcel_agent_chain **

Step 5 - Wrap the lcel_agent_chain into a python function to prepare for Gradio UI integration

Step 6 - A simple gradio UI so we can interactively upload arbitrary image