Notebooks
A
Arize AI
Llama Index Multimodal Image Reasoning

Llama Index Multimodal Image Reasoning

agentsllmsLlamaIndexarize-phoenixopenaitutorialsevalsllmopsai-monitoringaiengineeringprompt-engineeringdatasetsllm-evalai-observabilityllm-evaluationsmolagentsanthropiclangchain

Open In Colab

phoenix logo
Docs | GitHub | Community

Multimodal LLM Ops - Tracing, Evaluation, and Analysis of Multimodal Models

In this notebook, we show how to use a Mult-Modal LLM, i.e., OpenAI's gpt-4o, to ask questions about images (image reasoning) using the chat API. In addition, we use Arize's Phoenix and OpenInference AutoInstrumentor to trace the operation.

Steps:

  1. Install dependencies
  2. Setup Tracing
  3. Download Images from Tesla
  4. Setup the Multi-Modal LLM application
  5. Use the Multi-Modal LLM application

⚠️ This tutorial requires an OpenAI key to run

Install dependencies

[ ]

Setup Tracing

First, we launch the phoenix app, which will act as an OTEL collector of the generated spans.

[ ]

Next, we setup the tracing by declaring a tracer provider and a span processor with an OTLP span exporter:

[ ]

Finally, we use OpenInference's Llama-Index auto-instrumentor.

[ ]

That's it! With these 2 cells you have correctly set up the tracing of your Llama-Index application. As you use this application, spans will be exported to Phoenix for observability and analysis.

Download images from Tesla's website

[ ]
[ ]

Next, we simply plot the images so you know what we just downloaded

[ ]

Setup the Multi-Modal LLM application

First things first, we need an OpenAI (our LLM provider) API key.

[ ]

Last, we need to declare our OpenAI from the multi_nmodal_llms module and use a SimpleDirectoryReader to have access to the downloaded images

[ ]

Use the Multi-Modal LLM application

We set up multimodal chat messages and call the LLM

[ ]

We can also simulate a conversation by passing the response as a message from the "assistant", and ask further questions

[ ]

Let's try make the last question more difficult. We can ask the last question without directly passing the image documents. Sometimes the LLM will rembemer the images passed in the first message and responde correctly. However, some other times it will be unaware of them and give an incomplete or wrong answer.

[ ]

Observability

Now that we've run the application a couple of times, let's take a look at the traces in the UI:

[ ]

The UI will give you an interactive troubleshooting experience. You can sort, filter, and search for traces. You can also view the questions asked and the images in the message. For instance you can see how in the second trace, the images were passed to every user message, but in the third trace only the first message had the images attached.