Notebooks
O
OpenAI Image Understanding With Rag

Image Understanding With Rag

chatgptopenaigpt-4examplesopenai-apiopenai-cookbookmultimodal
alph-notebooks/openai-cookbook / image_understanding_with_rag.ipynb

Image Understanding with RAG using OpenAI's Vision & Responses APIs

Welcome! This notebook demonstrates how to build a Retrieval-Augmented Generation (RAG) system using OpenAI’s Vision and Responses APIs. It focuses on multimodal data, combining image and text inputs to analyze customer experiences. The system leverages GPT-5 and integrates image understanding with file search to provide context-aware responses.

Multimodal datasets are increasingly common, particularly in domains like healthcare, where records often contain both visual data (e.g. radiology scans) and accompanying text (e.g. clinical notes). Real-world datasets also tend to be noisy, with incomplete or missing information, making it critical to analyze multiple modalities in tandem.

This guide focuses on a customer service use case: evaluating customer feedback that may include photos, and written reviews. You’ll learn how to synthetically generate both image and text inputs, use file search for context retrieval, and apply the Evals API to assess how incorporating image understanding impacts overall performance.

Overview

Table of Contents

  1. Setup & Dependencies
  2. Example Generations
  3. Data Processing
    • Load synthetic datasets
    • Merge data
  4. Populating Vector Store
    • Upload data for file search
    • Set up attribute filters
  5. Retrieval and Filtering
    • Test retrieval performance
    • Apply attribute-based filters
  6. Evaluation and Analysis
    • Compare predictions to ground truth
    • Analyze performance metrics

Setup and Dependencies

[ ]
[ ]

Example Generations

Generating high-quality training and evaluation data for machine learning tasks can be costly and time-consuming. Synthetic data offers a practical and scalable alternative. In this notebook, the OpenAI Image API is used to generate synthetic images, while the Responses API is employed to create synthetic text, enabling efficient prototyping and experimentation across multimodal tasks.

[ ]
[4]
Order arrived 10 minutes early, food was hot and packaged securely. Tacos were fresh, well-seasoned, and the salsa tasted homemade. Driver was friendly, followed instructions, and left it at the door. Will definitely order again.

Data Processing

In this example, we’ll work with a pre-generated synthetic dataset of customer feedback that includes short text snippets, images from customer reviews, and occasionally combined multimodal entries. You can also generate your own synthetic dataset using the examples provided above to tailor the data to your specific use case.

[ ]
[6]
[ ]
[8]

Populating Vector Store

This example uses OpenAI's built-in vector store and file search capabilities to build a RAG system that can analyse customer experiences from their feedback, which can be both visual and text-based. We create two vector stores for comparisons, one with image understanding and one without.

[ ]
[10]
[ ]

Retrieval and Filtering

We can analyse our dataset with natural language queries with the help of File Search. For the text-only dataset, we see that information is missing that could inform our analysis.

The only positive review for spaghetti in July has visual feedback and we can see the RAG system with only text based context available is uncertain about positive details. However with image context provided the second RAG system is able to provide a more accurate response.

[12]
🔍 Query: Where there any comments about the 'spaghetti'?

📝 Response:
----------------------------------------
I couldn’t find any comments that explicitly mention “spaghetti.” The closest related note says “Pasta was overcooked” in context_9_july.txt . If you have a specific date or file in mind, I can check that directly.
[13]
🔍 Query: Where there any comments about the 'spaghetti'?

📝 Response:
----------------------------------------
Yes. There’s a positive note describing “a neatly plated spaghetti in tomato sauce with parsley, served alongside arugula, garlic bread, and grated cheese.”

We can confirm if this is correct by checking the retrieved images.

[ ]

Likewise we can test this for negative reviews in June concerning any burnt pizza.

[15]
🔍 Query: Were there any negative reviews for pizza, and if so, was the pizza burnt?

📝 Response:
----------------------------------------
Yes. One review explicitly describes a “burnt pepperoni pizza with charred crust and grease stains in the box” and is marked as negative sentiment .

We can confirm if this is correct by checking the retrieved images.

[ ]

Evaluation and Analysis

As our dataset likely evolves over time and we want to evaluate new models, we can use the OpenAI Evaluation API to evaluate the performance of our system for sentiment analysis. In this simple example, using the string_check criteria we checked if the output was one of the three possible values: positive, negative, or unclear.

[17]
[ ]
[ ]
[24]

We can retrieve the results of these evaluation runs and perform some local analysis. In this case, we will compare the performance of the text-only and text+image runs and evaluate how increasing the number of total tokens (through the addition of image context) affects the accuracy of the model. We can also do some basic error analysis by analysing the model input of the failed examples.

[25]
Output
[27]

Finally, let's clean up some of the resources we created.

[ ]