Notebooks
M
Mistral AI Llamaindex Mistral Multi Modal

Llamaindex Mistral Multi Modal

mistral-cookbookLlamaIndexthird_party
alph-notebooks/mistral-cookbook / llamaindex_mistral_multi_modal.ipynb

Open In Colab

Multi-Modal LLM using Mistral Pixtral-12B model for image reasoning

In this notebook, we show how to use MistralAI MultiModal LLM class/abstraction for image understanding/reasoning.

We demonstrate following functions that are supported for MistralAI Pixtral Multimodal LLM:

  • complete (both sync and async): for a single prompt and list of images
  • stream complete (both sync and async): for steaming output of complete
[ ]
[ ]

Initialize MistralAIMultiModal

[ ]

Load Images from URLs

[ ]

First Image

[ ]
https://tripfixers.com/wp-content/uploads/2019/11/eiffel-tower-with-snow.jpeg

<matplotlib.image.AxesImage at 0x13fab3500>
Output

Second Image

[ ]
https://cdn.statcdn.com/Infographic/images/normal/30322.jpeg

<matplotlib.image.AxesImage at 0x16a2458b0>
Output

Complete a prompt with a bunch of images

[ ]
[ ]

Steam Complete a prompt with a bunch of images

[ ]
[ ]
The image consists of two main parts: a photograph and an infographic. 

1. **Photograph**:
   - The photograph depicts the Eiffel Tower in Paris, France, covered in snow. The scene is serene with snow-covered trees and a pathway leading towards the Eiffel Tower. A traditional street lamp is visible in the foreground, adding to the picturesque winter setting.

2. **Infographic**:
   - The infographic is titled "France's Social Divide" and compares socio-economic indicators between disadvantaged areas and the whole of France.
   - **Indicators and Data**:
     - **% who are part of working-class**: 33.5% in disadvantaged areas vs. 14.5% in the whole of France.
     - **Unemployment rate**: 18.1% in disadvantaged areas vs. 7.3% in the whole of France.
     - **% of 16-25 y/o not in school & unemployed**: 25.2% in disadvantaged areas vs. 12.9% in the whole of France.
     - **Median monthly income**: €1,168 in disadvantaged areas vs. €1,822 in the whole of France.
     - **Poverty rate**: 43.3% in disadvantaged areas vs. 15.5% in the whole of France.
     - **Households living in overcrow

Async Complete

[ ]
[ ]

Async Steam Complete

[ ]
[ ]
The image consists of two distinct parts. The first part is a photograph of the Eiffel Tower in Paris, France, covered in snow. The tower stands tall in the background, surrounded by snow-covered trees and a pathway leading towards it. The scene is serene and picturesque, capturing the beauty of winter in Paris.

The second part is an infographic titled "France's Social Divide." This section compares socio-economic indicators between disadvantaged areas and the whole of France. The indicators include the percentage of people who are part of the working class, unemployment rates, the percentage of 16-25-year-olds not in school and unemployed, median monthly income, poverty rates, and the percentage of households living in overcrowded housing. The data shows significant disparities between disadvantaged areas and the rest of France. For example, disadvantaged areas have a higher percentage of people in the working class, higher unemployment rates, and a higher percentage of young people not in school or unemployed. Median monthly income is lower, poverty rates are higher, and a greater percentage of households live in overcrowded housing in these areas.

Overall, the image juxtaposes the iconic beauty of Paris with a detailed analysis of social and economic inequalities within the country.

Complete with Two images

[ ]

Lets Inspect the images.

First Image

[ ]
https://tripfixers.com/wp-content/uploads/2019/11/eiffel-tower-with-snow.jpeg

<matplotlib.image.AxesImage at 0x16a653bc0>
Output

Second Image

[ ]
https://assets.visitorscoverage.com/production/wp-content/uploads/2024/04/AdobeStock_626542468-min-1024x683.jpeg

<matplotlib.image.AxesImage at 0x16a6a90d0>
Output
[ ]
[ ]

Load Images from local files

[ ]
--2024-09-24 23:56:24--  https://www.boredpanda.com/blog/wp-content/uploads/2022/11/interesting-receipts-102-6364c8d181c6a__700.jpg
Resolving www.boredpanda.com (www.boredpanda.com)... 52.222.144.111, 52.222.144.94, 52.222.144.47, ...
Connecting to www.boredpanda.com (www.boredpanda.com)|52.222.144.111|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 112631 (110K) [image/jpeg]
Saving to: ‘receipt.jpg’

receipt.jpg         100%[===================>] 109.99K   411KB/s    in 0.3s    

2024-09-24 23:56:25 (411 KB/s) - ‘receipt.jpg’ saved [112631/112631]
[ ]
<matplotlib.image.AxesImage at 0x177478f80>
Output
[ ]
[ ]