Notebooks
M
Mistral AI Incremental Prompt Engineering And Model Comparison

Incremental Prompt Engineering And Model Comparison

mistral-cookbookthird_partyPixeltable
alph-notebooks/mistral-cookbook / incremental_prompt_engineering_and_model_comparison.ipynb

Kaggle   Colab


Pixeltable

Incremental Prompt Engineering and Model Comparison with Mistral using Pixeltable

This notebook shows how to use Pixeltable for iterative prompt engineering and model comparison with Mistral AI models. It showcases persistent storage, incremental updates, and how to benchmark different prompts and models easily.

Pixeltable is data infrastructure that provides a declarative, incremental approach for multimodal AI.

Category: Prompt Engineering & Model Comparison

1. Setup and Installation

[ ]
[2]
[3]
True
[4]
Mistral AI API Key:··········

2. Create a Pixeltable Table and Insert Examples

First, Pixeltable is persistent. Unlike in-memory Python libraries such as Pandas, Pixeltable is a database. When you reset a notebook kernel or start a new Python session, you'll have access to all the data you've stored previously in Pixeltable.

[5]
Creating a Pixeltable instance at: /root/.pixeltable
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/root/.pixeltable/pgdata
Created table `mistral_prompts`.
Inserting rows into `mistral_prompts`: 3 rows [00:00, 391.64 rows/s]
Inserted 3 rows with 0 errors.

UpdateStatus(num_rows=3, num_computed_values=9, num_excs=0, updated_cols=[], cols_with_excs=[])

3. Run Mistral Inference Functions

We create computed columns to instruct Pixeltable to run the Mistral chat_completions function and store the output. Because computed columns are a permanent part of the table, they will be automatically updated any time new data is added to the table. For more information, see our tutorial.

In this particular example we are running the open_mistral_nemo and mistral_medium models and make the output available in their respective columns.

[6]
Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:06<00:00,  2.28s/ cells]
Added 3 column values with 0 errors.
Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:09<00:00,  3.31s/ cells]
Added 3 column values with 0 errors.

The respective response columns have the JSON column type and we can now use JSON path expressions to extract the relevant pieces of data and make them available as additional computed columns.

[7]
Computing cells: 100%|███████████████████████████████████████████| 3/3 [00:00<00:00, 120.75 cells/s]
Added 3 column values with 0 errors.
Computing cells: 100%|███████████████████████████████████████████| 3/3 [00:00<00:00, 121.09 cells/s]
Added 3 column values with 0 errors.
[8]

We can see how data is computed across the different columns in our table.

[9]

4. Leveraging User-Defined Functions (UDFs) for Further Analysis

UDFs allow you to extend Pixeltable with custom Python code, enabling you to integrate any computation or analysis into your workflow. See our tutorial regarding UDFs to learn more.

We define three UDFs to compute two metrics (sentiment and readability scores) that give us insights into the quality of the LLM outputs.

[10]

For each model we want to compare we are adding the metrics as new computed columns, using the UDFs we created.

[11]
Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 23.99 cells/s]
Added 3 column values with 0 errors.
Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 22.87 cells/s]
Added 3 column values with 0 errors.
Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 68.75 cells/s]
Added 3 column values with 0 errors.
Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 93.24 cells/s]
Added 3 column values with 0 errors.
Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 46.49 cells/s]
Added 3 column values with 0 errors.
Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 61.91 cells/s]
Added 3 column values with 0 errors.

Once a UDF is defined and used in a computed column, Pixeltable automatically applies it to all relevant rows.

You don't need to write loops or worry about applying the function to each row manually.

[12]

5. Experiment with Different Prompts

We are inserting an additional two rows, and Pixeltable will automatically populate the computed columns.

[13]
Computing cells: 100%|██████████████████████████████████████████| 30/30 [00:05<00:00,  5.17 cells/s]
Inserting rows into `mistral_prompts`: 2 rows [00:00, 95.14 rows/s]
Computing cells: 100%|██████████████████████████████████████████| 30/30 [00:05<00:00,  5.12 cells/s]
Inserted 2 rows with 0 errors.

UpdateStatus(num_rows=2, num_computed_values=30, num_excs=0, updated_cols=[], cols_with_excs=[])

Often you want to select only certain rows and/or certain columns in a table. You can do this with where().

You can learn more about the available table and data operations here.

[14]

Pixeltable's schema provides a holistic view of data ingestion, inference API calls, and metric computation, reflecting your entire workflow.

[15]