Evaluations Example: Push Notifications Summarizer Monitoring

Evals are task-oriented and iterative, they're the best way to check how your LLM integration is doing and improve it.

In the following eval, we are going to focus on the task of detecting our prompt changes for regressions.

Our use-case is:

1. We have been logging chat completion requests by setting store=True in our production chat completions requests. Note that you can also enable "on by default" logging in your admin panel (https://platform.openai.com/settings/organization/data-controls/data-retention).
2. We want to see whether our prompt changes have introduced regressions.

Evals structure

Evals have two parts, the "Eval" and the "Run". An "Eval" holds the configuration for your testing criteria and the structure of the data for your "Runs". An Eval can have many Runs, which are each evaluated using your testing criteria.

Use-case

We're testing the following integration, a push notifications summary, which takes in multiple push notifications and collapses them into a single one, this is a chat completions call.

Generate our test data

I'm going to produce simulated production chat completions requests with two different prompt versions to test how each performs. The first is a "good" prompt, the second is a "bad" prompt. These will have different metadata which we'll use later.

You can view the completions you just created at https://platform.openai.com/logs.

Make sure that the chat completions show up, as they are necessary for the next step.

Setting up your eval

An Eval holds the configuration that is shared across multiple Runs, it has two components:

1. Data source configuration data_source_config - the schema (columns) that your future Runs conform to.
   • The data_source_config uses JSON Schema to define what variables are available in the Eval.
2. Testing Criteria testing_criteria - How you'll determine if your integration is working for each row of your data source.

For this use-case, we're using stored-completions, so we'll set up that data_source_config

Important You are likely to have many different stored completions use-cases, metadata is the best way to keep track of this for evals to keep them focused and task oriented.

This data_source_config defines what variables are available throughout the eval.

The stored completions config provides two variables for you to use throughout your eval:

1. {{item.input}} - the messages sent to the completions call
2. {{sample.output_text}} - the text response from the assistant

Now, we'll use those variables to set up our test criteria.

The push_notification_grader is a model grader (llm-as-a-judge), which looks at the input {{item.input}} and the generated summary {{sample.output_text}} and labels it as "correct" or "incorrect".

Note: under the hood, this uses structured outputs so that labels are always valid.

Now we'll create our eval!, and start adding data to it

Creating runs

Now that we have our eval set-up with our test_criteria, we can start adding runs. I want to compare the performance between my two prompt versions

To do this, we just define our source as "stored_completions" with a metadata filter for each of our prompt versions.

Just for to be thorough, let's see how this prompt would do with 4o, instead of 4o-mini, with both prompt versions as the starting point.

All we have to do is reference the input messages ({{item.input}}) and set the model to 4o. Since we don't already have any stored completions for 4o, this eval run will generate new completions.

If you view that report, you'll see that we can see that prompt_version=v2 has a regression!

Congratulations, you just discovered a bug, you could revert it, or make another prompt change, etc.!