Evaluations Example: Push Notifications Bulk Experimentation

Evals are task oriented and iterative, they're the best way to check how your LLM integration is doing and improve it.

In the following eval, we are going to focus on the task of testing many variants of models and prompts.

Our use-case is:

1. I want to get the best possible performance out of my push notifications summarizer

Evals structure

Evals have two parts, the "Eval" and the "Run". An "Eval" holds the configuration for your testing criteria and the structure of the data for your "Runs". An Eval has_many runs, that are evaluated by your testing criteria.

Use-case

We're testing the following integration, a push notifications summarizer, which takes in multiple push notifications and collapses them into a single message.

Setting up your eval

An Eval holds the configuration that is shared across multiple Runs, it has two components:

1. Data source configuration data_source_config - the schema (columns) that your future Runs conform to.
   • The data_source_config uses JSON Schema to define what variables are available in the Eval.
2. Testing Criteria testing_criteria - How you'll determine if your integration is working for each row of your data source.

For this use-case, we want to test if the push notification summary completion is good, so we'll set-up our eval with this in mind.

This data_source_config defines what variables are available throughout the eval.

This item schema:

{
  "properties": {
    "notifications": {
      "title": "Notifications",
      "type": "string"
    }
  },
  "required": ["notifications"],
  "title": "PushNotifications",
  "type": "object"
}

Means that we'll have the variable {{item.notifications}} available in our eval.

"include_sample_schema": True Mean's that we'll have the variable {{sample.output_text}} available in our eval.

Now, we'll use those variables to set up our test criteria.

The push_notification_grader is a model grader (llm-as-a-judge) which looks at the input {{item.notifications}} and the generated summary {{sample.output_text}} and labels it as "correct" or "incorrect" We then instruct via the "passing_labels" what constitutes a passing answer.

Note: under the hood, this uses structured outputs so that labels are always valid.

Now we'll create our eval, and start adding data to it!

Creating runs

Now that we have our eval set-up with our testing_criteria, we can start to add a bunch of runs! We'll start with some push notification data.

Now we're going to set up a bunch of prompts to test.

We want to test a basic prompt, with a couple of variations:

1. In one variation, we'll just have the basic prompt
2. In the next one, we'll include some positive examples of what we want the summaries to look like
3. In the final one, we'll include both positive and negative examples.

We'll also include a list of models to use.

Now we can just loop through all prompts and all models to test a bunch of configurations at once!

We'll use the 'completion' run data source with template variables for our push notification list.

OpenAI will handle making the completions calls for you and populating "sample.output_text"

Congratulations, you just tested 9 different prompt and model variations across your dataset!