Iterate and Evaluate LLM applications

AI application building is an experimental process where you likely don't know how a given system will perform on your task. To iterate on an application, we need a way to evaluate if it's improving. To do so, a common practice is to test it against the same dataset when there is a change.

This tutorial will show you how to:

• track input prompts and pipeline settings with wandb.config
• track final evaluation metrics e.g. F1 score or scores from LLM judges, with wandb.log
• track individual model predictions and metadata in W&B Tables

We'll track F1 score on extracting named entities from an example news headlines dataset from explosion/prodigy-recipes from the https://prodi.gy/ team.

Setup

Download Data

Installation

Create a W&B account and log in

Format data

Here we just remove data we're not using and format the examples for our task.

Set up LLM boilerplate

We'll call openai (you'll need to add an OpenAI API key) with a given prompt to extract the entities and replace <text> with our input. We'll also grab useful metadata from the openai response for logging.

Calculate Metric

Here, we make an evaluation metric for our task. Note: It's not shown here, but you could also use an LLM to evaluate your task if it's not as straight forward to evaluate as this task.

Perform inference in parallel

Running evaluations can be a bit slow. To speed it up, here is a bit of useful code to gather your examples in parallel. None of this is specific to W&B, but it's useful to have nonetheless.

Evaluate extracted entities, save in W&B Table for inspection later

Here, we calcualte our metric across all of our predictions and log them to a wandb.Table for later inspection.

Run our pipeline:

To start logging to W&B, you can call wandb.init and pass in the config to track the configurations you're experimenting with currently.

As you experiment, you can call wandb.log to track your work. This will log the metrics to W&B. Finally, we'll call wandb.finish to stop tracking. This will be tracked as one "Run" in W&B.

You'll be given a link to W&B to see all of your logs.

Set up experiments

Start a W&B run per experiment with wandb.init, store experiment details in config arg. Log results with wandb.log. Call wandb.finish to end experiment. Loop over all options in grid search to find best configuration.

Conclusion

You've learned how to use W&B to track evaluations of your LLM applications. You've used wandb.init to start tracking, wandb.log to log summary evaluation metrics and wandb.Table to track individual predictions & scores. We've also shared some best practices to format your code to make it easier to run evaluations in parallel and track every iteration.

Trace your LLM application

If you want to learn more and you're using complex pipelines of LLM calls, you can leverage W&B Prompts to view traces of your application and see inputs & ouputs of each LLM or function call.

Learn more about W&B Prompts in the documentation here: https://docs.wandb.ai/guides/prompts