Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.

This source code is licensed under the terms described in the LICENSE file in the root directory of this source tree.

Getting Started with prompt-ops

This notebook will guide you through the process of using prompt-ops to optimize your prompts for Llama models. We'll cover:

1. Introduction to prompt-ops
2. Setting up your environment
3. Creating a sample project
4. Running prompt optimization
5. Analyzing the results
6. Advanced usage and customization

1. Introduction to prompt-ops

What is prompt-ops?

prompt-ops is a Python package that automatically optimizes prompts for Llama models. It transforms prompts that work well with other LLMs into prompts that are optimized for Llama models, improving performance and reliability.

How It Works

prompt-ops takes three key inputs:

1. Your existing system prompt
2. A dataset of query-response pairs for evaluation and optimization
3. A configuration file specifying model parameters and optimization details

It then applies optimization techniques to transform your prompt into one that works better with Llama models, and provides metrics to measure the improvement.

2. Setting up your environment

Let's start by installing the Prompt ops package and setting up our environment. You can install it either from PyPI or directly from the source code.

Setting up your API key

Prompt ops requires an API key to access LLM services. You can use OpenRouter, which provides access to various models including Llama models.

Create a .env file in your project directory with your API key:

You'll need to replace your_key_here with your actual OpenRouter API key. You can get an OpenRouter API key by creating an account at OpenRouter.

Let's load the environment variables from the .env file:

3. Creating a Sample Project

Prompt ops provides a convenient way to create a sample project with all the necessary files. Let's create a sample project to get started.

This command creates a directory called my-notebook-project with a sample configuration and dataset. Let's explore the files that were created:

The sample project includes:

• .env: A file for your API key
• README.md: Documentation for the project
• config.yaml: Configuration file for prompt optimization
• data/dataset.json: Sample dataset for evaluation and optimization
• prompts/prompt.txt: Sample system prompt to optimize

Let's examine the configuration file:

The configuration file specifies:

• The system prompt to optimize
• The dataset to use for evaluation and optimization
• The model to use for optimization and evaluation
• The metric to use for evaluation
• The optimization strategy to use

Let's also look at the sample prompt and dataset:

4. Running Prompt Optimization

Now that we have our sample project set up, let's run the prompt optimization process. We'll use the migrate command, which takes a configuration file as input and outputs an optimized prompt.

The optimization process will take a few minutes to complete. It involves:

1. Loading your system prompt and dataset
2. Analyzing the prompt structure and content
3. Applying optimization techniques specific to Llama models
4. Evaluating the optimized prompt against the original prompt
5. Saving the optimized prompt to the results/ directory

Let's check the results directory to see the optimized prompt. If the optimizer successfully found a better prompt, it will be saved in the results/ directory. You may need to run the optimization process again with different parameters or a larger dataset if the prompt is the same as the original.

The optimized prompt is saved as a YAML file with a timestamp. Let's examine the contents of the optimized prompt:

5. Analyzing the Results

Let's compare the original prompt with the optimized prompt to understand the changes made during optimization.

Key Differences in the Optimized Prompt

The optimized prompt typically includes several improvements:

1. Better Structure: Llama models respond better to clear, structured instructions
2. Llama-Specific Formatting: Formatting that works better with Llama's training patterns
3. Few-Shot Examples: Examples that help the model understand the expected output format
4. Clear Output Expectations: More explicit instructions about what the output should look like

These changes can significantly improve the model's performance on your specific task.

6. Advanced Usage and Customization

Using Your Own Data

To use your own data with Prompt ops, you'll need to:

1. Prepare your dataset in JSON format
2. Create a system prompt file
3. Create a configuration file

Check out the comprehensive guide here to learn more.

Now, let's see how to create a custom configuration file:

Using Different Metrics

Prompt ops supports different metrics for evaluating prompt performance. The default is StandardJSONMetric, but you can use other metrics like FacilityMetric for specific use cases.

Here's an example of using the FacilityMetric for the facility support analyzer use case:

Using Different Models

Prompt ops supports different models through various inference providers. You can use OpenRouter, vLLM, or NVIDIA NIMs depending on your infrastructure needs.

Here's an example of using a different model through OpenRouter:

Conclusion

In this notebook, we've covered:

1. Introduction to Prompt ops and its benefits
2. Creating a sample project
3. Setting up your environment and API key
4. Running prompt optimization
5. Analyzing the results
6. Advanced usage and customization options

Prompt ops provides a powerful way to optimize your prompts for Llama models, improving performance and reliability. By following the steps in this notebook, you can start optimizing your own prompts and building more effective LLM applications.

For more information, check out the prompt-ops documentation and explore the example use cases in the repository.