Fine Tuning Open AI Models

This notebook is based on the current guidance provided in the Fine Tuning documentation from Open AI.

Fine-tuning improves the performance of foundation models for your application by retraining it with additional data and context relevant to that specific use case or scenario. Note that prompt engineering techniques like few shot learning and retrieval augmented generation allow you to enhance the default prompt with relevant data to improve quality. However, these approaches are limited by max token window size of the targeted foundation model.

With fine-tuning, we are effectively retraining the model itself with the required data (allowing us to use many more examples than can fit in the max token window) - and deploying a custom version of the model that no longer needs to have examples provided at inference time. This not only improves the effectivenenss of our prompt design (we have more flexibility in using the token window for other things) but potentially also improves our costs (by reducing the number of tokens we need to send to the model at inference time).

Fine tuning has 4 steps:

1. Prepare the training data and upload it.
2. Run the training job to get a fine-tuned model.
3. Evaluate the fine-tuned model and iterate for quality.
4. Deploy the fine-tuned model for inference when satisfied.

Note that not all foundation models support fine-tuning - check OpenAI documentation for the latest information. You can also fine-tune a previously fine-tuned model. In this tutorial, we'll use gpt-35-turbo as our target foundation model for fine-tuning.

Step 1.1: Prepare Your Dataset

Let's build a chatbot that helps you understand the periodic table of elements by answering questions about an element with a limerick. In this simple tutorial, we'll just create a dataset to train the model with a few sample examples of responses that show the expected format of the data. In a real-world use case, you would need to create a dataset with many more examples. You may also be able to use an open dataset (for your application domain) if one exists, and reformat it for use in fine-tuning.

Since we are focusing on gpt-35-turbo and looking for a single-turn response (chat completion) we can create examples using this suggested format reflecting the OpenAI chat completion requirements. If you expect multi-turn conversational content, you would use the multi-turn example format which includes a weight parameter to signal which messages should be used (or not) in the fine-tuning process.

We will use the simpler single-turn format for our tutorial here. The data is in the jsonl format with 1 record per line, each represented as a JSON-formatted object. The snippet below shows 2 records as a sample - see training-data.jsonl for full sample set (10 examples) we'll use for our fine-training tutorial. Note: Each record must be defined in a single line (not split across lines as is typical in a formatted JSON file)

{ "messages": [{"role": "system", "content": "Elle is a factual chatbot that answers questions about elements in the periodic table with a limerick"}, {"role": "user", "content": "Tell me about Gallium"}, {"role": "assistant", "content": "Gallium, oh gallium, so light - Melts in your hand, oh what a sight - At 86 degrees - Its liquid with ease - And in semiconductors, it's out of sight"}]}
{ "messages": [{"role": "system", "content": "Elle is a factual chatbot that answers questions about elements in the periodic table with a limerick"}, {"role": "user", "content": "Tell me about Hydrogen"}, {"role": "assistant", "content": "Hydrogen, the first in the line - The lightest of all, so divine - It's in water, you see - And in stars, it's the key - The universe's most common sign"}]}

In a real-world use case you will need a much larger examples set for good results - the tradeoff will be between quality of responses and the time/costs for fine-tuning. We are using a small set so we can complete fine-tuning quickly to illustrate the process. See this OpenAI Cookbook example for a more complex fine-tuning tutorial.

Step 1.2 Upload Your Dataset

Upload the data using the Files API as described here. Note that in order to run this code, you must have done the following steps first:

• Installed the openai Python package (make sure you use a version >=0.28.0 for latest features)
• Set the OPENAI_API_KEY environment variable to your OpenAI API key To learn more, see the Setup guide provided for the course.

Now, run the code to create a file for upload from your local JSONL file.

Step 2.1: Create the Fine-tuning job with the SDK

Step 2.2: Check the Status of the job

Here are a few things you can do with the client.fine_tuning.jobs API:

• client.fine_tuning.jobs.list(limit=<n>) - List the last n fine-tuning jobs
• client.fine_tuning.jobs.retrieve(<job_id>) - Get details of a specific fine-tuning job
• client.fine_tuning.jobs.cancel(<job_id>) - Cancel a fine-tuning job
• client.fine_tuning.jobs.list_events(fine_tuning_job_id=<job_id>, limit=<b>) - List up to n events from the job
• client.fine_tuning.jobs.create(model="gpt-35-turbo", training_file="your-training-file.jsonl", ...)

The first step of the process is validating the training file to make sure data is in the right format.

Step 2.3: Track events to monitor progress

Step 2.4: View status in OpenAI Dashboard

You can also view the status by visiting the OpenAI website and exploring the Fine-tuning section of the platform. This will show you the status of the current job, and also let you track the history of prior job execution runs. In this screenshot, you can see that the prior execution failed, and the second run succeeded. For context, this happened when the first run used a JSON file with incorrectly formatted records - once fixed, the second run completed successfully and made the model available for use.

You can also view the status messages and metrics by scrolling down further in the visual dashboard as shown:

Messages	Metrics

Step 3.1: Retrieve ID & Test Fine-Tuned Model in Code

Step 3.2: Load & Test Fine-Tuned Model in Playground

You can now test the fine-tuned model in two ways. First, you can visit the Playground and use the Models drop-down to select your newly fine-tuned model from the options listed. The other option is to use the "Playground" option shown in the Fine-tuning panel (see screenshot above) which launches the following comparitive view which shows the foundation and fine-tuned model versions side-by-side for quick evaluation.

Simply fill in the system context used in your training data and provide your test question. You will notice that both sides are updated with the identical context and question. Run the comparison and you will see the difference in outputs between them. Note how the fine-tuned model renders the response in the format you provided in your examples while the foundation model simply follows the system prompt.

You will notice that that the comparison also provides the token counts for each model, and the time taken for the inference. This specific example is a simplistic one meant to show the process but not actually reflecting a real world dataset or scenario. You may notice that both samples show the same number of tokens (system context and user prompt are identical) with the fine-tuned model taking more time for inference (custom model).

In real-world scenarios, you will not be using a toy example like this, but fine-tuning against real data (e.g., product catalog for customer service) where the quality of response will be much more apparent. In that context, getting an equivalent response quality with the foundation model will require more custom prompt engineering which will increase token usage and potentially the related processing time for inference. To try this out, check out the fine-tuning examples in the OpenAI Cookbook to start.