Fine Tune GPT 3 With Weights & Biases
Fine-tune GPT-3 with Weights & Biases
OpenAI’s API gives practitioners access to GPT-3, an incredibly powerful natural language model that can be applied to virtually any task that involves understanding or generating natural language.
If you use OpenAI's API to fine-tune GPT-3, you can now use the W&B integration to track experiments, models, and datasets in your central dashboard.
All it takes is one line: openai wandb sync
Set up your API key
WARNING: Remove the API key after running the cell and clear output so it does not get logged to wandb in case you sync code (see settings)
Install dependencies
You may see a warning to restart runtime. If so, restart it.
Optional: Fine-tune GPT-3
It's always more fun to experiment with your own projects so if you have already used the openai API to fine-tune GPT-3, just skip this section!
Otherwise let's fine-tune GPT-3 on Wikipedia!
Imports and initial set-up
Dataset Preparation
We created a dataset from Wikipedia-based Image Text (WIT) Dataset:
- only english items
- prompt: title of the page
- completion: first sentence of page description
The dataset was logged to W&B and can be explored at borisd13/GPT-3/wiki-dataset.
We now split it into training/validation dataset.
A copy of our dataset is now cached locally.
Let's look at a few samples.
We can verify that the data is correctly formatted with openai client.
The file is very large (1.5M samples). For this demo, we'll extract:
- training set: 50k top samples
- validation set: 10k bottom samples
Let's log our train/valid split as W&B artifact.
We can add any file and many types of objects into an artifact.
We create artifacts for training & validation sets that will contain the associated file as well as a W&B Table for interactive exploration.
We can now close our run.
Create a fine-tuned model
We'll now use OpenAI API to fine-tune GPT-3.
Let's first recover our training & validation files, latest version (could also be v0, v1 or any alias we associated with it)
Let's define our GPT-3 fine-tuning hyper-parameters.
Time to train the model!
We can run a few different fine-tunes with different parameters or even with different datasets.
Sync fine-tune jobs to Weights & Biases
We can log our fine-tunes with a simple command.
Our fine-tunes are now successfully synced to Weights & Biases.
Anytime we have new fine-tunes, we can just call openai wandb sync to add them to our dashboard.
Log inference samples
The best way to evaluate a generative model is to explore sample predictions.
Let's generate a few inference samples and log them to W&B.
We can easily retrieve all config parameters from a job file.
Job files are logged to W&B as artifacts and can be accessed with run.use_artifact('USERNAME/PROJECT/job_details:VERSION') where VERSION is either:
- a version number such as
v2 - the fine-tune id such as
ft-xxxxxxxxx - an alias added automatically such as
latestor manually
You can explore them in your artifacts dashboard.
All the details of the job are present in its metadata.
Let's take advantage to add metadata into our eval run config.
We can easily access model id from any job.
Let's now retrive our latest validation file and extract a few samples from it
We'll perform the inference only on a few examples.
We create and log a W&B Table to easily explore, query & compare model predictions.
We can also log predictions on celebrities.