🏃‍♀️ Introduction

Hugging Face provides tools to quickly train neural networks for NLP (Natural Language Processing) on any task (classification, translation, question answering, etc) and any dataset with PyTorch and TensorFlow 2.0.

🤔 Why should I use W&B?

• Unified dashboard: Central repository for all your model metrics and predictions
• Lightweight: No code changes required to integrate with Hugging Face
• Accessible: Free for individuals and academic teams
• Secure: All projects are private by default
• Trusted: Used by machine learning teams at OpenAI, Toyota, Lyft and more

Think of Weights & Biases like GitHub for machine learning models — save machine learning experiments to your private, hosted dashboard. Experiment quickly with the confidence that all the versions of your models are saved for you, no matter where you're running your scripts.

W&B lightweight integrations works with any Python script, and all you need to do is sign up for a free W&B account to start tracking and visualizing your models.

In the HuggingFace Transformers repo, we've instrumented the Trainer to automatically log training and evaluation metrics to W&B at each logging step.

Here's an in depth look at how the integration works: Hugging Face + W&B Report.

🌴 Installation and Setup

First, let us install the latest version of Weights and Biases. We will then setup a few environment variables to enable Weights & Biases logging and finally authenticate this colab instance to use W&B.

Note: To enable logging to W&B, you will also need to set the report_to argument in your TrainingArguments or script to wandb.

🖊️ Sign-up/login

If this is your first time using Weights & Biases or you are not logged in, the link that appears after running wandb.login() in the following code cell will take you to sign-up/login page. Signing up for a free account is as easy as a few clicks.

🔑 Authentication

Once you've signed up, run the next cell. You'll be prompted to create a new API key at wandb.ai/settings if you haven't already. Store your API key securely. It can only be viewed once when created.

Task

Text classification is a common NLP task that assigns a label or class to text. Some of the largest companies run text classification in production for a wide range of practical applications. In this example we will use the TweetEval dataset to classify tweets into identify the emotions evoked by a tweet. The dataset is used as a benchmark to train models for tweet classification tasks. We will use then use a distilled verison of RoBERTa model - distilroberta-base to recoganize the emotions evoked by the tweets.

Data

Loading the data

Start by loading the tweet_eval dataset from the 🤗 Datasets library:

Understanding the dataset

There are two fields in this dataset:

• text: The text of the tweet.
• label: The integer label of the emotion corresponding to the tweet

Preprocessing

We need to convert the text to integer tokens so that they can be passed into the model as inputs. To do this we will use the distilroberta tokenizer to preprocess the text field in the dataset.

Create a preprocessing function to tokenize text and truncate sequences to be no longer than distilroberta's maximum input length:

To apply the preprocessing function over the entire dataset, use 🤗 Datasets map function. You can speed up map by setting batched=True to process multiple elements of the dataset at once:

The above step added two new columns to our dataset. input_ids and attention_mask. These are the inputs we will be passing to our model.

Since all our examples are of different lengths and the model expects a batch of tokens with the same length we will need to pad our inputs. We can use the DataCollatorWithPadding utility to do this. To further speed up training we will pre-compute the length of texts in the tokenized dataset and sort the dataset by this column. This ensures that the batches of data have as minimal padding as possible.

Now create a batch of examples using DataCollatorWithPadding. It's more efficient to dynamically pad the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximium length.

Evaluation

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 Evaluate library. For this task, load the f1-score metric. This is the metric used in the TweetEval benchmark. You will notice that this metric get logged automatically to your weights & biases run while training.

Your compute_metrics function is ready to go now, and you'll return to it when you setup your training.

Train

We are almost ready to train our model. The steps that remain include:

1. Define your training hyperparameters in TrainingArguments. The only required parameter is output_dir which specifies where to save your model. You'll also add the report_to="wandb" argument here. At the end of each epoch, the Trainer will evaluate the accuracy and save the training checkpoint. These metrics and checkpoints are automatically pushed to your wandb project.
2. Pass the training arguments to Trainer along with the model, dataset, tokenizer, data collator, and compute_metrics function.
3. Call train() to finetune your model.

We can visuzalize the training logs by looking at the wandb.run object or by clicking the link printed out above, or go to wandb.ai to see your results stream in live. The link to see your run in the browser will appear just before the training begins — look for the following output: "wandb: 🚀 View run at [URL to your unique run]"

Finally, we can optionally call the wandb.finish() method to indicate that the experiment is complete.

Resuming Training

But wait!! Looks like the model did not converge. Perhaps we should train for a few more epochs. Additionally, since we are training the model on colab it is possible that the preemptible instance was shutdown midway and that the model was not fully trained. Don't worry the wandb integration got us fully covered. We can easily resume training from the last checkpoint by doing the following.

1. Initialize the last wandb run by passing the run id from your Weights & Biases workspace to wandb.init
2. Download the lastest checkpoint using wandb.artifact.
3. Reinitialize the trainer and pass the artifact_dir to the resume_from_checkpoint argument in the trainer.train method.

Note: Change the last_run_id in the below cell to the id from your wandb run`

Note: Change the latest_checpointin the below cell to the checkpoint artifact from your run

Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

The simplest way to try out your finetuned model for inference is to use it in a pipeline(). Instantiate a pipeline for sentiment analysis with your model, and pass your text to it. Here we will create a new wandb.run to download the model artifact. Then we simply pass the artifact_dir as the pretrained model to the model argument in the pipeline.