🏋️‍♀️ W&B + 🧪 Scikit-learn

Use Weights & Biases for machine learning experiment tracking, dataset versioning, and project collaboration.

What this notebook covers:

• Easy integration of Weights and Biases with Scikit.
• W&B Scikit plots for model interpretation and diagnostics for regression, classification, and clustering.

Note: Sections starting with Step are all you need to integrate W&B to existing code.

The interactive W&B Dashboard will look like this:

Step 0: Install W&B

Step 1: Import W&B and Login

Regression

Let's check out a quick example

Step 2: Initialize W&B run

Step 3: Visualize model performance

Residual Plot

Measures and plots the predicted target values (y-axis) vs the difference between actual and predicted target values (x-axis), as well as the distribution of the residual error.

Generally, the residuals of a well-fit model should be randomly distributed because good models will account for most phenomena in a data set, except for random error.

Check out the official documentation here $\rightarrow$

Outlier Candidate

Measures a datapoint's influence on regression model via Cook's distance. Instances with heavily skewed influences could potentially be outliers. Useful for outlier detection.

Check out the official documentation here $\rightarrow$

All-in-one: Regression plot

Using this all in one API one can:

• Log summary of metrics
• Log learning curve
• Log outlier candidates
• Log residual plot

Classification

Let's check out a quick example.

Step 2: Initialize W&B run

Step 3: Visualize model performance

Class Proportions

Plots the distribution of target classes in training and test sets. Useful for detecting imbalanced classes and ensuring that one class doesn't have a disproportionate influence on the model.

Check out the official documentation here $\rightarrow$

Learning Curve

Trains model on datasets of varying lengths and generates a plot of cross validated scores vs dataset size, for both training and test sets.

Check out the official documentation here $\rightarrow$

ROC

ROC curves plot true positive rate (y-axis) vs false positive rate (x-axis). The ideal score is a TPR = 1 and FPR = 0, which is the point on the top left. Typically we calculate the area under the ROC curve (AUC-ROC), and the greater the AUC-ROC the better.

Check out the official documentation here $\rightarrow$

Precision Recall Curve

Computes the tradeoff between precision and recall for different thresholds. A high area under the curve represents both high recall and high precision, where high precision relates to a low false positive rate, and high recall relates to a low false negative rate.

Check out the official documentation here $\rightarrow$

Feature Importances

Evaluates and plots the importance of each feature for the classification task. Only works with classifiers that have a feature_importances_ attribute, like trees.

Check out the official documentation here $\rightarrow$

All-in-one: Classifier Plot

Using this all in one API one can:

• Log feature importance
• Log learning curve
• Log confusion matrix
• Log summary metrics
• Log class proportions
• Log calibration curve
• Log roc curve
• Log precision recall curve

Clustering

Step 2: Initialize W&B run

Step 3: Visualize model performance

Elbow Plot

Measures and plots the percentage of variance explained as a function of the number of clusters, along with training times. Useful in picking the optimal number of clusters.

Check out the official documentation here $\rightarrow$

Silhouette Plot

Measures & plots how close each point in one cluster is to points in the neighboring clusters. The thickness of the clusters corresponds to the cluster size. The vertical line represents the average silhouette score of all the points.

Check out the official documentation here $\rightarrow$

All in one: Clusterer Plot

Using this all-in-one API you can:

• Log elbow curve
• Log silhouette plot

Sweep 101

Use Weights & Biases Sweeps to automate hyperparameter optimization and explore the space of possible models.

Check out Hyperparameter Optimization in PyTorch using W&B Sweeps $\rightarrow$

Running a hyperparameter sweep with Weights & Biases is very easy. There are just 3 simple steps:

1. Define the sweep: We do this by creating a dictionary or a YAML file that specifies the parameters to search through, the search strategy, the optimization metric et all.

2. Initialize the sweep: sweep_id = wandb.sweep(sweep_config)

3. Run the sweep agent: wandb.agent(sweep_id, function=train)

And voila! That's all there is to running a hyperparameter sweep! In the notebook below, we'll walk through these 3 steps in more detail.

Example Gallery

See examples of projects tracked and visualized with W&B in our gallery, Fully Connected →

Basic Setup

1. Projects: Log multiple runs to a project to compare them. wandb.init(project="project-name")
2. Groups: For multiple processes or cross validation folds, log each process as a runs and group them together. wandb.init(group='experiment-1')
3. Tags: Add tags to track your current baseline or production model.
4. Notes: Type notes in the table to track the changes between runs.
5. Reports: Take quick notes on progress to share with colleagues and make dashboards and snapshots of your ML projects.

Advanced Setup

1. Environment variables: Set API keys in environment variables so you can run training on a managed cluster.
2. Offline mode: Use dryrun mode to train offline and sync results later.
3. On-prem: Install W&B in a private cloud or air-gapped servers in your own infrastructure. We have local installations for everyone from academics to enterprise teams.