Weights & Biases Artifacts Time-to-live (TTL) Walkthrough

W&B Artifacts now supports setting time-to-live policies on each version of an Artifact. The feature is currently available in W&B SaaS Cloud and will be released to Enterprise customers using W&B Server in version 0.42.0. The following examples show the use TTL policy in a common Artifact logging workflow. We'll cover:

• Setting a TTL policy when creating an Artifact
• Retroactively setting TTL for a specific Artifact aliases
• Using the W&B API to set a TTL for all versions of an Artifact

Setup

Let's do a few things before we get started. Below we will:

• Install the wandb library and download a dataset

log to wandb

Image Sampling

For the purposes of the walkthrough, we will sample from the Imagenette dataset and organize them into training and validation directories in our Colab session. The block below:

• Creates folders for our sampled images if they don't already exist
• Selects a random sample of images from the Imagenette dataset
• Organizes the samples into training and validation directories

Note: we overwrite the files every time we execute this so we get new Artifact versions.

We are going to use Imagenette dataset for this example. Imagenette is a subset of 10 easily classified classes from Imagenet (tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute). It was created by Jeremy Howard and is a great dataset to experiment with.

Image Preview

Quick block to view some of the images in the sampled dataset.

Setting TTL on New Artifacts

Below we create two new Artifacts for our real and fake data. Because we have internal retention policies in hypothetical organization we'd like to remove any Artifact that has real data (potentially containing personal data). Below we:

• Create a W&B Run to track the logging of these raw data Artifacts
• Set the ttl attribute on the real raw data
• Log our two Artifacts

We will use the train dataset as our real data and the validation dataset as our fake data.

Updating/Retroactively Setting TTL on Artifacts

In our hypothetical organization we've been given approval to retain a specific version of our data indefinitely. We've also been given approval to extend the retention date of an additional dataset. Below we'll:

• Extend the TTL of an Artifact tagged with the extended alias
• Remove the TTL of an Artifact tagged with the compliant alias
• Programmatically check the status of these two Artifacts

Use W&B Import/Export API to Iterate Artifact Versions and Set TTL

Let's say we've received approval to retain all of the data within a given Artifact and we'd like to remove all TTL policies for every version of an Artifact. Below we:

• Use the W&B API to get a list of all Runs in a project
• Get a list of all versions of a specific Artifact (e.g. fake-raw)
• Iterate over each version and remove any existing TTL policy associated with the version

To apply a TTL policy to all artifacts within a team's projects, team admins can set default TTL policies for their team. The default will be applied to both existing and future artifacts logged to projects as long as no custom policies have been set. To learn more about configuring a team default TTL, visit this section of the W&B documentation.

Traverse an Artifact Graph to Set Downstream TTL

In this last section, we'll do some preprocessing on our images and log those as downstream Artifacts. Once again we'll use the W&B Import/Export API to set a TTL policy on our downstream images for images that originated from our "real" dataset.

Preprocess and log a new Artifact

Traverse the Artifact Graph and Set TTL

Let's take a look at the original real dataset and traverse downstream runs and Artifacts to set a TTL policy on anything that originated from the real dataset.