Automating Traditional Machine Learning with WANDB

featuring pandas-profiling, pycaret, evidently, and fastapi

What is AutoML?

AutoML automates the most time-consuming tasks of machine learning model development, allowing data scientists to train effective models more quickly. Traditionally, building a model has required domain expertise and time consuming iteration. AutoML accelerates this process.

The advantages of AutoML can be summed up in three major points:

• Increases productivity by automating repetitive tasks. This enables a data scientist to focus more on the problem rather than the models.
• Automating the ML pipeline also helps to avoid errors that might creep in manually.
• Ultimately, AutoML is a step towards democratizing machine learning by making the power of ML accessible to everybody.

Notebook TOC

We will be showing different aspects of automation we can add alongside flexible MLOPs tools.

W&B is the best choice for our use case due to its data artifacts with lineaging, interactive data driven reports (which can be orchestrated to update as our models and experiments expand), metadata capabilties for orchestration, and flexibility to add in these capabilities in a low code no setup fashion.

1. Adding easy, automatic Data Lineage for data artifacts via W&B and a prepared dataset. Alongside pandas-profiling which will automate a lot of the EDA process and provide an artifact for which we can refer to at any time for this dataset, which can be passed to colleagues

2. Adding low code, automatic model training and hyperparameter optimization which picks latest data artfiacts from W&B, marks it as an input to the PyCaret experiment we will run, and then train, analyze, and generate model artifacts to evaluate on for production purposes in W&B

3. Simulate analyzing possible data drift/model drift on data by using Evidently to analyze simulated incoming "production" data with ground truth. Using this "production" data we calculate a variety of metrics and assign it to the W&B metadata of the model which can be used as a selector of the best model artifacts

4. Use the W&B metadata and a selected metric to search through the model candidates and promote the model which has the best selected metric. In this case the promotion tag we hard-code is "production"

5. (optional) Do things with your production model! Load the model easily to flexibly do things such as serve a prediction endpoint in a backend framework of your choice.

Code

General Accessory Functions

1: Data Artifacts with W&B and Pandas Profiling

Pandas Profiling

Generates profile reports from a pandas DataFrame.

The pandas df.describe() function is great but a little basic for serious exploratory data analysis. pandas_profiling extends the pandas DataFrame with df.profile_report() for quick data analysis.

For each column the following statistics - if relevant for the column type - are presented in an interactive HTML report:

• Type inference: detect the types of columns in a dataframe.
• Essentials: type, unique values, missing values
• Quantile statistics like minimum value, Q1, median, Q3, maximum, range, interquartile range
• Descriptive statistics like mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness
• Most frequent values
• Histogram
• Correlations highlighting of highly correlated variables, Spearman, Pearson and Kendall matrices
• Missing values matrix, count, heatmap and dendrogram of missing values
• Text analysis learn about categories (Uppercase, Space), scripts (Latin, Cyrillic) and blocks (ASCII) of text data.
• File and Image analysis extract file sizes, creation dates and dimensions and scan for truncated images or those containing EXIF information.

In our example we will be finding ourselves utilizing our data pool for 3 general categories of data batches we may interact with as an ML practictioner:

• Training Data
• Validation Data
• "Production" Unseen Data

In our case our package of choice pycaret will be handling the partitioning of data into training and validation from a pool of provided data (the data for modeling)

For this tutorial we will use an UCI data set called Default of Credit Card Clients Dataset. This data set contains information about default payments, demographics, credit data, payment history and billing statements of credit card customers in Taiwan from April 2005 to September 2005. There are 24,000 samples and 24 characteristics.

There are 24 variables:

• LIMIT_BAL: Amount of given credit in NT dollars (includes individual and family/supplementary credit
• SEX: Gender (1=male, 2=female)
• EDUCATION: (1=graduate school, 2=university, 3=high school, 4=others, 5=unknown, 6=unknown)
• MARRIAGE: Marital status (1=married, 2=single, 3=others)
• AGE: Age in years
• PAY_0: Repayment status in September, 2005 (-1=pay duly, 1=payment delay for one month, 2=payment delay for two months, … 8=payment delay for eight months, 9=payment delay for nine months and above)
• PAY_2: Repayment status in August, 2005 (scale same as above)
• PAY_3: Repayment status in July, 2005 (scale same as above)
• PAY_4: Repayment status in June, 2005 (scale same as above)
• PAY_5: Repayment status in May, 2005 (scale same as above)
• PAY_6: Repayment status in April, 2005 (scale same as above)
• BILL_AMT1: Amount of bill statement in September, 2005 (NT dollar)
• BILL_AMT2: Amount of bill statement in August, 2005 (NT dollar)
• BILL_AMT3: Amount of bill statement in July, 2005 (NT dollar)
• BILL_AMT4: Amount of bill statement in June, 2005 (NT dollar)
• BILL_AMT5: Amount of bill statement in May, 2005 (NT dollar)
• BILL_AMT6: Amount of bill statement in April, 2005 (NT dollar)
• PAY_AMT1: Amount of previous payment in September, 2005 (NT dollar)
• PAY_AMT2: Amount of previous payment in August, 2005 (NT dollar)
• PAY_AMT3: Amount of previous payment in July, 2005 (NT dollar)
• PAY_AMT4: Amount of previous payment in June, 2005 (NT dollar)
• PAY_AMT5: Amount of previous payment in May, 2005 (NT dollar)
• PAY_AMT6: Amount of previous payment in April, 2005 (NT dollar)
• default.payment.next.month or default: Default payment (1=yes, 0=no)

2: Run and Log Experiment with W&B and PyCaret

PyCaret

PyCaret is an open-source, low-code machine learning library in Python that automates machine learning workflows. It is an end-to-end machine learning and model management tool that speeds up the experiment cycle exponentially and makes you more productive.

In comparison with the other open-source machine learning libraries, PyCaret is an alternate low-code library that can be used to replace hundreds of lines of code with few lines only. This makes experiments exponentially fast and efficient. PyCaret is essentially a Python wrapper around several machine learning libraries and frameworks such as scikit-learn, XGBoost, LightGBM, CatBoost, spaCy, Optuna, Hyperopt, Ray, and few more.

The design and simplicity of PyCaret are inspired by the emerging role of citizen data scientists, a term first used by Gartner. Citizen Data Scientists are power users who can perform both simple and moderately sophisticated analytical tasks that would previously have required more technical expertise.

Internally this is what is occuring with the data that was allowed for training as shown in the sample split graphic.

Cross-validation uses different portions of data to test and train a model on different iterations to estimate how accurately a predictive model will perform in practice.

In a prediction problem, a model is usually given a dataset of known data on which training is run (training dataset), and a dataset of unknown data (or first seen data) against which the model is tested (called the validation dataset or testing set).

The goal of cross-validation is to test the model's ability to predict new data that was not used in estimating it, in order to flag problems like overfitting or selection bias and to give an insight on how the model will generalize to an independent dataset (i.e., an unknown dataset, for instance production data).

One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, in most methods multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to give an estimate of the model's predictive performance.

In summary, cross-validation combines (averages) measures of fitness in prediction to derive a more accurate estimate of model prediction performance.

3: Detect Data Drift and Evaluate Models with W&B and Evidently

Evidently

Evidently helps evaluate machine learning models during validation and monitor them in production. The tool generates interactive visual reports and JSON profiles from pandas DataFrame or csv files.

You can use visual reports for ad hoc analysis, debugging and team sharing, and JSON profiles to integrate Evidently in prediction pipelines or with other visualization tools.

Currently 6 reports are available.

1. Data Drift Detects changes in feature distribution.

2. Numerical Target Drift Detects changes in numerical target and feature behavior.

3. Categorical Target Drift Detects changes in categorical target and feature behavior.

4. Regression Model Performance Analyzes the performance of a regression model and model errors.

5. Classification Model Performance Analyzes the performance and errors of a classification model. Works both for binary and multi-class models.

6. Probabilistic Classification Model Performance Analyzes the performance of a probabilistic classification model, quality of model calibration, and model errors. Works both for binary and multi-class models.

4: Promote Model based on Evaluation Metric with W&B

Note: This feature is currently in beta. Let us know what you think about it!

Aside: Let's look at our dashboards real quick!

5: Load promoted model to predict with W&B and PyCaret into FASTApi

Note: Use the link from the Ngroktunnel and append on /docs to be able to see the resultant endpoint definition and be able to try your own values!