Demonstrating Industrial Workplace Safety using Pre-trained Machine Learning Models

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Introduction

This sample notebook shows how to use pre-trained model packages from AWS Marketplace to detect industrial workspace safety related object labels, such as hard-hat, personal protective equipment, construction machinery, and construction worker in an image. The notebook also shows an approach to perform inference on a video by taking snapshots from the video file to generate an activity/status log. At the end of this you will become familiar on steps to integrate inferences from pre-trained models into your application. This notebook is intended for demonstration, we highly recommend you to evaluate the accuracy of machine learning models to see if they meet your expectations.

Pre-requisites

This sample notebook requires you to subscribe to pre-trained machine learning model packages. Follow the following steps to subscribe to the listings:

1. Open the following model package product detail pages, in separate tabs, in your web browser.

2. Construction Worker Detection to identify construction workers in an image.

3. Hard Hat Detector for Worker Safety model to infer if construction workers are wearing hard hats.

4. Personal Protective Equipments to infer if a person is wearing a high visibility safety vest.

5. Construction Machines Detector to identify construction machines in an image.

6. For each of the model packages, follow these steps:

7. Review the information available on the product details page including Support Terms .

8. Click on "Continue to Subscribe". You will now see the "Subscribe to this software" page.

9. Review End User License Agreement and Pricing Terms.

10. "Accept Offer" button needs to be clicked if your organization agrees with EULA, pricing information and support terms.

Notes:

1. Once you click on Continue to configuration button and then choose a region, you will see a Product Arn displayed. This is the model package ARN that you need to specify while creating a deployable model using Boto3. However, for this notebook, the model ARNs have been specified in src/model_package_arns.py file and you need not specify them explicitly. The configuration page also provides a "View in SageMaker" button to navigate to Amazon SageMaker to deploy via Amazon SageMaker Console.
2. Products with Free Trials, do not incur hourly software charges during free trial period, but AWS infrastructure charges still apply. Free Trials will automatically convert to a paid hourly subscription upon expiration. We have included steps below to cancel subscription at the end of this exercise.

Step 1: Set up environment and view sample images

Step 1.1: Set up environment

In this section, we will first import necessary libraries and define variables such as an S3 bucket, an IAM role, and an Amazon SageMaker session.

Next, we will create utility functions to:

1. Draw a bounding box on an image based on coordinates.
2. Display an image.
3. Perform a prediction.

Step 1.2: View sample images

We will now view the sample images used to perform an inference.

Step 1.2.1: View construction site image

Run the next cell to view an image of a construction site with workers and a truck. The workers are wearing personal protective equipment - hard hat, and safety vest.

Courtesy - https://pixabay.com/photos/construction-worker-safety-2578410

Step 1.2.2: View an image with a worker and a person at a workplace

The following image shows two people, a worker wearing a high-visibility vest and a person.

Courtesy - https://www.pexels.com/photo/two-men-wearing-white-hard-hat-901941

Step 1.2.3: View an image with an excavator and a truck at work

The following image shows a truck and an excavator.

Courtesy - https://pixabay.com/photos/earth-390f-hydraulic-excavators-2579434/

We will deploy pre-trained models to generate inferences using sample images.

Step 2: Deploy construction worker detection model

In this step, you will deploy the Construction Worker Detection model package and perform an inference using sample images.

Step 2.1: Deploy the model for performing real-time inference.

While the model is deploying, review the Usage Information and Additional Resources section from the model package detail page to understand the I/O interface of the model.

Step 2.2: Perform a prediction (Test 1)

In this step, we will perform a prediction using the construction-site image.

You can see that the model recognized all the three workers found in the image.

Step 2.3: Perform a prediction (Test 2)

Let us perform inference using the worker/person image and see how the model can identify a worker and a person.

You can see that the model can differentiate between a construction worker (on the left) and a non-construction worker (person on the right).

AWS Marketplace also contains another model you may want to try for construction worker detection.

Step 3: Deploy the hard-hat detection model.

In this step, we will deploy the Hard Hat Detector for Worker Safety model to identify whether people in the image are wearing hard hats.

Step 3.1: Deploy the model for performing real-time inference.

Step 3.2: Perform real-time inference on the model.

Note, the pre-trained model could identify all three hard-hats found in the picture with high probabilities.

Step 4. Deploy the Personal Protective Equipment (PPE) detection model

Next, we will deploy Personal Protective Equipment machine learning model to identify whether the person in the image is wearing PPE such as a high visibility vest.

Step 4.1: Deploy the model for performing real-time inference.

Step 4.2: Perform real-time inference on the model.

Note how the pre-trained model could identify the PPEs in the image with high probabilities.

Step 5. Deploy the Construction Machines detection model

Next, you will deploy Construction Machines Detector to identify construction machines from an image.

Step 5.1: Deploy the model for performing real-time inference.

Step 5.2: Perform real-time inference on the model (Test 1).

Model did not detect any construction machines since there were none. We will now perform inference on one more image that shows construction machinery such as an excavator.

Step 5.3: Perform real-time inference on the model (Test 2).

Note how the pre-trained model could detect both, a truck, and an excavator from the picture.

Step 6. Generate actionable insights on video input

The pre-trained models demonstrated above accept images as an input. However, the input data can also be in the form of a video. In this section, you will see how to extract actionable insights from a video by performing inference on snapshots.

Courtesy - https://pixabay.com/videos/construction-road-excavator-worker-26239/ (Edited)

Analyzing hours of video footage can be tedious. Status summary reports and rules can help detect non-compliance to trigger alarms. In this section, we will generate following status summary from the video:

Sample Summary report
No Alarm : 1 truck(s), 1 excavator(s), no workers found.
No Alarm : 2 truck(s), 1 excavator(s), 1 workers found.
No Alarm : 1 truck(s), 1 excavator(s), 1 workers found.
No Alarm : 1 truck(s), 1 excavator(s), no workers found.
ALARM : 1 worker(s) wearing PPE but 0 wearing hard hats, 1 truck(s), 1 excavator(s) found.
No Alarm : 1 truck(s), 1 excavator(s), no workers found.
ALARM : 1 worker(s) wearing PPE but 0 wearing hard hats, 1 truck(s), 1 excavator(s) found.

Note: There are couple instances in the video when the worker is not visible because of an obstruction.

In this task, we will take a snapshot from the video every 1.5 seconds and then perform inference on each snapshot to identify actionable insights. The snapshot images from the video enable you to to generate inferences from model packages that only support image payloads. In some cases, this approach may help you scale usage of endpoints.

Now that we have created snapshots from the video, let us create a utility function that generates status summary

Next, we will run the utility function on each snapshot to generate status summary log from the video.

Step 7. Explore other relevant models!

You just learnt how pre-trained machine learning models can identify metadata from workplace pictures (or snapshots of a video). This metadata can be used to set up alarms to detect non-compliance.

Checkout these additional relevant models:

1. Person and Truck Detector to identify trucks and people from an image.
2. Modjoul Geo Fence model informs an organization of employee and equipment location and the activities and movements within that location.
3. Modjoul Automotive Telematics Model can identify aggressive events such as hard braking and hard acceleration, duration of driving and distance of driving.
4. Modjoul Asset Utilization Model to understand the utilization of heavy equipments such as back hoes, generators, dump trucks, etc.

Step 8. Cleanup

Next, clean-up deployable models as well as endpoints from your account.

If you would like to unsubscribe to the model, follow these steps. Before you cancel the subscription, ensure that you do not have any deployable model created from the model package or using the algorithm. Note - You can find this information by looking at the container name associated with the model.

Steps to unsubscribe to product from AWS Marketplace:

1. Navigate to Machine Learning tab on Your Software subscriptions page
2. Locate the listing that you would need to cancel subscription for, and then choose Cancel Subscription to cancel the subscription.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.