Amazon Web Services Sm Clarify Time Series Bring Your Own Model

Sm Clarify Time Series Bring Your Own Model

data-scienceinferenceamazon-sagemaker-examplesreinforcement-learningmachine-learningawsresponsible_aiexamplesdeep-learningsm-clarify_time_series_bring_your_own_modelsagemakerjupyter-notebooktrainingmlops

alph-notebooks/amazon-sagemaker-examples / sm-clarify_time_series_bring_your_own_model.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

TimeSeries Bring Your Own Model

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Runtime

This notebook takes approximately 30 minutes to run.

Prerequisites

Install Mercury

If not already installed, the following cell will install the mercury package in order to display the analysis_config.json and explainability job output within the notebook.

[1]

WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: dj-rest-auth 3.0.0 does not provide the extra 'with-social'
WARNING: Error parsing requirements for pyyaml: [Errno 2] No such file or directory: '/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages/PyYAML-6.0.1.dist-info/METADATA'
WARNING: Error parsing requirements for boto3: [Errno 2] No such file or directory: '/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages/boto3-1.34.101.dist-info/METADATA'
WARNING: Error parsing requirements for botocore: [Errno 2] No such file or directory: '/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages/botocore-1.34.101.dist-info/METADATA'
WARNING: Error parsing requirements for s3transfer: [Errno 2] No such file or directory: '/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages/s3transfer-0.10.1.dist-info/METADATA'
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
    WARNING: No metadata found in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages
    WARNING: No metadata found in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages
    WARNING: No metadata found in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages
    WARNING: No metadata found in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
awscli 1.31.7 requires botocore==1.33.7, but you have botocore 1.29.165 which is incompatible.
awscli 1.31.7 requires s3transfer<0.9.0,>=0.8.0, but you have s3transfer 0.6.2 which is incompatible.
sagemaker 2.219.0 requires boto3<2.0,>=1.33.3, but you have boto3 1.26.83 which is incompatible.

Install SageMaker

[2]

WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
Collecting sagemaker
  Using cached sagemaker-2.219.0-py3-none-any.whl.metadata (14 kB)
Collecting attrs<24,>=23.1.0 (from sagemaker)
  Using cached attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting boto3<2.0,>=1.33.3 (from sagemaker)
  Using cached boto3-1.34.103-py3-none-any.whl.metadata (6.6 kB)
Collecting cloudpickle==2.2.1 (from sagemaker)
  Using cached cloudpickle-2.2.1-py3-none-any.whl.metadata (6.9 kB)
Collecting google-pasta (from sagemaker)
  Using cached google_pasta-0.2.0-py3-none-any.whl.metadata (814 bytes)
Collecting numpy<2.0,>=1.9.0 (from sagemaker)
  Using cached numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Collecting protobuf<5.0,>=3.12 (from sagemaker)
  Using cached protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Collecting smdebug-rulesconfig==1.0.1 (from sagemaker)
  Using cached smdebug_rulesconfig-1.0.1-py2.py3-none-any.whl.metadata (943 bytes)
Collecting importlib-metadata<7.0,>=1.4.0 (from sagemaker)
  Using cached importlib_metadata-6.11.0-py3-none-any.whl.metadata (4.9 kB)
Collecting packaging>=20.0 (from sagemaker)
  Using cached packaging-24.0-py3-none-any.whl.metadata (3.2 kB)
Collecting pandas (from sagemaker)
  Using cached pandas-2.2.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
Collecting pathos (from sagemaker)
  Using cached pathos-0.3.2-py3-none-any.whl.metadata (11 kB)
Collecting schema (from sagemaker)
  Using cached schema-0.7.7-py2.py3-none-any.whl.metadata (34 kB)
Collecting PyYAML~=6.0 (from sagemaker)
  Using cached PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting jsonschema (from sagemaker)
  Using cached jsonschema-4.22.0-py3-none-any.whl.metadata (8.2 kB)
Collecting platformdirs (from sagemaker)
  Using cached platformdirs-4.2.1-py3-none-any.whl.metadata (11 kB)
Collecting tblib<4,>=1.7.0 (from sagemaker)
  Using cached tblib-3.0.0-py3-none-any.whl.metadata (25 kB)
Collecting urllib3<3.0.0,>=1.26.8 (from sagemaker)
  Using cached urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting requests (from sagemaker)
  Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting docker (from sagemaker)
  Using cached docker-7.0.0-py3-none-any.whl.metadata (3.5 kB)
Collecting tqdm (from sagemaker)
  Using cached tqdm-4.66.4-py3-none-any.whl.metadata (57 kB)
Collecting psutil (from sagemaker)
  Using cached psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting botocore<1.35.0,>=1.34.103 (from boto3<2.0,>=1.33.3->sagemaker)
  Using cached botocore-1.34.103-py3-none-any.whl.metadata (5.7 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3<2.0,>=1.33.3->sagemaker)
  Using cached jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3<2.0,>=1.33.3->sagemaker)
  Using cached s3transfer-0.10.1-py3-none-any.whl.metadata (1.7 kB)
Collecting zipp>=0.5 (from importlib-metadata<7.0,>=1.4.0->sagemaker)
  Using cached zipp-3.18.1-py3-none-any.whl.metadata (3.5 kB)
Collecting charset-normalizer<4,>=2 (from requests->sagemaker)
  Using cached charset_normalizer-3.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests->sagemaker)
  Using cached idna-3.7-py3-none-any.whl.metadata (9.9 kB)
Collecting certifi>=2017.4.17 (from requests->sagemaker)
  Using cached certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Collecting six (from google-pasta->sagemaker)
  Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting jsonschema-specifications>=2023.03.6 (from jsonschema->sagemaker)
  Using cached jsonschema_specifications-2023.12.1-py3-none-any.whl.metadata (3.0 kB)
Collecting referencing>=0.28.4 (from jsonschema->sagemaker)
  Using cached referencing-0.35.1-py3-none-any.whl.metadata (2.8 kB)
Collecting rpds-py>=0.7.1 (from jsonschema->sagemaker)
  Using cached rpds_py-0.18.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting python-dateutil>=2.8.2 (from pandas->sagemaker)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas->sagemaker)
  Using cached pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.7 (from pandas->sagemaker)
  Using cached tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting ppft>=1.7.6.8 (from pathos->sagemaker)
  Using cached ppft-1.7.6.8-py3-none-any.whl.metadata (12 kB)
Collecting dill>=0.3.8 (from pathos->sagemaker)
  Using cached dill-0.3.8-py3-none-any.whl.metadata (10 kB)
Collecting pox>=0.3.4 (from pathos->sagemaker)
  Using cached pox-0.3.4-py3-none-any.whl.metadata (8.0 kB)
Collecting multiprocess>=0.70.16 (from pathos->sagemaker)
  Using cached multiprocess-0.70.16-py39-none-any.whl.metadata (7.2 kB)
Collecting urllib3<3.0.0,>=1.26.8 (from sagemaker)
  Using cached urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB)
Using cached sagemaker-2.219.0-py3-none-any.whl (1.5 MB)
Using cached cloudpickle-2.2.1-py3-none-any.whl (25 kB)
Using cached smdebug_rulesconfig-1.0.1-py2.py3-none-any.whl (20 kB)
Using cached attrs-23.2.0-py3-none-any.whl (60 kB)
Using cached boto3-1.34.103-py3-none-any.whl (139 kB)
Using cached importlib_metadata-6.11.0-py3-none-any.whl (23 kB)
Using cached numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
Using cached packaging-24.0-py3-none-any.whl (53 kB)
Using cached protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl (294 kB)
Using cached PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB)
Using cached tblib-3.0.0-py3-none-any.whl (12 kB)
Using cached docker-7.0.0-py3-none-any.whl (147 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
Using cached jsonschema-4.22.0-py3-none-any.whl (88 kB)
Using cached pandas-2.2.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.1 MB)
Using cached pathos-0.3.2-py3-none-any.whl (82 kB)
Using cached platformdirs-4.2.1-py3-none-any.whl (17 kB)
Using cached psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB)
Using cached schema-0.7.7-py2.py3-none-any.whl (18 kB)
Using cached tqdm-4.66.4-py3-none-any.whl (78 kB)
Using cached botocore-1.34.103-py3-none-any.whl (12.2 MB)
Using cached urllib3-1.26.18-py2.py3-none-any.whl (143 kB)
Using cached certifi-2024.2.2-py3-none-any.whl (163 kB)
Using cached charset_normalizer-3.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
Using cached dill-0.3.8-py3-none-any.whl (116 kB)
Using cached idna-3.7-py3-none-any.whl (66 kB)
Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Using cached jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB)
Using cached multiprocess-0.70.16-py39-none-any.whl (133 kB)
Using cached pox-0.3.4-py3-none-any.whl (29 kB)
Using cached ppft-1.7.6.8-py3-none-any.whl (56 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached pytz-2024.1-py2.py3-none-any.whl (505 kB)
Using cached referencing-0.35.1-py3-none-any.whl (26 kB)
Using cached rpds_py-0.18.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Using cached s3transfer-0.10.1-py3-none-any.whl (82 kB)
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Using cached tzdata-2024.1-py2.py3-none-any.whl (345 kB)
Using cached zipp-3.18.1-py3-none-any.whl (8.2 kB)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
Installing collected packages: schema, pytz, zipp, urllib3, tzdata, tqdm, tblib, smdebug-rulesconfig, six, rpds-py, PyYAML, psutil, protobuf, ppft, pox, platformdirs, packaging, numpy, jmespath, idna, dill, cloudpickle, charset-normalizer, certifi, attrs, requests, referencing, python-dateutil, multiprocess, importlib-metadata, google-pasta, pathos, pandas, jsonschema-specifications, docker, botocore, s3transfer, jsonschema, boto3, sagemaker
  Attempting uninstall: schema
    Found existing installation: schema 0.7.7
    Uninstalling schema-0.7.7:
      Successfully uninstalled schema-0.7.7
  Attempting uninstall: pytz
    Found existing installation: pytz 2024.1
    Uninstalling pytz-2024.1:
      Successfully uninstalled pytz-2024.1
  Attempting uninstall: zipp
    Found existing installation: zipp 3.18.1
    Uninstalling zipp-3.18.1:
      Successfully uninstalled zipp-3.18.1
  Attempting uninstall: urllib3
    Found existing installation: urllib3 1.26.18
    Uninstalling urllib3-1.26.18:
      Successfully uninstalled urllib3-1.26.18
  Attempting uninstall: tzdata
    Found existing installation: tzdata 2024.1
    Uninstalling tzdata-2024.1:
      Successfully uninstalled tzdata-2024.1
  Attempting uninstall: tqdm
    Found existing installation: tqdm 4.66.4
    Uninstalling tqdm-4.66.4:
      Successfully uninstalled tqdm-4.66.4
  Attempting uninstall: tblib
    Found existing installation: tblib 3.0.0
    Uninstalling tblib-3.0.0:
      Successfully uninstalled tblib-3.0.0
  Attempting uninstall: smdebug-rulesconfig
    Found existing installation: smdebug-rulesconfig 1.0.1
    Uninstalling smdebug-rulesconfig-1.0.1:
      Successfully uninstalled smdebug-rulesconfig-1.0.1
  Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
  Attempting uninstall: rpds-py
    Found existing installation: rpds-py 0.18.1
    Uninstalling rpds-py-0.18.1:
      Successfully uninstalled rpds-py-0.18.1
  Attempting uninstall: PyYAML
    WARNING: No metadata found in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages
    Found existing installation: pyyaml 6.0
    Can't uninstall 'pyyaml'. No files were found to uninstall.
  Attempting uninstall: psutil
    Found existing installation: psutil 5.9.8
    Uninstalling psutil-5.9.8:
      Successfully uninstalled psutil-5.9.8
  Attempting uninstall: protobuf
    Found existing installation: protobuf 4.25.3
    Uninstalling protobuf-4.25.3:
      Successfully uninstalled protobuf-4.25.3
  Attempting uninstall: ppft
    Found existing installation: ppft 1.7.6.8
    Uninstalling ppft-1.7.6.8:
      Successfully uninstalled ppft-1.7.6.8
  Attempting uninstall: pox
    Found existing installation: pox 0.3.4
    Uninstalling pox-0.3.4:
      Successfully uninstalled pox-0.3.4
  Attempting uninstall: platformdirs
    Found existing installation: platformdirs 4.2.1
    Uninstalling platformdirs-4.2.1:
      Successfully uninstalled platformdirs-4.2.1
  Attempting uninstall: packaging
    Found existing installation: packaging 24.0
    Uninstalling packaging-24.0:
      Successfully uninstalled packaging-24.0
  Attempting uninstall: numpy
    Found existing installation: numpy 1.26.4
    Uninstalling numpy-1.26.4:
      Successfully uninstalled numpy-1.26.4
  Attempting uninstall: jmespath
    Found existing installation: jmespath 1.0.1
    Uninstalling jmespath-1.0.1:
      Successfully uninstalled jmespath-1.0.1
  Attempting uninstall: idna
    Found existing installation: idna 3.7
    Uninstalling idna-3.7:
      Successfully uninstalled idna-3.7
  Attempting uninstall: dill
    Found existing installation: dill 0.3.8
    Uninstalling dill-0.3.8:
      Successfully uninstalled dill-0.3.8
  Attempting uninstall: cloudpickle
    Found existing installation: cloudpickle 2.2.1
    Uninstalling cloudpickle-2.2.1:
      Successfully uninstalled cloudpickle-2.2.1
  Attempting uninstall: charset-normalizer
    Found existing installation: charset-normalizer 3.3.2
    Uninstalling charset-normalizer-3.3.2:
      Successfully uninstalled charset-normalizer-3.3.2
  Attempting uninstall: certifi
    Found existing installation: certifi 2024.2.2
    Uninstalling certifi-2024.2.2:
      Successfully uninstalled certifi-2024.2.2
  Attempting uninstall: attrs
    Found existing installation: attrs 23.2.0
    Uninstalling attrs-23.2.0:
      Successfully uninstalled attrs-23.2.0
  Attempting uninstall: requests
    Found existing installation: requests 2.31.0
    Uninstalling requests-2.31.0:
      Successfully uninstalled requests-2.31.0
  Attempting uninstall: referencing
    Found existing installation: referencing 0.35.1
    Uninstalling referencing-0.35.1:
      Successfully uninstalled referencing-0.35.1
  Attempting uninstall: python-dateutil
    Found existing installation: python-dateutil 2.9.0.post0
    Uninstalling python-dateutil-2.9.0.post0:
      Successfully uninstalled python-dateutil-2.9.0.post0
  Attempting uninstall: multiprocess
    Found existing installation: multiprocess 0.70.16
    Uninstalling multiprocess-0.70.16:
      Successfully uninstalled multiprocess-0.70.16
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 6.11.0
    Uninstalling importlib-metadata-6.11.0:
      Successfully uninstalled importlib-metadata-6.11.0
  Attempting uninstall: google-pasta
    Found existing installation: google-pasta 0.2.0
    Uninstalling google-pasta-0.2.0:
      Successfully uninstalled google-pasta-0.2.0
  Attempting uninstall: pathos
    Found existing installation: pathos 0.3.2
    Uninstalling pathos-0.3.2:
      Successfully uninstalled pathos-0.3.2
  Attempting uninstall: pandas
    Found existing installation: pandas 2.2.2
    Uninstalling pandas-2.2.2:
      Successfully uninstalled pandas-2.2.2
  Attempting uninstall: jsonschema-specifications
    Found existing installation: jsonschema-specifications 2023.12.1
    Uninstalling jsonschema-specifications-2023.12.1:
      Successfully uninstalled jsonschema-specifications-2023.12.1
  Attempting uninstall: docker
    Found existing installation: docker 7.0.0
    Uninstalling docker-7.0.0:
      Successfully uninstalled docker-7.0.0
  Attempting uninstall: botocore
    Found existing installation: botocore 1.29.165
    Uninstalling botocore-1.29.165:
      Successfully uninstalled botocore-1.29.165
  Attempting uninstall: s3transfer
    Found existing installation: s3transfer 0.6.2
    Uninstalling s3transfer-0.6.2:
      Successfully uninstalled s3transfer-0.6.2
  Attempting uninstall: jsonschema
    Found existing installation: jsonschema 4.22.0
    Uninstalling jsonschema-4.22.0:
      Successfully uninstalled jsonschema-4.22.0
  Attempting uninstall: boto3
    Found existing installation: boto3 1.26.83
    Uninstalling boto3-1.26.83:
      Successfully uninstalled boto3-1.26.83
  Attempting uninstall: sagemaker
    Found existing installation: sagemaker 2.219.0
    Uninstalling sagemaker-2.219.0:
      Successfully uninstalled sagemaker-2.219.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
awscli 1.31.7 requires botocore==1.33.7, but you have botocore 1.34.103 which is incompatible.
awscli 1.31.7 requires s3transfer<0.9.0,>=0.8.0, but you have s3transfer 0.10.1 which is incompatible.
mercury 2.3.7 requires boto3==1.26.83, but you have boto3 1.34.103 which is incompatible.
mercury 2.3.7 requires pyyaml==6.0, but you have pyyaml 6.0.1 which is incompatible.
numba 0.58.0 requires numpy<1.26,>=1.21, but you have numpy 1.26.4 which is incompatible.
sphinx 6.1.3 requires docutils<0.20,>=0.18, but you have docutils 0.16 which is incompatible.
tox 4.4.6 requires colorama>=0.4.6, but you have colorama 0.4.4 which is incompatible.
virtualenv 20.20.0 requires platformdirs<4,>=2.4, but you have platformdirs 4.2.1 which is incompatible.
Successfully installed PyYAML-6.0.1 attrs-23.2.0 boto3-1.34.103 botocore-1.34.103 certifi-2024.2.2 charset-normalizer-3.3.2 cloudpickle-2.2.1 dill-0.3.8 docker-7.0.0 google-pasta-0.2.0 idna-3.7 importlib-metadata-6.11.0 jmespath-1.0.1 jsonschema-4.22.0 jsonschema-specifications-2023.12.1 multiprocess-0.70.16 numpy-1.26.4 packaging-24.0 pandas-2.2.2 pathos-0.3.2 platformdirs-4.2.1 pox-0.3.4 ppft-1.7.6.8 protobuf-4.25.3 psutil-5.9.8 python-dateutil-2.9.0.post0 pytz-2024.1 referencing-0.35.1 requests-2.31.0 rpds-py-0.18.1 s3transfer-0.10.1 sagemaker-2.219.0 schema-0.7.7 six-1.16.0 smdebug-rulesconfig-1.0.1 tblib-3.0.0 tqdm-4.66.4 tzdata-2024.1 urllib3-1.26.18 zipp-3.18.1

Import Libraries

The model used in this example notebook is the bring your own time series model which uses a model.pth and inference.py to handle the model invocation.

[3]

sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /home/zicanl/.config/sagemaker/config.yaml

Compress the model package

[4]

code/
code/inference.py
code/requirements.txt
model.pth

Upload model to S3. The example model is the linear regression model from darts

[5]

[6]

----------!

Verify the endpoint

[7]

{'instances': {'target': [1008.89, 1008.76, 1008.66],
,  'start': '2020-01-01 16:20:00',
,  'dynamic_feat': [[0.23, 0.21, 0.19, 0.16, 0.13, 0.08, 0.0, 0.0, 0.0],
,   [0.71, 0.75, 0.73, 0.37, 0.33, 0.34, 0.19, 0.03, 0.11]]}}

[8]

{'predictions': {'mean': [1008.5656193655171, 1008.511339974418, 1008.4186285678663, 1008.3432658290359, 1008.2426915722609, 1008.1615810491433, 1008.1274301835906, 1008.0939011448609, 1008.0749872974072, 1008.0284006424769]}}

Time Series Explainability

Import Components

Import the components needed to make a TSX call.

[9]

Set Configurations

Here is an example of content_template and record_template for time series for more information: please check: https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-processing-job-data-format-time-series-request-jsonlines.html

[10]

Asymmetric Shapley value

Our time series forecasting explainability algorithm hinges on the application of the asymmetric Shapley values (ASV) from the theory of cooperative games. The ASV is a modification of the well-known Shapley value (e.g SHAP) that discards the symmetry axiom, but retains the efficiency exioms (i.e. attributions sum up to the predictions). Coalitions of features are generated based on a given probability distribution over feature permutations (rather than over subsets in the case of the Shapley value). In the case of time series, the distributions we use puts zero probability on permutations of features that do not respect the temporal dependencies, i.e. that have "holes".

References:

Our main scientific reference is https://arxiv.org/abs/1910.06358. We scale the approach of the paper to include also static covariates, related time series and implement a stochastic estimator for efficiency.
A very useful math reference is Probabilistic values by RJ Weber; specifically, section 8 about random-order values (these are the same mathematical construction of ASV).

Create `AsymmetricShapleyValueConfig`

An AsymmetricShapleyValueConfig is used to configure the algorithm Clarify uses for time series explainability. It takes the following arguments:

direction: direction of explanation to be used. Available explanation types are "chronological", "anti_chronological", "bidirectional". The cronological direction highlights the effect of older timesteps over more recent one, while the anti-chronological direction higlights the effect of timesteps closer to the forecasting. Bidirectional is a combination of the previous two modes.
granularity: Granularity of explanation to be used. Available granularities are "timewise" and "fine_grained". The first granularity is fast and computes the attribution of individual timesteps toward the forecast, not making distinctions of related time series. The fine-grained mode is slower, but computes an attribution for every timestep and every feature dynamic, distinguishing between related and target TS.
num_samples: Number of samples to be used in the Asymmetric Shapley Value forecasting algorithm. Only applicable when using "fine_grained" explanations. This represents the number of permutations sampled for computing the ASV.
baseline: baseline configuration (dictionary). The baseline config is used to replace out-of-coalition values for the corresponding datasets (also known as background data). For temporal data (target time series, related time series), the baseline value types are "zero", where all out-of-coalition values will be replaced with 0.0, or "mean", all out-of-coalition values will be replaced with the average of a time series. For static data(static covariates), a baseline value for each covariate should be provided for each possible item_id. An example config follows, where item1 and item2 are item ids::

{
 "target_time_series": "zero",
 "related_time_series": "zero",
 "static_covariates":
  "item1": [1, 1],
  "item2": [0, 1],
 }
}

The notebook sets explanation_direction and granularity as variables for later reference.

[11]

Only then does the notebook create the AsymmetricShapleyValueConfig object.

[12]

Create `TimeSeriesDataConfig`

A TimeSeriesDataConfig object is used to configure data I/O settings specific to TSX. It takes the following arguments:

target_time_series: A string or a zero-based integer index. Used to locate the target time series in the shared input dataset. If this parameter is a string, then all other parameters must also be strings or lists of strings. If this parameter is an int, then all others must be ints or lists of ints.
item_id: A string or a zero-based integer index. Used to locate item id in the shared input dataset.
timestamp: A string or a zero-based integer index. Used to locate timestamp in the shared input dataset.
related_time_series: Optional. An array of strings or array of zero-based integer indices. Used to locate all related time series in the shared input dataset (if present).
static_covariates: Optional. An array of strings or array of zero-based integer indices. Used to locate all item metadata fields in the shared input dataset (if present).
dataset_format: Optional. A string which describes the format of the data files provided for analysis. Should only be provided when dataset is in JSON format. Currently, we support columns and timestamp_records where example mock data files ts_cols.json and time_series_mock_data.json are provided respectively.

This TimeSeriesDataConfig helps the container to parse the data needed for the analysis. Any additional data columns will be excluded if not providing corresponding Jmes_path to locate them.

[13]

Create `TimeSeriesModelConfig`

A TimeSeriesModelConfig is used to configure model settings specific to TSX. At the moment it has only one argument:

forecast: JMESPath expression to extract the forecast result.

[14]

Create DataConfig

General information about the dataset the TimeSeries model uses is provided to DataConfig. Here, we are providing where to retrieve the dataset, where to output the explainability job results, what format the dataset is in, and our TSX specific data settings.

[15]

Create ModelConfig

With ModelConfig is configured here, Clarify will deploy the specified model to a new endpoint.

[16]

It is also possible to specify an existing endpoint for Clarify to use with the following modifications to the ModifyConfig call:

Omitting model_name, instance_count, instance_type, and endpoint_name_prefix.
Provided endpoint_name.

Setup Processor

Create the Processor object that will setup the explainability job.

[17]

Run Explainability Call

[18]

INFO:sagemaker:Creating processing-job with name clarify-tsx-byom-job-demo-2024-05-10-23-11-14-749

................................WARNING:root:logging.conf not found when configuring logging, using default logging configuration.
INFO:sagemaker-clarify-processing:Starting SageMaker Clarify Processing job
INFO:analyzer.data_loading.data_loader_util:Analysis config path: /opt/ml/processing/input/config/analysis_config.json
INFO:analyzer.data_loading.data_loader_util:Analysis result path: /opt/ml/processing/output
INFO:analyzer.data_loading.data_loader_util:This host is algo-1.
INFO:analyzer.data_loading.data_loader_util:This host is the leader.
INFO:analyzer.data_loading.data_loader_util:Number of hosts in the cluster is 1.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/05/10 23:16:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
#015[Stage 0:>                                                          (0 + 8) / 8]#015#015                                                                                #015#015[Stage 4:===================================================>       (7 + 1) / 8]#015#015                                                                                #015INFO:analyzer.predictor.managed_endpoint:Using endpoint timeseries-byom-endpoint-1715382341-39c2
INFO:analyzer.predictor.managed_endpoint:Checking endpoint status:
Legend:
(OutOfService: x, Creating: -, Updating: -, InService: !, RollingBack: <, Deleting: o, Failed: *)
INFO:analyzer.predictor.managed_endpoint:Endpoint is in service after 0 seconds
INFO:analyzer.predictor.predictor:Stop using endpoint: timeseries-byom-endpoint-1715382341-39c2
INFO:analyzer.predictor.managed_endpoint:Model endpoint delivered 14.42070 requests per second and a total of 1 requests over 0 seconds
#015[Stage 12:>                                                         (0 + 1) / 1]#015#015                                                                                #015INFO:sagemaker-clarify-processing:Wrote 2 lines to /opt/ml/processing/output/asymmetric_shapley_value/fine_grained_chronological/out.jsonl
INFO:sagemaker-clarify-processing:Collected analyses: 
{'version': '1.0', 'explanations': {'asymmetric_shapley_value': {'direction': 'chronological', 'granularity': 'fine_grained', 'explanation_results_path': 's3://sagemaker-us-west-2-678264136642/byom/output_byom/asymmetric_shapley_value/fine_grained_chronological/out.jsonl'}}}
INFO:analyzer.utils.system_util:exit_message: Completed: SageMaker XAI Analyzer ran successfully
INFO:py4j.clientserver:Closing down clientserver connection

Analysis Config

Retrieve Config From s3

[19]

Display Config

[20]

{'dataset_type': 'application/json',
 'headers': ['item_id', 'timestamp', 'p_mbar', 'rain_mm', 'T_degC'],
 'methods': {'asymmetric_shapley_value': {'baseline': {'related_time_series': 'zero', 'target_time_series': 'zero'}, 'direction': 'chronological', 'granularity': 'fine_grained', 'num_samples': 2},
             'report': {'name': 'report', 'title': 'Analysis Report'}},
 'predictor': {'accept_type': 'application/json',
               'content_template': '{"instances": $record}',
               'content_type': 'application/json',
               'endpoint_name': 'timeseries-byom-endpoint-1715382341-39c2',
               'record_template': '{"start": $start_time, "target": $target_time_series, "dynamic_feat": $related_time_series}',
               'time_series_predictor_config': {'forecast': 'predictions.mean'}},
 'time_series_data_config': {'dataset_format': 'timestamp_records',
                             'item_id': '[].item_id',
                             'related_time_series': ['[].rain_mm', '[].T_degC'],
                             'target_time_series': '[].p_mbar',
                             'timestamp': '[].timestamp'}}

Explainability Results

Retrieve Results From s3

[21]

Display Results

[22]

[{'explanations': [{'feature_name': 'p_mbar',
                    'scores': [161.99614546052845,
                               266.016195372142,
                               171.9754724539663,
                               -171.07476604383538,
                               -313.6433436163125,
                               -239.94510772430738,
                               -85.12952959394124,
                               -57.90547536857707,
                               -207.1411908350123,
                               -379.59127645091314],
                    'timestamp': '2020-01-01 16:20:00'},
                   {'feature_name': 'p_mbar',
                    'scores': [-242.53171819948116,
                               -178.51561960848613,
                               20.504771980246744,
                               283.65833398610687,
                               116.71883538707854,
                               -102.86798765521405,
                               -146.4874269981686,
                               8.14449669685348,
                               194.95205623712206,
                               223.11109391304643],
                    'timestamp': '2020-01-01 16:30:00'},
                   {'feature_name': 'p_mbar',
                    'scores': [1010.1941985733024,
                               795.0553673348827,
                               642.8274810307071,
                               682.7588591520664,
                               942.0005592421893,
                               1034.9501875444607,
                               882.5456028112928,
                               669.6075075328768,
                               607.9790225048051,
                               718.9410031751296],
                    'timestamp': '2020-01-01 16:40:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 16:20:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 16:30:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 16:40:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 16:50:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:00:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:10:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:20:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:30:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:40:00'},
                   {'feature_name': 'T_degC',
                    'scores': [-0.08714719958848605,
                               0.07035004670223088,
                               0.057821527911926296,
                               0.09521297487265201,
                               0.2316662672997154,
                               0.35016081212063455,
                               0.5944301743015217,
                               0.7031256598847904,
                               0.7530252031138502,
                               0.8773061468182846],
                    'timestamp': '2020-01-01 16:20:00'},
                   {'feature_name': 'T_degC',
                    'scores': [0.25522850377080886,
                               -0.023008325401406182,
                               0.08935537901516,
                               0.07432224632441375,
                               -0.02099284843376381,
                               0.03808045819243944,
                               -0.12744614800971021,
                               -0.044815576172311467,
                               0.007387391643192132,
                               -0.07124181014808073],
                    'timestamp': '2020-01-01 16:30:00'},
                   {'feature_name': 'T_degC',
                    'scores': [-0.28369788212489766,
                               0.0005212027526795282,
                               -0.15772370675017555,
                               -0.0051055361683438605,
                               -0.03238935407966892,
                               -0.21258560003138882,
                               -0.15587474690755698,
                               -0.2650242940226235,
                               -0.12797639289135532,
                               -0.022552261672558416],
                    'timestamp': '2020-01-01 16:40:00'},
                   {'feature_name': 'T_degC',
                    'scores': [0.10025229473171748,
                               -0.20299805713239039,
                               0.0036295981972216396,
                               -0.15789278404247398,
                               -0.04784570674723909,
                               -0.011186161681962403,
                               -0.16199305896884653,
                               -0.120134946101075,
                               -0.23073982878611332,
                               -0.13886565144207452],
                    'timestamp': '2020-01-01 16:50:00'},
                   {'feature_name': 'T_degC',
                    'scores': [-0.05089328385463432,
                               0.016269546530566004,
                               -0.18819813736070046,
                               -0.22929538149924156,
                               -0.37628273091195297,
                               -0.4463468139155111,
                               -0.47193314366836603,
                               -0.6292941653790649,
                               -0.7734106313403117,
                               -1.0187577778201558],
                    'timestamp': '2020-01-01 17:00:00'},
                   {'feature_name': 'T_degC', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:10:00'},
                   {'feature_name': 'T_degC', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:20:00'},
                   {'feature_name': 'T_degC', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:30:00'},
                   {'feature_name': 'T_degC', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:40:00'}],
  'item_id': 'sanjose',
  'offset': [47.44921461578576,
             100.62646867069782,
             152.21050681227143,
             191.15803860751862,
             234.3224739726919,
             284.7873864453612,
             330.3183057947489,
             367.38654787214057,
             393.2080365560922,
             423.5261096178664]},
 {'explanations': [{'feature_name': 'p_mbar',
                    'scores': [162.6069955165382,
                               267.01928101581973,
                               172.62395224761923,
                               -171.7198494816089,
                               -314.82602023784847,
                               -240.84988531686048,
                               -85.45053339173353,
                               -58.12382354452666,
                               -207.92227243213168,
                               -381.02262749832033],
                    'timestamp': '2020-01-01 16:20:00'},
                   {'feature_name': 'p_mbar',
                    'scores': [-243.41488016208197,
                               -179.16567151154757,
                               20.57943864570069,
                               284.6912555883247,
                               117.14385870567043,
                               -103.24257409916794,
                               -147.02085052101535,
                               8.174154300982877,
                               195.66196025247172,
                               223.92353705673543],
                    'timestamp': '2020-01-01 16:30:00'},
                   {'feature_name': 'p_mbar',
                    'scores': [1041.9487078011975,
                               820.0471887434583,
                               663.0341612978904,
                               704.2207449201605,
                               971.6114652376746,
                               1067.4828781174288,
                               910.2875978931187,
                               690.6559928706965,
                               627.0902740916401,
                               741.5402407790272],
                    'timestamp': '2020-01-01 16:40:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 16:20:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 16:30:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 16:40:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 16:50:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:00:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:10:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:20:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:30:00'},
                   {'feature_name': 'rain_mm', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:40:00'},
                   {'feature_name': 'T_degC',
                    'scores': [-0.024949399882189027,
                               0.020140537563975158,
                               0.016553743877977922,
                               0.02725855329015303,
                               0.06632381039628399,
                               0.10024765185710294,
                               0.17017960635243412,
                               0.2012980719831603,
                               0.21558382831082668,
                               0.25116425977461176],
                    'timestamp': '2020-01-01 16:20:00'},
                   {'feature_name': 'T_degC',
                    'scores': [0.07942795760502008,
                               -0.007160267241090423,
                               0.027807690564884524,
                               0.023129329769005835,
                               -0.006533044118384623,
                               0.011850764997632268,
                               -0.03966166431837337,
                               -0.013946756070254196,
                               0.0022989808018110125,
                               -0.022170687805441958],
                    'timestamp': '2020-01-01 16:30:00'},
                   {'feature_name': 'T_degC',
                    'scores': [-0.0873837358444689,
                               0.0001605392446890619,
                               -0.0485815636825464,
                               -0.0015725913092410337,
                               -0.00997646771224936,
                               -0.06547995275241192,
                               -0.048012052844796926,
                               -0.08163195554277536,
                               -0.039418888949739994,
                               -0.006946477224005321],
                    'timestamp': '2020-01-01 16:40:00'},
                   {'feature_name': 'T_degC',
                    'scores': [0.015455562104534692,
                               -0.03129553380813377,
                               0.0005595630553898445,
                               -0.02434180420652865,
                               -0.007376213123507114,
                               -0.0017245332592210616,
                               -0.024973929924499316,
                               -0.018520804190529816,
                               -0.03557239027111336,
                               -0.02140845459734919],
                    'timestamp': '2020-01-01 16:50:00'},
                   {'feature_name': 'T_degC',
                    'scores': [-0.006968789905386075,
                               0.0022277802303278804,
                               -0.0257698694310875,
                               -0.03139729290239757,
                               -0.051524191369708205,
                               -0.061118028461464746,
                               -0.06462155079282184,
                               -0.08616891061205934,
                               -0.10590270055695328,
                               -0.1394979529795819],
                    'timestamp': '2020-01-01 17:00:00'},
                   {'feature_name': 'T_degC', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:10:00'},
                   {'feature_name': 'T_degC', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:20:00'},
                   {'feature_name': 'T_degC', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:30:00'},
                   {'feature_name': 'T_degC', 'scores': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'timestamp': '2020-01-01 17:40:00'}],
  'item_id': 'seattle',
  'offset': [47.44921461578576,
             100.62646867069782,
             152.21050681227143,
             191.15803860751862,
             234.3224739726919,
             284.7873864453612,
             330.3183057947489,
             367.38654787214057,
             393.2080365560922,
             423.5261096178664]}]

Clean Up

Remove downloaded/installed files and deployed resources as necessary.

[23]

{'ResponseMetadata': {'RequestId': 'ab60df21-91b8-4ccd-9c45-c8115eb4c454',
,  'HTTPStatusCode': 200,
,  'HTTPHeaders': {'x-amzn-requestid': 'ab60df21-91b8-4ccd-9c45-c8115eb4c454',
,   'content-type': 'application/x-amz-json-1.1',
,   'date': 'Fri, 10 May 2024 23:17:38 GMT',
,   'content-length': '0'},
,  'RetryAttempts': 0}}

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.