🍿 Snowflake Trail - Step 1: Demo Setup

Creating a dummy data ingestion and transformation project with some intentional errors and anomalies that we can monitor and alert on.

• Warehouse
• Internal Stage
• Pipe
• Dynamic Tables
• Stream and Task graph
• nested SQL & Python Procedures & Functions

Setup Details:

• 1st Task creates a new csv-file in the internal Stage
• 2nd Task creates a broken csv-file into the internal stage
• 3rd Task manually refreshes the Pipe to load the 2 new files from the source Stage into a target table
• Pipe copy of the broken csv-file will fail -> logging file ingestion error to the event table
• the Stream on the target table will catch the new data from the successful file ingestion
• Stream triggers the root Task to run the graph
• child Tasks call procedures and functions, which may fail at random -> logging Procedure and Task errors
• parallel child Task will change the name of the source table of a Dynamic Table and then manually refresh the Dynamic Table -> logging Dynamic Table refresh error

-> manually run the Trigger Task via the UI or SQL to create new logs

Now you can intentionally trigger a range of event logs from

Task run, Pipe copy, Dynamic Table refresh, Snowpark Procedure and Function calls

to test or demo the logging to the event table.

continue with the

Snowflake Trail - Step 2: Observability Setup Notebook

to add a comprehensive observability-layer to this pipeline.