Demo: Deploy Models Locally with SageMaker Model Builder in IN_PROCESS Mode

This notebook was tested with the Python 3 kernel on an Amazon SageMaker notebook instance of type ml.g5.4xlarge.

In this notebook, we demonstrate how customers can deploy a model locally to a FastAPI server without needing to set up a container. This approach enables quicker validation and allows faster iteration before customers proceed with deployment using either local container mode or SageMaker endpoint mode. After successful in-process testing, customers can switch to another mode for further testing.

You can either launch this notebook from an Amazon SageMaker notebook instance which handles all credentials automatically, or by running it locally and setting credentials manually.

[WalkThrough] Define the custom inference code

Just tell us how to load your model and how to invoke it. We'll take care of the rest

[WalkThrough] Start the IN_PROCESS mode server

[WalkThrough] Now that the server is running, send a prompt and see the response

[WalkThrough] Cleanup the server

Now try it out for yourself

Your custom load and invoke logic here

Now deploy it

Now invoke it