Amazon Web Services OpenChat Streaming Tgi

OpenChat Streaming Tgi

data-scienceinferencearchivedamazon-sagemaker-examplesreinforcement-learningmachine-learningWorkshopsawsexamplesdeep-learningsagemakerjupyter-notebooktrainingmlops

alph-notebooks/amazon-sagemaker-examples / OpenChat-streaming_tgi.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

Real-Time Inference Streaming with HuggingFace TGI and the OpenChat Model in SageMaker

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This tutorial will guide you through using the Hugging Face Large Language Model Inference Container on Amazon SageMaker to deploy OpenChat, which is a conversational AI assistant that uses large language models to engage in open-ended dialogue and assist with a variety of tasks. You'll run inference streaming with this container, which is powered by Text Generation Inference (TGI) - an open-source, purpose-built solution for deploying and serving LLMs.

TGI enables high-performance text generation by leveraging Tensor Parallelism and dynamic batching. It supports the most popular open-source LLMs, including StarCoder, BLOOM, GPT-NeoX, Llama, and T5.

Please make sure the following permission granted before running the notebook:

S3 bucket push access
SageMaker access

Introduction

OpenChat is an innovative library of open-source language models, fine-tuned with C-RLFT - a strategy inspired by offline reinforcement learning.
The models learn from mixed-quality data without preference labels, delivering exceptional performance on par with ChatGPT, even with a 7B model which can be run on a consumer GPU (e.g. RTX 3090).
Despite their simple approach, they are committed to developing a high-performance, commercially viable, open-source large language model, and continue to make significant strides toward this vision.

For more information:

Step 1: Installing and importing dependencies

We begin by importing the necessary libraries and configuring several global variables using the boto3 library.

[2]

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Note: you may need to restart the kernel to use updated packages.

[13]

Deploying Hugging Face models in Amazon SageMaker is slightly different from deploying regular Hugging Face models. To do this, you need to first retrieve the container URI and provide it to your HuggingFaceModel model class, along with an image_uri that points to the image you want to use.

To retrieve the URI for the new Hugging Face Large Language Model (LLM) Deep Learning Containers (DLC) in Amazon SageMaker, you can use the get_huggingface_llm_image_uri() method provided by the SageMaker SDK. This method allows you to retrieve the URI for the desired Hugging Face LLM DLC based on the specified backend, session, AWS region, and version.

By using this method, you can easily obtain the necessary container URI to deploy your Hugging Face LLM model in Amazon SageMaker, without having to manually look up and manage the URI information. You can find the available versions here.

[4]

llm image uri: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-tgi-inference:2.1.1-tgi1.4.2-gpu-py310-cu121-ubuntu22.04

[6]

Step 2: Deploy OpenChat using the TGI image

To deploy the HuggingFaceModel on Amazon SageMaker, we'll use the deploy method. We'll be deploying the model on the ml.g5.12xlarge instance type, as specified earlier. Details of the below environment variables are described at here

[7]

[8]

----------!

Step 3: Initiate streaming inference requests to the deployed SageMaker model endpoint

[9]

[10]

[11]

[14]

# How to study GenAI

GenAI is a complex and rapidly evolving field, and studying it requires a multidisciplinary approach. Here are some steps you can take to study GenAI effectively:

1. Understand the basics: Start by learning the fundamentals of artificial intelligence, machine learning, and natural language processing. This will provide you with a solid foundation for understanding GenAI.

2. Learn programming languages: Familiarize yourself with programming languages such as Python, which is commonly used in AI and machine learning. This will enable you to work with AI tools and frameworks.

3. Study AI models and algorithms: Learn about different AI models and algorithms, such as neural networks, deep learning, and reinforcement learning. This will help you understand how AI systems work and how they can be applied to various tasks.

4. Explore GenAI applications: Study the different applications of GenAI, such as natural language understanding, computer vision, and robotics. This will help you understand the practical implications of GenAI and how it can be used to solve real-world problems.

5. Keep up with research: Stay up-to-date with the latest research in GenAI by following relevant journals, conferences, and online resources. This will help you stay informed about the latest developments in the field.

6. Practice with hands-on projects: Apply your knowledge by working on hands-on projects, such as building a chatbot or creating a machine learning model. This will help you gain practical experience and improve your skills.

7. Collaborate with others: Join online forums, attend workshops, and participate in hackathons to collaborate with other GenAI enthusiasts. This will help you learn from others and expand your network.

8. Develop critical thinking skills: Develop your critical thinking skills to evaluate the effectiveness and ethical implications of GenAI technologies. This will help you become a responsible and informed AI practitioner.

By following these steps, you can effectively study GenAI and gain the skills and knowledge needed to succeed in this rapidly evolving field.

##### GenAI

GenAI is a complex and rapidly evolving field, and studying it requires a multidisciplinary approach. Here are some steps you can take to study GenAI effectively:

By following these steps, you can effectively study GenAI and gain the skills and knowledge needed to succeed in this rapidly evolving field.

##### GenAI

GenAI is a complex and rapidly evolving field, and studying it requires a multidisciplinary approach. Here are some steps you can take to study GenAI effectively:

By following these steps, you can effectively study GenAI and gain the skills and knowledge needed to succeed in this rapidly evolving field.

##### GenAI

GenAI is a complex and rapidly evolving field, and studying it requires a multidisciplinary approach. Here are some steps you can take to study GenAI effectively:

By following these steps, you can effectively study GenAI and gain the skills and knowledge needed to succeed in this rapidly evolving field.

##### GenAI

GenAI is a complex and rapidly evolving field, and studying it requires a multidisciplinary approach. Here are some steps you can take to study GenAI effectively:

5. Keep up with research: Stay up-to-date with the latest research in GenAI by following relevant journals, conferences, and online resources. This will help you

Clean up

[ ]

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.