[Gemma 2]Using With SGLang
Copyright 2024 Google LLC.
Interacting with Gemma 2 using SGLang
Gemma is a family of lightweight, state-of-the-art open-source language models from Google. Built from the same research and technology used to create the Gemini models, Gemma models are text-to-text, decoder-only large language models (LLMs), available in English, with open weights, pre-trained variants, and instruction-tuned variants. Gemma models are well-suited for various text-generation tasks, including question-answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop, or your cloud infrastructure, democratizing access to state-of-the-art AI models and helping foster innovation for everyone.
SGLang is a serving framework for Large Language models. It offers a fast backend runtime and a flexible front end language allowing you to control and customize model interactions.
In this notebook, you will learn how to prompt Gemma 2 model in various ways using the SGLang http server, backend runtime and frontend language in a Google Colab environment.
Setup
Select the Colab runtime
To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:
- In the upper-right of the Colab window, select ▾ (Additional connection options).
- Select Change runtime type.
- Under Hardware accelerator, select T4 GPU.
Setup Hugging Face
Before you dive into the tutorial, let's get you set up with Hugging face:
-
Hugging Face Account: If you don't already have one, you can create a free Hugging Face account by clicking here.
-
Hugging Face Token: Generate a Hugging Face access (preferably
writepermission) token by clicking here. You'll need this token later in the tutorial.
Once you've completed these steps, you're ready to move on to the next section where you'll set up environment variables in your Colab environment.
Configure your HF token
Add your Hugging Face token to the Colab Secrets manager to securely store it.
- Open your Google Colab notebook and click on the 🔑 Secrets tab in the left panel.

- Create a new secret with the name
HF_TOKEN. - Copy/paste your HF token key into the Value input box of
HF_TOKEN. - Toggle the button on the left to allow notebook access to the secret.
Install dependencies
First, you must install the necessary packages for SGLang.
Collecting sglang[all]
Downloading sglang-0.3.5-py3-none-any.whl.metadata (21 kB)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (2.32.3)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (4.66.6)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (1.26.4)
Requirement already satisfied: IPython in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (7.34.0)
Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (75.1.0)
Collecting jedi>=0.16 (from IPython->sglang[all])
Downloading jedi-0.19.1-py2.py3-none-any.whl.metadata (22 kB)
Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (4.4.2)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (0.7.5)
Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (5.7.1)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (3.0.48)
Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (2.18.0)
Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (0.2.0)
Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (0.1.7)
Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (4.9.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->sglang[all]) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->sglang[all]) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->sglang[all]) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->sglang[all]) (2024.8.30)
Collecting anthropic>=0.20.0 (from sglang[all])
Downloading anthropic-0.39.0-py3-none-any.whl.metadata (22 kB)
Collecting litellm>=1.0.0 (from sglang[all])
Downloading litellm-1.51.3-py3-none-any.whl.metadata (32 kB)
Requirement already satisfied: openai>=1.0 in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (1.52.2)
Collecting tiktoken (from sglang[all])
Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (2.5.0+cu121)
Collecting vllm==0.6.3.post1 (from sglang[all])
Downloading vllm-0.6.3.post1-cp38-abi3-manylinux1_x86_64.whl.metadata (10 kB)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (5.9.5)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (0.2.0)
Requirement already satisfied: py-cpuinfo in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (9.0.0)
Collecting transformers>=4.45.2 (from vllm==0.6.3.post1->sglang[all])
Downloading transformers-4.46.1-py3-none-any.whl.metadata (44 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 4.1 MB/s eta 0:00:00
Requirement already satisfied: tokenizers>=0.19.1 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (0.19.1)
Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (3.20.3)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (3.10.10)
Collecting uvicorn[standard] (from vllm==0.6.3.post1->sglang[all])
Downloading uvicorn-0.32.0-py3-none-any.whl.metadata (6.6 kB)
Requirement already satisfied: pydantic>=2.9 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (2.9.2)
Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (10.4.0)
Requirement already satisfied: prometheus-client>=0.18.0 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (0.21.0)
Collecting prometheus-fastapi-instrumentator>=7.0.0 (from vllm==0.6.3.post1->sglang[all])
Downloading prometheus_fastapi_instrumentator-7.0.0-py3-none-any.whl.metadata (13 kB)
Collecting lm-format-enforcer==0.10.6 (from vllm==0.6.3.post1->sglang[all])
Downloading lm_format_enforcer-0.10.6-py3-none-any.whl.metadata (16 kB)
Collecting outlines<0.1,>=0.0.43 (from vllm==0.6.3.post1->sglang[all])
Downloading outlines-0.0.46-py3-none-any.whl.metadata (15 kB)
Requirement already satisfied: typing-extensions>=4.10 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (4.12.2)
Requirement already satisfied: filelock>=3.10.4 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (3.16.1)
Collecting partial-json-parser (from vllm==0.6.3.post1->sglang[all])
Downloading partial_json_parser-0.2.1.1.post4-py3-none-any.whl.metadata (6.2 kB)
Requirement already satisfied: pyzmq in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (24.0.1)
Collecting msgspec (from vllm==0.6.3.post1->sglang[all])
Downloading msgspec-0.18.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.9 kB)
Collecting gguf==0.10.0 (from vllm==0.6.3.post1->sglang[all])
Downloading gguf-0.10.0-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (8.5.0)
Collecting mistral-common>=1.4.4 (from mistral-common[opencv]>=1.4.4->vllm==0.6.3.post1->sglang[all])
Downloading mistral_common-1.4.4-py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (6.0.2)
Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (0.8.0)
Collecting compressed-tensors==0.6.0 (from vllm==0.6.3.post1->sglang[all])
Downloading compressed_tensors-0.6.0-py3-none-any.whl.metadata (6.8 kB)
Collecting ray>=2.9 (from vllm==0.6.3.post1->sglang[all])
Downloading ray-2.38.0-cp310-cp310-manylinux2014_x86_64.whl.metadata (17 kB)
Collecting nvidia-ml-py (from vllm==0.6.3.post1->sglang[all])
Downloading nvidia_ml_py-12.560.30-py3-none-any.whl.metadata (8.6 kB)
Collecting torch (from sglang[all])
Downloading torch-2.4.0-cp310-cp310-manylinux1_x86_64.whl.metadata (26 kB)
Collecting torchvision==0.19 (from vllm==0.6.3.post1->sglang[all])
Downloading torchvision-0.19.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.0 kB)
Collecting xformers==0.0.27.post2 (from vllm==0.6.3.post1->sglang[all])
Downloading xformers-0.0.27.post2-cp310-cp310-manylinux2014_x86_64.whl.metadata (1.0 kB)
Collecting fastapi!=0.113.*,!=0.114.0,>=0.107.0 (from vllm==0.6.3.post1->sglang[all])
Downloading fastapi-0.115.4-py3-none-any.whl.metadata (27 kB)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->sglang[all]) (1.13.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->sglang[all]) (3.4.2)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->sglang[all]) (3.1.4)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->sglang[all]) (2024.10.0)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch->sglang[all])
Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch->sglang[all])
Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch->sglang[all])
Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu12==9.1.0.70 (from torch->sglang[all])
Downloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch->sglang[all])
Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch->sglang[all])
Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-curand-cu12==10.3.2.106 (from torch->sglang[all])
Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch->sglang[all])
Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch->sglang[all])
Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-nccl-cu12==2.20.5 (from torch->sglang[all])
Downloading nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch->sglang[all])
Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
Collecting triton==3.0.0 (from torch->sglang[all])
Downloading triton-3.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)
Collecting interegular>=0.3.2 (from lm-format-enforcer==0.10.6->vllm==0.6.3.post1->sglang[all])
Downloading interegular-0.3.3-py37-none-any.whl.metadata (3.0 kB)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from lm-format-enforcer==0.10.6->vllm==0.6.3.post1->sglang[all]) (24.1)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->sglang[all]) (12.6.77)
Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (3.7.1)
Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (0.27.2)
Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (0.6.1)
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (1.3.1)
Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->IPython->sglang[all]) (0.8.4)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from litellm>=1.0.0->sglang[all]) (8.1.7)
Requirement already satisfied: jsonschema<5.0.0,>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from litellm>=1.0.0->sglang[all]) (4.23.0)
Collecting python-dotenv>=0.2.0 (from litellm>=1.0.0->sglang[all])
Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->IPython->sglang[all]) (0.7.0)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->IPython->sglang[all]) (0.2.13)
Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->sglang[all]) (2024.9.11)
Collecting decord (from sglang[all])
Downloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl.metadata (422 bytes)
Collecting hf-transfer (from sglang[all])
Downloading hf_transfer-0.1.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (0.24.7)
Requirement already satisfied: orjson in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (3.10.10)
Collecting python-multipart (from sglang[all])
Downloading python_multipart-0.0.17-py3-none-any.whl.metadata (1.8 kB)
Collecting torchao (from sglang[all])
Downloading torchao-0.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)
Collecting uvloop (from sglang[all])
Downloading uvloop-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)
Collecting zmq (from sglang[all])
Downloading zmq-0.0.0.zip (2.2 kB)
Preparing metadata (setup.py) ... done
Collecting modelscope (from sglang[all])
Downloading modelscope-1.19.2-py3-none-any.whl.metadata (40 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 kB 3.2 MB/s eta 0:00:00
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->anthropic>=0.20.0->sglang[all]) (1.2.2)
Collecting starlette<0.42.0,>=0.40.0 (from fastapi!=0.113.*,!=0.114.0,>=0.107.0->vllm==0.6.3.post1->sglang[all])
Downloading starlette-0.41.2-py3-none-any.whl.metadata (6.0 kB)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->anthropic>=0.20.0->sglang[all]) (1.0.6)
Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->anthropic>=0.20.0->sglang[all]) (0.14.0)
Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->vllm==0.6.3.post1->sglang[all]) (3.20.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->sglang[all]) (3.0.2)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.0.0->sglang[all]) (24.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.0.0->sglang[all]) (2024.10.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.0.0->sglang[all]) (0.35.1)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.0.0->sglang[all]) (0.20.0)
Collecting tiktoken (from sglang[all])
Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: opencv-python-headless<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from mistral-common[opencv]>=1.4.4->vllm==0.6.3.post1->sglang[all]) (4.10.0.84)
Collecting lark (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])
Downloading lark-1.2.2-py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (1.6.0)
Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (3.1.0)
Collecting diskcache (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])
Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)
Requirement already satisfied: numba in /usr/local/lib/python3.10/dist-packages (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (0.60.0)
Collecting datasets (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])
Downloading datasets-3.1.0-py3-none-any.whl.metadata (20 kB)
Collecting pycountry (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])
Downloading pycountry-24.6.1-py3-none-any.whl.metadata (12 kB)
Collecting pyairports (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])
Downloading pyairports-2.1.1-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.9->vllm==0.6.3.post1->sglang[all]) (0.7.0)
Requirement already satisfied: pydantic-core==2.23.4 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.9->vllm==0.6.3.post1->sglang[all]) (2.23.4)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==0.6.3.post1->sglang[all]) (1.1.0)
Requirement already satisfied: aiosignal in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==0.6.3.post1->sglang[all]) (1.3.1)
Requirement already satisfied: frozenlist in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==0.6.3.post1->sglang[all]) (1.5.0)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.45.2->vllm==0.6.3.post1->sglang[all]) (0.4.5)
Collecting tokenizers>=0.19.1 (from vllm==0.6.3.post1->sglang[all])
Downloading tokenizers-0.20.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->vllm==0.6.3.post1->sglang[all]) (2.4.3)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->vllm==0.6.3.post1->sglang[all]) (6.1.0)
Requirement already satisfied: yarl<2.0,>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->vllm==0.6.3.post1->sglang[all]) (1.17.0)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->vllm==0.6.3.post1->sglang[all]) (4.0.3)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->sglang[all]) (1.3.0)
Collecting httptools>=0.5.0 (from uvicorn[standard]->vllm==0.6.3.post1->sglang[all])
Downloading httptools-0.6.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB)
Collecting watchfiles>=0.13 (from uvicorn[standard]->vllm==0.6.3.post1->sglang[all])
Downloading watchfiles-0.24.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)
Collecting websockets>=10.4 (from uvicorn[standard]->vllm==0.6.3.post1->sglang[all])
Downloading websockets-13.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB)
Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from yarl<2.0,>=1.12.0->aiohttp->vllm==0.6.3.post1->sglang[all]) (0.2.0)
Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (17.0.0)
Collecting dill<0.3.9,>=0.3.0 (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])
Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (2.2.2)
Collecting xxhash (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])
Downloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Collecting multiprocess<0.70.17 (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])
Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)
Collecting fsspec (from torch->sglang[all])
Downloading fsspec-2024.9.0-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (0.43.0)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (2024.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (1.16.0)
Downloading sglang-0.3.5-py3-none-any.whl (436 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 436.8/436.8 kB 26.7 MB/s eta 0:00:00
Downloading vllm-0.6.3.post1-cp38-abi3-manylinux1_x86_64.whl (194.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.8/194.8 MB 5.4 MB/s eta 0:00:00
Downloading torch-2.4.0-cp310-cp310-manylinux1_x86_64.whl (797.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 797.2/797.2 MB 2.0 MB/s eta 0:00:00
Downloading compressed_tensors-0.6.0-py3-none-any.whl (92 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.0/92.0 kB 8.0 MB/s eta 0:00:00
Downloading gguf-0.10.0-py3-none-any.whl (71 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.6/71.6 kB 6.8 MB/s eta 0:00:00
Downloading lm_format_enforcer-0.10.6-py3-none-any.whl (43 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.7/43.7 kB 3.8 MB/s eta 0:00:00
Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 4.0 MB/s eta 0:00:00
Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 41.3 MB/s eta 0:00:00
Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 64.7 MB/s eta 0:00:00
Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 53.9 MB/s eta 0:00:00
Downloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl (664.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 664.8/664.8 MB 2.6 MB/s eta 0:00:00
Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 7.4 MB/s eta 0:00:00
Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 11.6 MB/s eta 0:00:00
Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 7.3 MB/s eta 0:00:00
Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 5.7 MB/s eta 0:00:00
Downloading nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 176.2/176.2 MB 6.7 MB/s eta 0:00:00
Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 8.8 MB/s eta 0:00:00
Downloading torchvision-0.19.0-cp310-cp310-manylinux1_x86_64.whl (7.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.0/7.0 MB 109.7 MB/s eta 0:00:00
Downloading triton-3.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (209.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.4/209.4 MB 1.9 MB/s eta 0:00:00
Downloading xformers-0.0.27.post2-cp310-cp310-manylinux2014_x86_64.whl (20.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.8/20.8 MB 90.4 MB/s eta 0:00:00
Downloading anthropic-0.39.0-py3-none-any.whl (198 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 198.4/198.4 kB 16.2 MB/s eta 0:00:00
Downloading jedi-0.19.1-py2.py3-none-any.whl (1.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 69.1 MB/s eta 0:00:00
Downloading litellm-1.51.3-py3-none-any.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 92.3 MB/s eta 0:00:00
Downloading fastapi-0.115.4-py3-none-any.whl (94 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94.7/94.7 kB 9.2 MB/s eta 0:00:00
Downloading interegular-0.3.3-py37-none-any.whl (23 kB)
Downloading mistral_common-1.4.4-py3-none-any.whl (6.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 103.9 MB/s eta 0:00:00
Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 60.3 MB/s eta 0:00:00
Downloading outlines-0.0.46-py3-none-any.whl (101 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 101.9/101.9 kB 9.2 MB/s eta 0:00:00
Downloading prometheus_fastapi_instrumentator-7.0.0-py3-none-any.whl (19 kB)
Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Downloading ray-2.38.0-cp310-cp310-manylinux2014_x86_64.whl (66.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.0/66.0 MB 9.8 MB/s eta 0:00:00
Downloading transformers-4.46.1-py3-none-any.whl (10.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/10.0 MB 84.5 MB/s eta 0:00:00
Downloading tokenizers-0.20.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 74.0 MB/s eta 0:00:00
Downloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl (13.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.6/13.6 MB 68.1 MB/s eta 0:00:00
Downloading hf_transfer-0.1.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 35.0 MB/s eta 0:00:00
Downloading modelscope-1.19.2-py3-none-any.whl (5.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 63.5 MB/s eta 0:00:00
Downloading msgspec-0.18.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (210 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 210.3/210.3 kB 16.4 MB/s eta 0:00:00
Downloading nvidia_ml_py-12.560.30-py3-none-any.whl (40 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.5/40.5 kB 3.2 MB/s eta 0:00:00
Downloading partial_json_parser-0.2.1.1.post4-py3-none-any.whl (9.9 kB)
Downloading python_multipart-0.0.17-py3-none-any.whl (24 kB)
Downloading torchao-0.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 67.5 MB/s eta 0:00:00
Downloading uvicorn-0.32.0-py3-none-any.whl (63 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.7/63.7 kB 6.0 MB/s eta 0:00:00
Downloading uvloop-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 69.7 MB/s eta 0:00:00
Downloading httptools-0.6.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (442 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 442.1/442.1 kB 26.6 MB/s eta 0:00:00
Downloading starlette-0.41.2-py3-none-any.whl (73 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.3/73.3 kB 7.1 MB/s eta 0:00:00
Downloading watchfiles-0.24.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (425 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 425.7/425.7 kB 30.8 MB/s eta 0:00:00
Downloading websockets-13.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (164 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.1/164.1 kB 15.7 MB/s eta 0:00:00
Downloading datasets-3.1.0-py3-none-any.whl (480 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 480.6/480.6 kB 38.2 MB/s eta 0:00:00
Downloading fsspec-2024.9.0-py3-none-any.whl (179 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 179.3/179.3 kB 16.9 MB/s eta 0:00:00
Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 4.3 MB/s eta 0:00:00
Downloading lark-1.2.2-py3-none-any.whl (111 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.0/111.0 kB 10.5 MB/s eta 0:00:00
Downloading pyairports-2.1.1-py3-none-any.whl (371 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 371.7/371.7 kB 27.5 MB/s eta 0:00:00
Downloading pycountry-24.6.1-py3-none-any.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 97.2 MB/s eta 0:00:00
Downloading dill-0.3.8-py3-none-any.whl (116 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 11.2 MB/s eta 0:00:00
Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 14.5 MB/s eta 0:00:00
Downloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 18.0 MB/s eta 0:00:00
Building wheels for collected packages: zmq
Building wheel for zmq (setup.py) ... done
Created wheel for zmq: filename=zmq-0.0.0-py3-none-any.whl size=1265 sha256=72c1d4d8588518e3127e52eb54d1bbbe8c082845ed37bd5d5c8edb19f807a508
Stored in directory: /root/.cache/pip/wheels/ab/c5/fe/d853f71843cae26c123d37a7a5934baac20fc66f35a913951d
Successfully built zmq
Installing collected packages: torchao, pyairports, nvidia-ml-py, zmq, xxhash, websockets, uvloop, uvicorn, triton, python-multipart, python-dotenv, pycountry, partial-json-parser, nvidia-nvtx-cu12, nvidia-nccl-cu12, nvidia-cusparse-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, msgspec, lark, jedi, interegular, httptools, hf-transfer, gguf, fsspec, diskcache, dill, decord, watchfiles, tiktoken, starlette, nvidia-cusolver-cu12, nvidia-cudnn-cu12, multiprocess, modelscope, torch, tokenizers, sglang, prometheus-fastapi-instrumentator, lm-format-enforcer, fastapi, anthropic, xformers, transformers, torchvision, ray, mistral-common, litellm, datasets, compressed-tensors, outlines, vllm
Attempting uninstall: nvidia-nccl-cu12
Found existing installation: nvidia-nccl-cu12 2.23.4
Uninstalling nvidia-nccl-cu12-2.23.4:
Successfully uninstalled nvidia-nccl-cu12-2.23.4
Attempting uninstall: nvidia-cusparse-cu12
Found existing installation: nvidia-cusparse-cu12 12.5.4.2
Uninstalling nvidia-cusparse-cu12-12.5.4.2:
Successfully uninstalled nvidia-cusparse-cu12-12.5.4.2
Attempting uninstall: nvidia-curand-cu12
Found existing installation: nvidia-curand-cu12 10.3.7.77
Uninstalling nvidia-curand-cu12-10.3.7.77:
Successfully uninstalled nvidia-curand-cu12-10.3.7.77
Attempting uninstall: nvidia-cufft-cu12
Found existing installation: nvidia-cufft-cu12 11.3.0.4
Uninstalling nvidia-cufft-cu12-11.3.0.4:
Successfully uninstalled nvidia-cufft-cu12-11.3.0.4
Attempting uninstall: nvidia-cuda-runtime-cu12
Found existing installation: nvidia-cuda-runtime-cu12 12.6.77
Uninstalling nvidia-cuda-runtime-cu12-12.6.77:
Successfully uninstalled nvidia-cuda-runtime-cu12-12.6.77
Attempting uninstall: nvidia-cuda-cupti-cu12
Found existing installation: nvidia-cuda-cupti-cu12 12.6.80
Uninstalling nvidia-cuda-cupti-cu12-12.6.80:
Successfully uninstalled nvidia-cuda-cupti-cu12-12.6.80
Attempting uninstall: nvidia-cublas-cu12
Found existing installation: nvidia-cublas-cu12 12.6.3.3
Uninstalling nvidia-cublas-cu12-12.6.3.3:
Successfully uninstalled nvidia-cublas-cu12-12.6.3.3
Attempting uninstall: fsspec
Found existing installation: fsspec 2024.10.0
Uninstalling fsspec-2024.10.0:
Successfully uninstalled fsspec-2024.10.0
Attempting uninstall: nvidia-cusolver-cu12
Found existing installation: nvidia-cusolver-cu12 11.7.1.2
Uninstalling nvidia-cusolver-cu12-11.7.1.2:
Successfully uninstalled nvidia-cusolver-cu12-11.7.1.2
Attempting uninstall: nvidia-cudnn-cu12
Found existing installation: nvidia-cudnn-cu12 9.5.1.17
Uninstalling nvidia-cudnn-cu12-9.5.1.17:
Successfully uninstalled nvidia-cudnn-cu12-9.5.1.17
Attempting uninstall: torch
Found existing installation: torch 2.5.0+cu121
Uninstalling torch-2.5.0+cu121:
Successfully uninstalled torch-2.5.0+cu121
Attempting uninstall: tokenizers
Found existing installation: tokenizers 0.19.1
Uninstalling tokenizers-0.19.1:
Successfully uninstalled tokenizers-0.19.1
Attempting uninstall: transformers
Found existing installation: transformers 4.44.2
Uninstalling transformers-4.44.2:
Successfully uninstalled transformers-4.44.2
Attempting uninstall: torchvision
Found existing installation: torchvision 0.20.0+cu121
Uninstalling torchvision-0.20.0+cu121:
Successfully uninstalled torchvision-0.20.0+cu121
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
gcsfs 2024.10.0 requires fsspec==2024.10.0, but you have fsspec 2024.9.0 which is incompatible.
torchaudio 2.5.0+cu121 requires torch==2.5.0, but you have torch 2.4.0 which is incompatible.
Successfully installed anthropic-0.39.0 compressed-tensors-0.6.0 datasets-3.1.0 decord-0.6.0 dill-0.3.8 diskcache-5.6.3 fastapi-0.115.4 fsspec-2024.9.0 gguf-0.10.0 hf-transfer-0.1.8 httptools-0.6.4 interegular-0.3.3 jedi-0.19.1 lark-1.2.2 litellm-1.51.3 lm-format-enforcer-0.10.6 mistral-common-1.4.4 modelscope-1.19.2 msgspec-0.18.6 multiprocess-0.70.16 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-ml-py-12.560.30 nvidia-nccl-cu12-2.20.5 nvidia-nvtx-cu12-12.1.105 outlines-0.0.46 partial-json-parser-0.2.1.1.post4 prometheus-fastapi-instrumentator-7.0.0 pyairports-2.1.1 pycountry-24.6.1 python-dotenv-1.0.1 python-multipart-0.0.17 ray-2.38.0 sglang-0.3.5 starlette-0.41.2 tiktoken-0.7.0 tokenizers-0.20.2 torch-2.4.0 torchao-0.6.1 torchvision-0.19.0 transformers-4.46.1 triton-3.0.0 uvicorn-0.32.0 uvloop-0.21.0 vllm-0.6.3.post1 watchfiles-0.24.0 websockets-13.1 xformers-0.0.27.post2 xxhash-3.5.0 zmq-0.0.0
Looking in indexes: https://flashinfer.ai/whl/cu121/torch2.4/
Collecting flashinfer
Downloading https://github.com/flashinfer-ai/flashinfer/releases/download/v0.1.6/flashinfer-0.1.6%2Bcu121torch2.4-cp310-cp310-linux_x86_64.whl (1322.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 GB 416.0 kB/s eta 0:00:00
Installing collected packages: flashinfer
Successfully installed flashinfer-0.1.6+cu121torch2.4
Overview
SGLang offers a fast backend runtime and flexible frontend language. To showcase the different ways in which Gemma 2 can be prompted using SGLang, this notebook is divided into the following sections:
- Launch a HTTP server using SGLang. Use Python
requeststo prompt Gemma using SGLang's native genration APIs. - Set up a SGLang backend inference engine to prompt Gemma without a HTTP server.
- Use SGLang's frontend generation language to prompt Gemma and also explore a few of its capabilities.
1. Sending requests to SGLang server running Gemma 2
In this section, you will launch an HTTP server to run Gemma 2 using SGLang and send a prompt to the model using the native generation API endpoint.
Launch a server
The SGLang server can be launched by running the following command in the terminal:
python -m sglang.launch_server --model-path google/gemma-2-2b-it --port YOUR_PREFERRED_PORT
In a Colab environment, you must run the SGLang server as a Python subprocess and manage its termination using Python's subprocess package. SGLang provides some utility methods that abstract these details for you. The execute_shell_command function lets you launch the server as a Python subprocess, while the wait_for_server function waits for the server to be up and running before you can send requests to it.
You can specify Gemma 2's Hugging Face repo ID directly for the --model-path argument. SGLang will download the necessary files from the Hugging Face repository to start the server.
You can set any port of your choice to run SGLang using the --port argument.
Throughout this notebook, --mem-fraction-static is set to 0.6 to avoid CUDA Out of Memory errors when running on the Colab free tier. Setting the --mem-fraction-static argument to a lower value reduces the memory usage of the KV cache memory pool. Feel free to experiment with different values according to your use case.
Note: The following code snippet defines a function that executes the shell command and waits for the server to be ready. This function will be reused later in this notebook.
Invoke the previously defined start_server function to start the server and obtain a reference to the server process.
Note: It takes 2 - 4 minutes for the server to be up and running.
The server is now ready and can be reached at http://localhost:9000/ from within this notebook.
Send a request to Gemma 2 using SGLang's Native Generation API
The following code snippet uses Python's requests library to invoke SGLang's native generation API on the server to send a prompt to Gemma-2.
You can specify your preferred values for the sampling parameters like temperature, top_p max_new_tokens etc.
For a full list of sampling parameters supported by SGLang, please refer to SGLang's Sampling Parameters in SGLang Runtime guide.
To generate a streaming response from the model, specify an additional key, stream set to True in the request json and set the stream parameter of requests.post to True.
An example of a streaming generation is provided in SGLang's Quick Start documentation.
{
"text": " \n\nI'm confused. \n\nIs it billions of years old?\n\nPlease explain. \n\n\nYou're right to be confused! It's a big number. Here's a breakdown:\n\n**Earth is about 4.54 \u00b1 0.05 billion years old.**\n\n* **Billions:** This means it's older than you and me, for sure! \n* **4.54 billion:** This is the most precise estimate we have. \n* **\u00b1 0.05:** This means there's a range of 0.0",
"meta_info": {
"prompt_tokens": 8,
"completion_tokens": 128,
"completion_tokens_wo_jump_forward": 128,
"cached_tokens": 1,
"finish_reason": {
"type": "length",
"length": 128
},
"id": "1485c86977304adf92b2db1f77054a07"
}
}
You can stop the server by using the terminate_process function from sglang.utils. This is equivalent to pressing Ctrl+C to stop the server from the terminal.
2. Offline batch inference using SGLang backend engine
SGLang provides an inference engine that allows you to directly interact with local models like Gemma 2 without requiring an HTTP server. You can use this for building custom servers or for offline batch inference.
In this section, you will initialize the inference engine to run Gemma 2 and send a batch of prompts to it.
Initialize SGLang inference engine with Gemma 2
Create an instance of sglang.Engine class to run Gemma 2 by specifying its Hugging repo ID for themodel_path argument.
Batch prompting Gemma 2 using SGLang inference engine
You can send a batch of prompts for inference to the SGLang engine in one of the following ways:
- Non-streaming synchronous call
- Streaming synchronous call
- Non-streaming asynchronous call
- Streaming asynchronous call
You will explore how to perform inference on a batch of prompts using the SGLang engine's synchronous generation function to generate both streaming and non-streaming responses from Gemma 2 in the following sections.
You can refer to SGLang's Offline Engine API guide for examples of asynchronous response generation.
Non-streaming synchronous prompting
Define a list of prompts to query Gemma 2 with.
Generate a batch of non-streaming responses from Gemma 2 using the inference engine's generate function. Pass the list of prompts you defined earlier and an optional dictionary of sampling parameters to this function. The function returns a list of complete responses from the model to the batch of prompts.
================================================================= Prompt: Summarize what a galaxy is in three to four lines. Generated text: A galaxy is a vast collection of stars, gas, dust, and dark matter held together by gravity. It is a massive, gravitationally bound system that can range in size from a few hundred thousand to billions of stars. Galaxies come in various shapes and sizes, from spiral galaxies like our Milky Way, to elliptical galaxies, and irregular galaxies. ================================================================= Prompt: List any 3 observatories in the world. Generated text: Here are 3 observatories in the world: 1. **Keck Observatory:** Located on Mauna Kea in Hawaii, the Keck Observatory is home to two of the world's largest optical/infrared telescopes. 2. **Very Large Telescope (VLT):** Located in the Atacama Desert of Chile, the VLT is a collection of four telescopes that work together to provide high-resolution images of distant objects. 3. **James Webb Space Telescope (JWST):** Launched in December 2021, the JWST is the largest and most powerful space telescope ever built, designed to study
Streaming synchronous prompting
To generate streaming responses from the model to the previously defined batch of prompts, iterate over the prompts and invoke the inference engine's generate function with an additional argument stream set to True. You can access each chunk in the streaming response by iterating over the response of the generate function.
=============================================================== Prompt: Summarize what a galaxy is in three to four lines. Generated text: A galaxy is a massive collection of stars, gas, dust, and dark matter held together by gravity. These vast structures range in size from a few tens of thousands to billions of light-years across. Galaxies are the building blocks of the universe, containing billions of stars and countless planets. They come in various shapes and sizes, from spiral galaxies like our own Milky Way to elliptical galaxies and irregular galaxies. =============================================================== Prompt: List any 3 observatories in the world. Generated text: Here are 3 observatories in the world: 1. **Keck Observatory:** Located on Mauna Kea in Hawaii, the Keck Observatory is one of the world's most powerful optical/infrared telescopes. 2. **Very Large Telescope (VLT):** Located in the Atacama Desert of Chile, the VLT is a collection of four telescopes that work together to provide high-resolution images of distant objects. 3. **European Southern Observatory (ESO) Very Large Telescope (VLT):** Located in the Atacama Desert of Chile, the VLT is a collection of four telescopes that work together to
Now you can shut down and clean up the SGLang inference engine.
W1105 16:28:30.518000 131977277298240 torch/_inductor/compile_worker/subproc_pool.py:126] SubprocPool unclean exit
3. Inference using Frontend Structured Generated Language (SGLang)
In addition to the HTTP server and the offline backend engine, SGLang also offers a frontend language that supports more customization and complex prompting workflows.
In the following sections, you will explore how to start a multi-turn conversation with Gemma 2 using SGLang's frontend language. You will also see how to obtain responses from Gemma 2 in JSON format.
Launch a server
First, you must launch a server using SGLang specifying the Hugging Face repo ID of Gemma 2. You can use the function defined in the introductory sections to launch the server.
Use the function decorator provided by SGLang to define a function that accepts a few questions you want to ask the model as its arguments. The user function is used to add the user's question to the conversation. The sglang.gen function is used to generate a response from the model, which is in turn appended to the conversation using the assistant function.
The function prompts the model with question_1 and then in turn prompts it with question_2. The model is expected to answer question_2 based on the history of the conversation.
Connect to the server
Connect to the server using sglang.set_default_backend by specifying its URL.
Send multi-turn questions to Gemma 2
Now, you can run the previously defined multi_turn_question function to generate responses from the model.
user : Who are the first humans to land on the moon? assistant : The first humans to ever land on the moon were a team from the **Apollo 11 mission**: * **Neil Armstrong**: He became the first person to walk on the moon. His famous quote, "One small step for man, one giant leap for mankind," encapsulates the magnitude of thishistoric event. * **Buzz Aldrin**: Aldrin was the second human to walk on the moon and stayed with Armstrong for several hours on the lunar surface. They landed on the moon on **July 20, 1969**, bringing back a wealth of lunar samples and photos that remain highly significant user : Which country did they belong to? assistant : The first people to land on the moon were part of **the United States**, often simply referred to as Americans. They were a team from NASA, the National Aeronautics and Space Administration, the US government's space program.
Notice how the history of the conversation is preserved, and the model answered the second question as a continuation of the conversation.
Run a batch of multi-turn questions
You can also batch a set of multi turn questions to the model by passing a list of dictionaries to run_batch whose keys specify the arguments to the multi_turn_question function.
=============================================================== user : Who are the first humans to land on moon? assistant : The first humans to land on the Moon were **Neil Armstrong** and **Buzz Aldrin** of the Apollo 11 mission on **July 20, 1969.** user : Which country did they belong to ? assistant : Neil Armstrong and Buzz Aldrin were from the **United States**. =============================================================== user : Who is the first human to reach space? assistant : The first human to reach space was **Yuri Gagarin**. On April 12, 1961, he completed one orbit of Earth in the Soviet Vostok 1 spacecraft. This event marked a significant moment in the history of human exploration, paving the way for further advancements and spaceflight feats. user : Which country did they belong to? assistant : Yuri Gagarin was from **Soviet Union** at the time.
JSON Decoding
You can use a regular expression (regex) to specify a JSON schema that the model's generated answer must adhere to.
Define a function to generate specific information about any animal in JSON format using Gemma 2. Specify the regex JSON schema in the regex argument of the sglang.gen function.
Run the function with the name of any animal as input to get its features in JSON format.
Fish is an animal. Please fill in the following information about this animal.
{
"name": "Fish",
"type": "Mammals",
"reproduction": "Sexual",
"life expectancy": "10",
}
Terminate the server process.
These are just a few examples of how a prompting workflow with Gemma 2 can be designed using SGLang's frontend language. To learn more about its capabilities, you can refer to SGLang's Frontend: Structured Generation Language (SGLang) guide.
Congratulations! You've successfully explored how Gemma 2 can be served using SGLang, run using the SGLang backend runtime and frontend language in a Colab environment. You can now experiment with more complex prompting workflows in SGLang to interact with Gemma 2.
Run in Google Colab