Notebooks
G
Google Gemini
[Gemma 3]Gradio LlamaCpp Chatbot

[Gemma 3]Gradio LlamaCpp Chatbot

recurrentgemmagemmapaligemmagemma-cookbookcodegemma
Copyright 2025 Google LLC.
[ ]

Build Chatbot with Gradio & Llama.cpp

Author: Sitam Meur

Description: Google recently released Gemma 3 QAT—the Quantization Aware Trained (QAT) Gemma 3 checkpoints. These models maintain similar quality to half precision while using three times less memory. This notebook demonstrates creating a user-friendly chat interface for the gemma-3-1b-it-qat text model using Llama.cpp (for inference) and Gradio (for user interface).

Setup

Select the Colab runtime

To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma 3 QAT model. In this case, you can use a CPU:

  1. In the upper-right of the Colab window, select ▾ (Additional connection options).
  2. Select Change runtime type.
  3. Under Hardware accelerator, select CPU.

Gemma Setup

Before we dive into the tutorial, let's get you set up:

  1. Hugging Face Account: If you don't already have one, you can create a free Hugging Face account by clicking here.
  2. Model Access: Head over to the model page and accept the usage conditions.
  3. Hugging Face Token: Generate a Hugging Face access (preferably write permission) token by clicking here. You'll need this token later in the tutorial.

Once you've completed these steps, you're ready to move on to the next section where we'll set up environment variables in your Colab environment.

Configure your HF token

Add your Hugging Face token to the Colab Secrets manager to securely store it.

  1. Open your Google Colab notebook and click on the 🔑 Secrets tab in the left panel. The Secrets tab is found on the left panel.
  2. Create a new secret with the name HF_TOKEN.
  3. Copy/paste your token key into the Value input box of HF_TOKEN.
  4. Toggle the button on the left to allow notebook access to the secret.
[ ]

Install dependencies

Run the cell below to install all the required dependencies.

[ ]

Log into Hugging Face Hub

[ ]

Instantiate the Gemma 3 QAT model

We’ll use the Hugging Face Hub to download the 4-bit (Q4_0) quantized Gemma 3 1B instruction-tuned text-only model. New quantized Gemma 3 models, created using Quantization-Aware Training (QAT), offer improved accessibility through reduced memory usage (3x less than bfloat16) without significant accuracy loss. QAT simulates low-precision operations during training for smaller, faster models.

Let's get started by downloading the model from Hugging Face Hub.

Loading the model from HF Hub

[ ]

Prompt Formatting

Gemma 3 model requires specific formatting to understand the roles of different participants in a conversation. The prompt format is as follows:

<bos><start_of_turn>user
{system_prompt}

{prompt}<end_of_turn>
<start_of_turn>model
[ ]

Chat with Gemma 3

This function handles loading the model and generating responses. Streaming provides real-time generation rather than waiting for the complete response.

[ ]

Gradio UI

This offers a user interface with customizable buttons, editable messages, and options for advanced users to tune model parameters like temperature and top-p.

[ ]

Finally, we’ll launch our chat interface with Gradio

[ ]

What's Next?

That's it! Here are some ideas to explore further:

  • Explore the Gemma family models: Visit Gemma Open Models to learn about the latest updates regarding the Gemma family models, new capabilities, versions, and more.

  • Gradio Customization: Explore the Gradio documentation to learn about customizing your chat interface, adding new options and features.

  • Share Your Gradio Dashboard: Check out the Sharing your Gradio app page to learn how to safely share your Gradio dashboard with others!