Notebooks
G
Google Gemini
[Gemma 2]Using With Langfun And LlamaCpp Python Bindings

[Gemma 2]Using With Langfun And LlamaCpp Python Bindings

recurrentgemmagemmapaligemmagemma-cookbookcodegemma
Copyright 2024 Google LLC.
[ ]

Guide to using Gemma 2 with Langfun, PyGlove and llama-cpp-python

Langfun is a versatile library designed to bridge the gap between natural language processing (NLP) and structured data manipulation. It allows you to effortlessly parse, interpret, and convert unstructured text into well-defined Python objects. This capability is invaluable for applications that require seamless integration between language models and data-driven systems.

PyGlove is a powerful library for flexible and scalable configuration management and automated machine learning (AutoML). It provides tools for defining complex schemas and object representations, enabling you to create highly customizable and maintainable code structures. PyGlove's compatibility with Langfun ensures that the objects you generate from natural language prompts are both robust and easy to use.

llama-cpp-python provides Python bindings for the llama.cpp C++ library. This allows you to enjoy the performance optimizations of llama.cpp while benefiting from the simplicity and flexibility of Python. With llama-cpp-python, you get a convenient API for loading models, generating text, and customizing inference parameters.

Gemma is a family of lightweight, state-of-the-art open-source language models from Google. Built from the same research and technology used to create the Gemini models, Gemma models are text-to-text, decoder-only large language models (LLMs), available in English, with open weights, pre-trained variants, and instruction-tuned variants. Gemma models are well-suited for various text-generation tasks, including question-answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop, or your cloud infrastructure, democratizing access to state-of-the-art AI models and helping foster innovation for everyone.

By combining Langfun, PyGlove, llama.cpp and Gemma, you can create applications that not only understand and process natural language but also convert that understanding into structured, actionable data. This integration allows you to build sophisticated systems that can interact with external APIs, databases, and other services using well-defined objects, streamlining workflows and enhancing functionality.

In this guide, you'll learn how to set up your environment, configure the Gemma 2 model, and implement various real-world use cases.

Setup

Select the Colab runtime

To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:

  1. In the upper-right of the Colab window, select β–Ύ (Additional connection options).
  2. Select Change runtime type.
  3. Under Hardware accelerator, select T4 GPU.

Gemma setup

Before you dive into the tutorial, let's get you set up with Gemma:

  1. Hugging Face Account: If you don't already have one, you can create a free Hugging Face account by clicking here.
  2. Gemma Model Access: Head over to the Gemma model page and accept the usage conditions.
  3. Colab with Gemma Power: For this tutorial, you'll need a Colab runtime with enough resources to handle the Gemma 9B model. Choose an appropriate runtime when starting your Colab session.
  4. Hugging Face Token: Generate a Hugging Face access (preferably write permission) token by clicking here. You'll need this token later in the tutorial.

Once you've completed these steps, you're ready to move on to the next section where you'll set up environment variables in your Colab environment.

Configure your HF token

Add your Hugging Face token to the Colab Secrets manager to securely store it.

  1. Open your Google Colab notebook and click on the πŸ”‘ Secrets tab in the left panel. The Secrets tab is found on the left panel.
  2. Create a new secret with the name HF_TOKEN.
  3. Copy/paste your token key into the Value input box of HF_TOKEN.
  4. Toggle the button on the left to allow notebook access to the secret.
[ ]

Install dependencies

You'll need to install a few Python packages for Langfun and PyGlove and some dependencies to interact with HuggingFace along with llama-cpp-python and run the model. Find some of the releases supporting CUDA 12.2 here.

Run the following cell to install or upgrade it:

[ ]
  Preparing metadata (setup.py) ... done
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 297.5/297.5 kB 3.6 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 70.1/70.1 kB 3.8 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 585.4/585.4 kB 9.1 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 244.3/244.3 kB 13.2 MB/s eta 0:00:00
  Building wheel for termcolor (setup.py) ... done
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 436.4/436.4 kB 8.3 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 443.8/443.8 MB 3.4 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 4.3 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94.6/94.6 kB 8.9 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.7/63.7 kB 5.8 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 5.5 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.5/71.5 kB 5.1 MB/s eta 0:00:00

Logging into Hugging Face Hub

Next, you'll have to log into the Hugging Face Hub using your access token. This will allow us to download the Gemma model.

[ ]
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: read).
Your token has been saved to /root/.cache/huggingface/token
Login successful

Downloading the Gemma 2 Model

Once you're logged in, you can download the Gemma 2 model files from Hugging Face. The Gemma 2 model is available in GGUF format, which is optimized for use with llama.cpp and compatible tools like Llamafile.

[ ]
gemma-2-9b-it-Q6_K.gguf:   0%|          | 0.00/7.59G [00:00<?, ?B/s]
'gemma-2-9b-it-Q6_K.gguf'

Running the Gemma 2 model using llama-cpp-python

You will initialize the model using the llama-cpp-python library by loading a pre-trained Gemma 2 model from HuggingFace.

[ ]

Hint: You can use tail -f server.log to preview the command's output if necessary.

[ ]

Import Libraries

[ ]

Configuring the Gemma 2 Model

To use the Gemma 2 model with Langfun, we need to create a custom llama.cpp Python class since Langfun may not have built-in support for it yet.

[ ]

To programmatically add control tokens to any prompt for use with an instruction-tuned model, you can create a function that wraps the prompt with the necessary control tokens. Here’s an example Python function that can be reused to apply this formatting:

[ ]

Using Langfun with Gemma 2 from Google AI

Langfun allows you to define a clear schema and effortlessly convert natural language inputs into structured objects. These objects can then be directly plugged into external systems, databases, or external APIs, facilitating seamless data flow and integration without the need for manual parsing or data cleaning.

First, let's try a simple use case by asking Langfun to represent a simple mathematical statement's final answer using a Python class Answer

[ ]
Answer(
  result = 3
)

Next, you'll take a look at real-world use cases demonstrating how you can leverage Langfun and Gemma 2 to build robust applications.

Parsing a Job Description into a Structured Object

Automating the recruitment process can significantly enhance efficiency and accuracy in HR departments. By parsing job descriptions into structured objects, you can seamlessly integrate this data into your HR systems, enabling functionalities like resume matching, job postings, and analytics.

Let's start first by defining a Job schema using PyGlove, then provide a job description as input. Langfun, powered by the Gemma 2 model, processes the text and extracts relevant information, populating the Job object accordingly.

[ ]
Job(
  title = 'Software Engineer',
  company = 'TechCorp',
  location = 'San Francisco',
  responsibilities = [
    0 : 'developing software solutions',
    1 : 'collaborating with cross-functional teams',
    2 : 'participating in code reviews'
  ],
  requirements = [
    0 : "Bachelor's degree in Computer Science",
    1 : '3+ years of experience in software development',
    2 : 'proficiency in Python and JavaScript'
  ]
)

This structured data can then be used directly in your HR systems or external APIs for various purposes such as job matching and integration with Applicant Tracking Systems (ATS) for streamlined recruitment workflows.

Enabling Automated Handling of Support Tickets

In this second example, you'll create a support ticket by incorporating variables into your prompt. In customer service applications, generating support tickets efficiently is crucial for timely issue resolution. By creating structured support tickets from user inputs, you can integrate seamlessly with ticketing systems, prioritize issues, and assign them to the appropriate teams.

Define a SupportTicket schema and use variables like customer_name and issue in your prompt. Langfun processes these variables within the context, generating a structured SupportTicket object that can be integrated into your customer service systems or external APIs. This lets you automatically create tickets in a system, easily track ticket statuses and generate reports.

[ ]
[ ]
SupportTicket(
  customer_name = 'John Doe',
  issue = 'I am unable to log into my account.',
  steps_taken = [
    0 : 'Verify username and password accuracy.',
    1 : 'Attempt password reset.',
    2 : 'Check for browser cache or cookie issues.'
  ],
  resolution = None
)

In a scenario where your customer service team receives numerous inquiries daily, by automating the creation of support tickets, you can ensure that each issue is documented consistently and efficiently. The structured SupportTicket objects can be directly fed into your CRM or ticketing software, facilitating faster response times and better customer satisfaction.

Parsing Customer Feedback into Structured Data

Understanding customer feedback is essential for improving products and services. You'll start by first defining a Feedback schema and provide customer feedback text. Langfun processes the feedback, determining sentiment and identifying key topics, and outputs a structured Feedback object. This allows you to parse customer feedback into structured data, aiding in sentiment analysis, report generation and topic modeling.

[ ]
Feedback(
  customer_id = 1,
  feedback_text = 'I love the new features in the app, but it crashes frequently when I try to upload photos.',
  sentiment = 'mixed',
  topics = [
    0 : 'new features',
    1 : 'app crashes',
    2 : 'photo uploads'
  ]
)

This example shows how the model can extract valuable insights from customer feedback, which is crucial for product improvement. Imagine you manage a mobile app with a large user base. By systematically parsing feedback, you can identify which features are well-received and which areas require attention. For instance, detecting frequent crashes related to photo uploads allows your development team to prioritize bug fixes, enhancing the overall user experience.

Classifying Support Tickets Using Literal Types

Efficiently categorizing and prioritizing support tickets can significantly enhance your ticket triaging system. Proper classification ensures that issues are addressed by the right teams promptly, improving resolution times and customer satisfaction.

Define a Ticket schema with specific categories and priorities. Provide a support ticket description, and Langfun processes it to classify the ticket accordingly. Next, let's perform classification tasks using Literal types from the typing module, which is useful for ticket triaging systems.

[ ]
Ticket(
  id = 12345,
  issue = 'Customer reports that they are unable to update their billing information on the website.',
  priority = 'medium',
  category = 'billing',
  assigned_to = None
)

In a customer service center, tickets often vary widely in nature and urgency. By automatically classifying tickets, you ensure that billing issues are handled by a finance team, technical problems by the IT department, and so forth. Prioritizing tickets based on their severity helps in managing workloads effectively and ensuring that high-priority issues are resolved swiftly. This demonstrates the model's ability to categorize and prioritize support tickets automatically.

Chain-of-Thought Reasoning

Chain-of-thought (CoT) reasoning involves breaking down complex problems into a series of logical steps. This approach enhances the model's ability to solve intricate tasks by systematically reasoning through each component.

In this example, you'll demonstrate the model's capability in performing CoT reasoning for a simple puzzle.

[ ]
Solution(
  question = "How much in dollars does she make every day at the farmers' market?",
  steps = [
    0 : Step(
      description = 'Eggs laid daily',
      step_output = 16.0
    ),
    1 : Step(
      description = 'Eggs eaten for breakfast',
      step_output = -3.0
    ),
    2 : Step(
      description = 'Eggs used for baking',
      step_output = -4.0
    ),
    3 : Step(
      description = 'Total eggs remaining',
      step_output = 9.0
    ),
    4 : Step(
      description = 'Earnings from selling eggs',
      step_output = 18.0
    )
  ],
  final_answer = 18
)

This demonstrates the model's capability in performing chain-of-thought reasoning.

In this notebook, you've explored how to leverage Langfun, PyGlove, llama.cpp and the Gemma 2 model from Hugging Face to address a variety of real-world scenarios. Through practical use casesβ€”including parsing job descriptions, creating support tickets, analyzing customer feedback, classifying support tickets, and performing chain-of-thought reasoningβ€”you've seen how Langfun effortlessly converts unstructured text into structured, actionable objects. This showcases the model's effectiveness in real-world scenarios, enhancing the capabilities of your applications and seamlessly integrating with external systems.

Feel free to extend these examples to fit your specific needs and explore the full potential of Langfun and Gemma in your projects!