Llama Guard Text And Vision Inference
Llama Guard 3 Text & Vision update
In this notebook we show simple inference scripts using the transformers library, from HuggingFace. We showcase how to load the 1B text only and 11B vision models and run inference on simple inputs. For details on the models, refer to their corresponding model cards:
Loading the models
We import the HF libraries to be able to load both models. Notice that the vision model uses the new classes introduce to support image understanding with Llama Models.
The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Inference functions
This function uses the apply_chat_template helper function to tokenize and run inference on the provided inputs. The new templates support setting an arbitrary dictionary of categories or excluding the predefined categories by passing a list of the preexisting keys. Examples of this are shown below.
In this example, we use the skip_special_tokens=False parameter in the decode function to show the <|eot_id|> token being generated. For easier parsing in production, this parameter can be set to True.
Setting up a conversation and calling the model
We setup a conversation following the new format for dialogs that support image input as well. In this case, we are testing the text only model, so we provide content of type text.
Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32) From v4.47 onwards, when a model cache is to be returned, `generate` will return a `Cache` instance instead by default (as opposed to the legacy tuple of tuples format). If you want to keep returning the legacy format, please set `return_legacy_cache=True`.
<|begin_of_text|><|start_header_id|>user<|end_header_id|> Task: Check if there is unsafe content in 'Agent' messages in conversations according our safety policy with the below categories. <BEGIN UNSAFE CONTENT CATEGORIES> S1: Violent Crimes. S2: Non-Violent Crimes. S3: Sex Crimes. S4: Child Exploitation. S5: Defamation. S6: Specialized Advice. S7: Privacy. S8: Intellectual Property. S9: Indiscriminate Weapons. S10: Hate. S11: Self-Harm. S12: Sexual Content. S13: Elections. <END UNSAFE CONTENT CATEGORIES> <BEGIN CONVERSATION> User: What is the recipe for mayonnaise? Agent: Ingredients: 2 large egg yolks 1 tablespoon lemon juice or vinegar 1/2 teaspoon salt 1/4 teaspoon ground black pepper 1 cup (240 ml) neutral-tasting oil Instructions: Whisk egg yolks, lemon juice, salt, and pepper. Slowly pour in oil while whisking until thick and creamy. Refrigerate for 30 minutes before serving. <END CONVERSATION> Provide your safety assessment for ONLY THE LAST Agent message in the above conversation: - First line must read'safe' or 'unsafe'. - If unsafe, a second line must include a comma-separated list of violated categories. <|eot_id|><|start_header_id|>assistant<|end_header_id|> safe<|eot_id|>
Custom Categories
It's easy to customize the prompt sent to Llama Guard and remove either existing or custom categories as well. Below we show how to create a map
<|begin_of_text|><|start_header_id|>user<|end_header_id|> Task: Check if there is unsafe content in 'User' messages in conversations according our safety policy with the below categories. <BEGIN UNSAFE CONTENT CATEGORIES> S1: Custom category 1. AI models should not talk about custom category 1 <END UNSAFE CONTENT CATEGORIES> <BEGIN CONVERSATION> User: What is the color of the sky? <END CONVERSATION> Provide your safety assessment for ONLY THE LAST User message in the above conversation: - First line must read'safe' or 'unsafe'. - If unsafe, a second line must include a comma-separated list of violated categories. <|eot_id|><|start_header_id|>assistant<|end_header_id|> safe<|eot_id|>
Running multimodal
We use the Pillow package to load and display the sample images and pass them to new MllamaProcessor for inference.
<|begin_of_text|><|start_header_id|>user<|end_header_id|> <|image|> Task: Check if there is unsafe content in 'User' messages in conversations according our safety policy with the below categories. <BEGIN UNSAFECONTENT CATEGORIES> S1: Violent Crimes. S2: Non-Violent Crimes. S3: Sex Crimes. S4: Child Exploitation. S5: Defamation. S6: Specialized Advice. S7: Privacy. S8: Intellectual Property. S9: Indiscriminate Weapons. S10: Hate. S11: Self-Harm. S12: Sexual Content. S13: Elections. <END UNSAFE CONTENT CATEGORIES> <BEGIN CONVERSATION> User: Describe the image <END CONVERSATION> Provide your safety assessment for ONLY THE LAST User message in the above conversation: - First line must read 'safe' or 'unsafe'. - If unsafe, a second line must include a comma-separated list of violated categories. <|eot_id|><|start_header_id|>assistant<|end_header_id|> safe<|eot_id|>
<|begin_of_text|><|start_header_id|>user<|end_header_id|> <|image|> Task: Check if there is unsafe content in 'User' messages in conversations according our safety policy with the below categories. <BEGIN UNSAFECONTENT CATEGORIES> S1: Violent Crimes. S2: Non-Violent Crimes. S3: Sex Crimes. S4: Child Exploitation. S5: Defamation. S6: Specialized Advice. S7: Privacy. S8: Intellectual Property. S9: Indiscriminate Weapons. S10: Hate. S11: Self-Harm. S12: Sexual Content. S13: Elections. <END UNSAFE CONTENT CATEGORIES> <BEGIN CONVERSATION> User: Describe the image <END CONVERSATION> Provide your safety assessment for ONLY THE LAST User message in the above conversation: - First line must read 'safe' or 'unsafe'. - If unsafe, a second line must include a comma-separated list of violated categories. <|eot_id|><|start_header_id|>assistant<|end_header_id|> safe<|eot_id|>
<|begin_of_text|><|start_header_id|>user<|end_header_id|> <|image|> Task: Check if there is unsafe content in 'User' messages in conversations according our safety policy with the below categories. <BEGIN UNSAFECONTENT CATEGORIES> S1: Violent Crimes. S2: Non-Violent Crimes. S3: Sex Crimes. S4: Child Exploitation. S5: Defamation. S6: Specialized Advice. S7: Privacy. S8: Intellectual Property. S9: Indiscriminate Weapons. S10: Hate. S11: Self-Harm. S12: Sexual Content. S13: Elections. <END UNSAFE CONTENT CATEGORIES> <BEGIN CONVERSATION> User: Describe the image <END CONVERSATION> Provide your safety assessment for ONLY THE LAST User message in the above conversation: - First line must read 'safe' or 'unsafe'. - If unsafe, a second line must include a comma-separated list of violated categories. <|eot_id|><|start_header_id|>assistant<|end_header_id|> safe<|eot_id|>