Meta Llama Part 1 Data Preparation

Part 1 Data Preparation

llamaMulti-Modal-RAGAIvllmmachine-learningend-to-end-use-casesllama2LLMllama-cookbooknotebooksPythonfinetuningpytorchlangchain

alph-notebooks/llama-cookbook / Part_1_Data_Preparation.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

Data Preparation Notebook

To make the experience consistent, we will use this link for getting access to our dataset. To credit, thanks to the author here for making it available.

As thanks to original author-Please upvote the dataset version on Kaggle if you enjoy this course.

Data Cleanup

Removing Corrupt Images

We will start by cleaning up the dataset first and checking for any corrupt images.

Variables and Paths

Let's first download the dataset and set our variables to point to it.

Remember, this is something you will change, don't rush the shift+enter fingers yet! Please also set your hf-token in the line below

[1]

All the imports

We import all the libraries here.

PIL: For handling images to be passed to our Llama model
Huggingface Transformers: For running the model
Concurrent Library: To clean up faster

[4]

Clean Corrupt Images

This might take a few moments since we have 5000 images in our dataset.

[5]

Corrupt images:
./DATA/images_compressed/d028580f-9a98-4fb5-a6c9-5dc362ad3f09.jpg
./DATA/images_compressed/784d67d4-b95e-4abb-baf7-8024f18dc3c8.jpg
./DATA/images_compressed/b72ed5cd-9f5f-49a7-b12e-63a078212a17.jpg
./DATA/images_compressed/1d0129a1-f29a-4a3f-b103-f651176183eb.jpg
./DATA/images_compressed/c60e486d-10ed-4f64-abab-5bb698c736dd.jpg
./DATA/images_compressed/040d73b7-21b5-4cf2-84fc-e1a80231b202.jpg
Total corrupt images found: 6

[6]

['./DATA/images_compressed/d028580f-9a98-4fb5-a6c9-5dc362ad3f09.jpg',
, './DATA/images_compressed/784d67d4-b95e-4abb-baf7-8024f18dc3c8.jpg',
, './DATA/images_compressed/b72ed5cd-9f5f-49a7-b12e-63a078212a17.jpg',
, './DATA/images_compressed/1d0129a1-f29a-4a3f-b103-f651176183eb.jpg',
, './DATA/images_compressed/c60e486d-10ed-4f64-abab-5bb698c736dd.jpg',
, './DATA/images_compressed/040d73b7-21b5-4cf2-84fc-e1a80231b202.jpg']

Let's load in the Meta-Data of the images and remove the rows with the corrupt images

[7]

[8]

Corrupt filenames:
['d028580f-9a98-4fb5-a6c9-5dc362ad3f09', '784d67d4-b95e-4abb-baf7-8024f18dc3c8', 'b72ed5cd-9f5f-49a7-b12e-63a078212a17', '1d0129a1-f29a-4a3f-b103-f651176183eb', 'c60e486d-10ed-4f64-abab-5bb698c736dd', '040d73b7-21b5-4cf2-84fc-e1a80231b202']

We can now "clean" up the dataframe by subtracting the corrupt images.

[9]

Number of rows removed: 5
                                  image  sender_id     label   kids
0  4285fab0-751a-4b74-8e9b-43af05deee22        124  Not sure  False
1  ea7b6656-3f84-4eb3-9099-23e623fc1018        148   T-Shirt  False
2  00627a3f-0477-401c-95eb-92642cbe078d         94  Not sure  False
3  ea2ffd4d-9b25-4ca8-9dc2-bd27f1cc59fa         43   T-Shirt  False
4  3b86d877-2b9e-4c8b-a6a2-1d87513309d0        189     Shoes  False

[10]

EDA

Let's start by double-checking any empty values

[11]

[12]

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5398 entries, 0 to 5397
Data columns (total 4 columns):
 #   Column     Non-Null Count  Dtype 
---  ------     --------------  ----- 
 0   image      5398 non-null   object
 1   sender_id  5398 non-null   int64 
 2   label      5398 non-null   object
 3   kids       5398 non-null   bool  
dtypes: bool(1), int64(1), object(2)
memory usage: 131.9+ KB

[13]

[14]


Missing values:
image        0
sender_id    0
label        0
kids         0
dtype: int64

Understanding the Label Distribution

The existing dataset comes with multi-labels, let's take a look at all categories:

[15]


Unique labels:
20

 Label Distribution:
label
T-Shirt       1011
Longsleeve     699
Pants          692
Shoes          431
Shirt          378
Dress          357
Outwear        312
Shorts         308
Not sure       228
Hat            171
Skirt          155
Polo           120
Undershirt     118
Blazer         109
Hoodie         100
Body            69
Other           67
Top             43
Blouse          23
Skip             7
Name: count, dtype: int64

[16]


Distribution of kids vs. non-kids images:
kids
False    0.911819
True     0.088181
Name: proportion, dtype: float64

Let's take a look at the distribution skew to understand what's in our dataset:

[17]

Let's start with some more cleanup:

Remove kids clothing since that is a smaller subset
Let's use our lack of understanding of fashion to reduce categories and also make our lives with pre-processing easier

[18]

Original dataset shape: (5398, 4)
Cleaned dataset shape: (4922, 3)

[19]

For once, lack of fashion knowledge is useful-we can reduce our work by creating less categories.

[20]

Unique categories after merging:
['Other' 'T-Shirt' 'Shoes' 'Shorts' 'Tops' 'Pants' 'Skirts']

[21]

This is the part that makes Thanos happy, we will balance our universe of clothes by randomly sampling.

[22]


Category counts in the balanced dataset:
merged_category
Pants      500
T-Shirt    500
Tops       500
Skirts     457
Shoes      371
Shorts     284
Other      266
Name: count, dtype: int64

/tmp/ipykernel_2065289/1389168415.py:7: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  df_balanced = df_cleaned.groupby('merged_category').apply(balance_category).reset_index(drop=True)

[23]

Balanced dataset shape: (2878, 4)
merged_category
Pants      500
T-Shirt    500
Tops       500
Skirts     457
Shoes      371
Shorts     284
Other      266
Name: count, dtype: int64

[24]

Synthetic Labelling using Llama 3.2

All the effort so far was to prepare our dataset for labelling.

At this stage, we are ready to start labelling the images using Llama-3.2 models. We will use 11B here for testing.

We suggest testing 90B as an assignment. Although you will find that 11B is a great candidate for this model.

Read more about the model capabilities here

[25]

The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]

[41]

Feel free to randomly grab any example from the ls command above. This shirt is colorful enough for us to use-so we will go with the current example

[27]

[28]

[29]

Labelling Prompt

We did a few sample runs to arrive on the prompt below:

Run a simple prompt on an image
See output and iterate

After painfully trying this a few times, we learn that for some reason the model doesn't follow JSON formatting unless it's strongly urged. So we fix this with the dramatic prompt:

[34]

[37]

'end_header_id|>\n\n{"Title": "Striped Collared Shirt", "Size": "L", "Category": "Tops", "Gender": "F", "Type": "Casual", "Description": "This shirt features a classic design with thin vertical stripes in multiple colors, including red, blue, yellow, and green, giving it a fun and playful look. The collar and cuffs are both long, with the collar being open and unbuttoned, and the cuffs rolled up slightly. The buttons are small and round. The fabric appears to be lightweight, and the shirt appears to be slightly wrinkled, adding to its casual charm. The solid grey background of the image suggests a plain backdrop, and the dark shadows of the shirt hanging on a hanger indicate that it is a product photo. Overall, this shirt is perfect for a casual, everyday look, and its fun and playful pattern makes it a great addition to any wardrobe."}<|eot_id|>'

[39]

end_header_id|>

{"Title": "Striped Collared Shirt", "Size": "L", "Category": "Tops", "Gender": "F", "Type": "Casual", "Description": "This shirt features a classic design with thin vertical stripes in multiple colors, including red, blue, yellow, and green, giving it a fun and playful look. The collar and cuffs are both long, with the collar being open and unbuttoned, and the cuffs rolled up slightly. The buttons are small and round. The fabric appears to be lightweight, and the shirt appears to be slightly wrinkled, adding to its casual charm. The solid grey background of the image suggests a plain backdrop, and the dark shadows of the shirt hanging on a hanger indicate that it is a product photo. Overall, this shirt is perfect for a casual, everyday look, and its fun and playful pattern makes it a great addition to any wardrobe."}<|eot_id|>

Testing Labelling Script

The results from labelling above look promising, we can now start building a script skeleton in the notebook to test our label logic.

Let's test our approach for first 50 images after which we can let this run on multi-GPUs in a script. Remember, Llama-3.2 models can only look at one image at once.

[43]

The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]

Processing files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [03:31<00:00,  4.23s/it]

[44]

It's always a great idea to validate LLM outputs, we can check our labels here:

[48]

[46]