Fine Tune BLIP2 On An Image Captioning Dataset PEFT
hf-notebookspeft
Export
Fine-tune BLIP using Hugging Face transformers, datasets, peft 🤗 and bitsandbytes
Let's leverage recent advances from Parameter Efficient Fine-Tuning methods to fine-tune a large image to text model! We will show through this tutorial that it is possible to fine-tune a 3B scale model (~6GB in half-precision)
Here we will use a dummy dataset of football players ⚽ that is uploaded on the Hub. The images have been manually selected together with the captions. Check the 🤗 documentation on how to create and upload your own image-text dataset.
Set-up environment
[1]
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.8/6.8 MB 102.7 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.3/104.3 MB 10.2 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.7/468.7 kB 45.8 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 215.3/215.3 kB 19.0 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 44.6 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.1/200.1 kB 21.8 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 55.6 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.9/132.9 kB 17.1 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 212.2/212.2 kB 25.7 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.5/110.5 kB 13.0 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 264.6/264.6 kB 26.9 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.2/114.2 kB 17.8 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.8/158.8 kB 20.5 MB/s eta 0:00:00
Building wheel for peft (pyproject.toml) ... done
Load the image captioning dataset
Let's load the image captioning dataset, you just need few lines of code for that.
[2]
Downloading readme: 0%| | 0.00/371 [00:00<?, ?B/s]
Downloading and preparing dataset None/None to /root/.cache/huggingface/datasets/ybelkada___parquet/ybelkada--football-dataset-1ad065f8e9005a29/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/2.07M [00:00<?, ?B/s]
Extracting data files: 0%| | 0/1 [00:00<?, ?it/s]
Generating train split: 0%| | 0/6 [00:00<?, ? examples/s]
Dataset parquet downloaded and prepared to /root/.cache/huggingface/datasets/ybelkada___parquet/ybelkada--football-dataset-1ad065f8e9005a29/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec. Subsequent calls will reuse this data.
Let's retrieve the caption of the first example:
[3]
"Benzema after Real Mardid's win against PSG"
And the corresponding image
[4]
Create PyTorch Dataset
Let's define below the dataset as well as the data collator!
[5]
Load model and processor
[ ]
Downloading (…)rocessor_config.json: 0%| | 0.00/432 [00:00<?, ?B/s]
Downloading (…)okenizer_config.json: 0%| | 0.00/904 [00:00<?, ?B/s]
Downloading (…)olve/main/vocab.json: 0%| | 0.00/798k [00:00<?, ?B/s]
Downloading (…)olve/main/merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s]
Downloading (…)/main/tokenizer.json: 0%| | 0.00/2.11M [00:00<?, ?B/s]
Downloading (…)cial_tokens_map.json: 0%| | 0.00/548 [00:00<?, ?B/s]
Overriding torch_dtype=None with `torch_dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don't pass this argument at all to remove this warning.
Downloading (…)lve/main/config.json: 0%| | 0.00/7.05k [00:00<?, ?B/s]
Downloading (…)model.bin.index.json: 0%| | 0.00/122k [00:00<?, ?B/s]
Downloading shards: 0%| | 0/8 [00:00<?, ?it/s]
Downloading (…)l-00001-of-00008.bin: 0%| | 0.00/995M [00:00<?, ?B/s]
Downloading (…)l-00002-of-00008.bin: 0%| | 0.00/999M [00:00<?, ?B/s]
Downloading (…)l-00003-of-00008.bin: 0%| | 0.00/985M [00:00<?, ?B/s]
Downloading (…)l-00004-of-00008.bin: 0%| | 0.00/997M [00:00<?, ?B/s]
Downloading (…)l-00005-of-00008.bin: 0%| | 0.00/997M [00:00<?, ?B/s]
Downloading (…)l-00006-of-00008.bin: 0%| | 0.00/997M [00:00<?, ?B/s]
Downloading (…)l-00007-of-00008.bin: 0%| | 0.00/997M [00:00<?, ?B/s]
Downloading (…)l-00008-of-00008.bin: 0%| | 0.00/782M [00:00<?, ?B/s]
===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run python -m bitsandbytes and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues ================================================================================ bin /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0 CUDA SETUP: Highest compute capability among GPUs detected: 7.5 CUDA SETUP: Detected CUDA version 118 CUDA SETUP: Loading binary /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so...
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: /usr/lib64-nvidia did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.28.0.1'), PosixPath('8013'), PosixPath('http')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('--listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-t4-s-1ay3dkij00hxw --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
Loading checkpoint shards: 0%| | 0/8 [00:00<?, ?it/s]
Next we define our LoraConfig object. We explicitly tell
[7]
trainable params: 5242880 || all params: 3749922816 || trainable%: 0.13981301102065136
Now that we have loaded the processor, let's load the dataset and the dataloader:
[8]
Train the model
Let's train the model! Run the simply the cell below for training the model
[9]
Epoch: 0 Loss: 5.8984375 Loss: 5.84375 Epoch: 1 Loss: 6.13671875 Loss: 4.12109375 Epoch: 2 Loss: 4.23046875 Loss: 4.0 Epoch: 3 Loss: 3.548828125 Loss: 3.345703125 Epoch: 4 Loss: 3.19921875 Loss: 2.767578125 Epoch: 5 Loss: 2.212890625 Loss: 1.6611328125 Epoch: 6 Loss: 1.8349609375 Loss: 1.0029296875 Epoch: 7 Loss: 1.3994140625 Loss: 1.6484375 Epoch: 8 Loss: 1.150390625 Loss: 1.3671875 Epoch: 9 Loss: 1.2900390625 Loss: 0.89990234375 Epoch: 10 Loss: 0.87744140625 Loss: 1.2373046875 Epoch: 11 Loss: 0.45947265625 Loss: 0.73828125 Epoch: 12 Loss: 0.73046875 Loss: 1.001953125 Epoch: 13 Loss: 0.407958984375 Loss: 0.61767578125 Epoch: 14 Loss: 0.3701171875 Loss: 0.61474609375 Epoch: 15 Loss: 0.833984375 Loss: 0.54638671875 Epoch: 16 Loss: 0.7841796875 Loss: 0.51318359375 Epoch: 17 Loss: 0.496337890625 Loss: 0.81591796875 Epoch: 18 Loss: 0.55029296875 Loss: 0.64404296875 Epoch: 19 Loss: 0.44189453125 Loss: 0.736328125 Epoch: 20 Loss: 0.428955078125 Loss: 0.26806640625 Epoch: 21 Loss: 0.61767578125 Loss: 0.39208984375 Epoch: 22 Loss: 0.2469482421875 Loss: 0.17578125 Epoch: 23 Loss: 0.50732421875 Loss: 0.407958984375 Epoch: 24 Loss: 0.2100830078125 Loss: 0.1507568359375 Epoch: 25 Loss: 0.2042236328125 Loss: 0.1383056640625 Epoch: 26 Loss: 0.365478515625 Loss: 0.404052734375 Epoch: 27 Loss: 0.39892578125 Loss: 0.336669921875 Epoch: 28 Loss: 0.3671875 Loss: 0.318115234375 Epoch: 29 Loss: 0.34326171875 Loss: 0.288330078125 Epoch: 30 Loss: 0.358154296875 Loss: 0.230224609375 Epoch: 31 Loss: 0.1097412109375 Loss: 0.1370849609375 Epoch: 32 Loss: 0.305908203125 Loss: 0.202880859375 Epoch: 33 Loss: 0.2242431640625 Loss: 0.2578125 Epoch: 34 Loss: 0.2354736328125 Loss: 0.2108154296875 Epoch: 35 Loss: 0.2236328125 Loss: 0.20751953125 Epoch: 36 Loss: 0.2261962890625 Loss: 0.171630859375 Epoch: 37 Loss: 0.1500244140625 Loss: 0.0928955078125 Epoch: 38 Loss: 0.1981201171875 Loss: 0.168212890625 Epoch: 39 Loss: 0.2115478515625 Loss: 0.1307373046875 Epoch: 40 Loss: 0.075439453125 Loss: 0.11053466796875 Epoch: 41 Loss: 0.07513427734375 Loss: 0.116455078125 Epoch: 42 Loss: 0.12164306640625 Loss: 0.170166015625 Epoch: 43 Loss: 0.1531982421875 Loss: 0.14501953125 Epoch: 44 Loss: 0.0966796875 Loss: 0.0709228515625 Epoch: 45 Loss: 0.055755615234375 Loss: 0.04937744140625 Epoch: 46 Loss: 0.08843994140625 Loss: 0.0589599609375 Epoch: 47 Loss: 0.055694580078125 Loss: 0.059051513671875 Epoch: 48 Loss: 0.1199951171875 Loss: 0.126953125 Epoch: 49 Loss: 0.101318359375 Loss: 0.1429443359375 Epoch: 50 Loss: 0.14306640625 Loss: 0.09930419921875 Epoch: 51 Loss: 0.09539794921875 Loss: 0.12249755859375 Epoch: 52 Loss: 0.06939697265625 Loss: 0.06854248046875 Epoch: 53 Loss: 0.1177978515625 Loss: 0.08428955078125 Epoch: 54 Loss: 0.108154296875 Loss: 0.08770751953125 Epoch: 55 Loss: 0.1002197265625 Loss: 0.0902099609375 Epoch: 56 Loss: 0.107421875 Loss: 0.071533203125 Epoch: 57 Loss: 0.08551025390625 Loss: 0.0997314453125 Epoch: 58 Loss: 0.09747314453125 Loss: 0.07208251953125 Epoch: 59 Loss: 0.033477783203125 Loss: 0.03790283203125 Epoch: 60 Loss: 0.06304931640625 Loss: 0.05072021484375 Epoch: 61 Loss: 0.04107666015625 Loss: 0.062347412109375 Epoch: 62 Loss: 0.056793212890625 Loss: 0.037872314453125 Epoch: 63 Loss: 0.06414794921875 Loss: 0.0963134765625 Epoch: 64 Loss: 0.07843017578125 Loss: 0.07122802734375 Epoch: 65 Loss: 0.0853271484375 Loss: 0.05902099609375 Epoch: 66 Loss: 0.08074951171875 Loss: 0.059844970703125 Epoch: 67 Loss: 0.05963134765625 Loss: 0.07269287109375 Epoch: 68 Loss: 0.035125732421875 Loss: 0.0509033203125 Epoch: 69 Loss: 0.0487060546875 Loss: 0.02960205078125 Epoch: 70 Loss: 0.047515869140625 Loss: 0.031036376953125 Epoch: 71 Loss: 0.024505615234375 Loss: 0.0286407470703125 Epoch: 72 Loss: 0.0584716796875 Loss: 0.05377197265625 Epoch: 73 Loss: 0.06536865234375 Loss: 0.046417236328125 Epoch: 74 Loss: 0.0265655517578125 Loss: 0.03814697265625 Epoch: 75 Loss: 0.0458984375 Loss: 0.059906005859375 Epoch: 76 Loss: 0.0191497802734375 Loss: 0.0211334228515625 Epoch: 77 Loss: 0.038177490234375 Loss: 0.0267181396484375 Epoch: 78 Loss: 0.040863037109375 Loss: 0.058441162109375 Epoch: 79 Loss: 0.053924560546875 Loss: 0.039642333984375 Epoch: 80 Loss: 0.041534423828125 Loss: 0.05572509765625 Epoch: 81 Loss: 0.03936767578125 Loss: 0.052459716796875 Epoch: 82 Loss: 0.0377197265625 Loss: 0.05426025390625 Epoch: 83 Loss: 0.0357666015625 Loss: 0.054168701171875 Epoch: 84 Loss: 0.0213775634765625 Loss: 0.0297393798828125 Epoch: 85 Loss: 0.031036376953125 Loss: 0.0242156982421875 Epoch: 86 Loss: 0.0189971923828125 Loss: 0.033782958984375 Epoch: 87 Loss: 0.016815185546875 Loss: 0.02264404296875 Epoch: 88 Loss: 0.02191162109375 Loss: 0.044769287109375 Epoch: 89 Loss: 0.036102294921875 Loss: 0.0218658447265625 Epoch: 90 Loss: 0.0163726806640625 Loss: 0.02069091796875 Epoch: 91 Loss: 0.051300048828125 Loss: 0.033782958984375 Epoch: 92 Loss: 0.0222625732421875 Loss: 0.03656005859375 Epoch: 93 Loss: 0.04345703125 Loss: 0.0474853515625 Epoch: 94 Loss: 0.047515869140625 Loss: 0.031829833984375 Epoch: 95 Loss: 0.037933349609375 Loss: 0.042449951171875 Epoch: 96 Loss: 0.019195556640625 Loss: 0.0291748046875 Epoch: 97 Loss: 0.0360107421875 Loss: 0.0389404296875 Epoch: 98 Loss: 0.04498291015625 Loss: 0.03240966796875 Epoch: 99 Loss: 0.01953125 Loss: 0.0301361083984375 Epoch: 100 Loss: 0.04443359375 Loss: 0.038665771484375 Epoch: 101 Loss: 0.04327392578125 Loss: 0.030548095703125 Epoch: 102 Loss: 0.0312042236328125 Loss: 0.040252685546875 Epoch: 103 Loss: 0.020355224609375 Loss: 0.0134735107421875 Epoch: 104 Loss: 0.040863037109375 Loss: 0.0361328125 Epoch: 105 Loss: 0.0423583984375 Loss: 0.03167724609375 Epoch: 106 Loss: 0.0333251953125 Loss: 0.04595947265625 Epoch: 107 Loss: 0.026458740234375 Loss: 0.018463134765625 Epoch: 108 Loss: 0.028350830078125 Loss: 0.041717529296875 Epoch: 109 Loss: 0.0282135009765625 Loss: 0.038543701171875 Epoch: 110 Loss: 0.0389404296875 Loss: 0.025848388671875 Epoch: 111 Loss: 0.0384521484375 Loss: 0.027313232421875 Epoch: 112 Loss: 0.0276031494140625 Loss: 0.03875732421875 Epoch: 113 Loss: 0.033050537109375 Loss: 0.0302581787109375 Epoch: 114 Loss: 0.0361328125 Loss: 0.02783203125 Epoch: 115 Loss: 0.03631591796875 Loss: 0.0252685546875 Epoch: 116 Loss: 0.022796630859375 Loss: 0.01430511474609375 Epoch: 117 Loss: 0.030914306640625 Loss: 0.027435302734375 Epoch: 118 Loss: 0.0258026123046875 Loss: 0.03466796875 Epoch: 119 Loss: 0.0123443603515625 Loss: 0.01148223876953125 Epoch: 120 Loss: 0.0248260498046875 Loss: 0.03369140625 Epoch: 121 Loss: 0.01428985595703125 Loss: 0.021331787109375 Epoch: 122 Loss: 0.031524658203125 Loss: 0.0257720947265625 Epoch: 123 Loss: 0.021026611328125 Loss: 0.033355712890625 Epoch: 124 Loss: 0.0269775390625 Loss: 0.0257110595703125 Epoch: 125 Loss: 0.0182037353515625 Loss: 0.01335906982421875 Epoch: 126 Loss: 0.018707275390625 Loss: 0.01153564453125 Epoch: 127 Loss: 0.013397216796875 Loss: 0.0190887451171875 Epoch: 128 Loss: 0.0213623046875 Loss: 0.028594970703125 Epoch: 129 Loss: 0.0271759033203125 Loss: 0.026214599609375 Epoch: 130 Loss: 0.0233612060546875 Loss: 0.0269317626953125 Epoch: 131 Loss: 0.00980377197265625 Loss: 0.01043701171875 Epoch: 132 Loss: 0.02825927734375 Loss: 0.01971435546875 Epoch: 133 Loss: 0.01197052001953125 Loss: 0.0190887451171875 Epoch: 134 Loss: 0.0185699462890625 Loss: 0.0122222900390625 Epoch: 135 Loss: 0.019439697265625 Loss: 0.01220703125 Epoch: 136 Loss: 0.0189208984375 Loss: 0.01122283935546875 Epoch: 137 Loss: 0.02947998046875 Loss: 0.018157958984375 Epoch: 138 Loss: 0.0196533203125 Loss: 0.02728271484375 Epoch: 139 Loss: 0.0266265869140625 Loss: 0.0185546875 Epoch: 140 Loss: 0.0099639892578125 Loss: 0.0158843994140625 Epoch: 141 Loss: 0.024322509765625 Loss: 0.02093505859375 Epoch: 142 Loss: 0.018524169921875 Loss: 0.0286407470703125 Epoch: 143 Loss: 0.00910186767578125 Loss: 0.016387939453125 Epoch: 144 Loss: 0.0212249755859375 Loss: 0.0223388671875 Epoch: 145 Loss: 0.0116119384765625 Loss: 0.0163421630859375 Epoch: 146 Loss: 0.025848388671875 Loss: 0.0186767578125 Epoch: 147 Loss: 0.01163482666015625 Loss: 0.01537322998046875 Epoch: 148 Loss: 0.0231475830078125 Loss: 0.0204620361328125 Epoch: 149 Loss: 0.0156707763671875 Loss: 0.01084136962890625 Epoch: 150 Loss: 0.0238494873046875 Loss: 0.017608642578125 Epoch: 151 Loss: 0.017181396484375 Loss: 0.0242767333984375 Epoch: 152 Loss: 0.018707275390625 Loss: 0.0234832763671875 Epoch: 153 Loss: 0.017303466796875 Loss: 0.0233917236328125 Epoch: 154 Loss: 0.01708984375 Loss: 0.0238800048828125 Epoch: 155 Loss: 0.00951385498046875 Loss: 0.014862060546875 Epoch: 156 Loss: 0.0253753662109375 Loss: 0.0163116455078125 Epoch: 157 Loss: 0.01971435546875 Loss: 0.02081298828125 Epoch: 158 Loss: 0.00997161865234375 Loss: 0.01543426513671875 Epoch: 159 Loss: 0.0164337158203125 Loss: 0.0222320556640625 Epoch: 160 Loss: 0.0083465576171875 Loss: 0.0078125 Epoch: 161 Loss: 0.0222320556640625 Loss: 0.01529693603515625 Epoch: 162 Loss: 0.0161895751953125 Loss: 0.0211181640625 Epoch: 163 Loss: 0.014373779296875 Loss: 0.0092010498046875 Epoch: 164 Loss: 0.01456451416015625 Loss: 0.021331787109375 Epoch: 165 Loss: 0.0214691162109375 Loss: 0.0190887451171875 Epoch: 166 Loss: 0.022186279296875 Loss: 0.01427459716796875 Epoch: 167 Loss: 0.007549285888671875 Loss: 0.00870513916015625 Epoch: 168 Loss: 0.0226898193359375 Loss: 0.01491546630859375 Epoch: 169 Loss: 0.00823974609375 Loss: 0.01337432861328125 Epoch: 170 Loss: 0.01471710205078125 Loss: 0.0204620361328125 Epoch: 171 Loss: 0.01399993896484375 Loss: 0.0214385986328125 Epoch: 172 Loss: 0.0063629150390625 Loss: 0.007556915283203125 Epoch: 173 Loss: 0.01430511474609375 Loss: 0.020904541015625 Epoch: 174 Loss: 0.0156402587890625 Loss: 0.01959228515625 Epoch: 175 Loss: 0.0210418701171875 Loss: 0.01389312744140625 Epoch: 176 Loss: 0.01236724853515625 Loss: 0.0083770751953125 Epoch: 177 Loss: 0.00662994384765625 Loss: 0.00736236572265625 Epoch: 178 Loss: 0.00799560546875 Loss: 0.006198883056640625 Epoch: 179 Loss: 0.020233154296875 Loss: 0.01413726806640625 Epoch: 180 Loss: 0.01389312744140625 Loss: 0.019195556640625 Epoch: 181 Loss: 0.00634765625 Loss: 0.006984710693359375 Epoch: 182 Loss: 0.02020263671875 Loss: 0.0140380859375 Epoch: 183 Loss: 0.01474761962890625 Loss: 0.01885986328125 Epoch: 184 Loss: 0.01605224609375 Loss: 0.01812744140625 Epoch: 185 Loss: 0.0134429931640625 Loss: 0.020172119140625 Epoch: 186 Loss: 0.01326751708984375 Loss: 0.0205078125 Epoch: 187 Loss: 0.00818634033203125 Loss: 0.01342010498046875 Epoch: 188 Loss: 0.00838470458984375 Loss: 0.01180267333984375 Epoch: 189 Loss: 0.0195465087890625 Loss: 0.01323699951171875 Epoch: 190 Loss: 0.0176239013671875 Loss: 0.01413726806640625 Epoch: 191 Loss: 0.01007080078125 Loss: 0.0119781494140625 Epoch: 192 Loss: 0.0160064697265625 Loss: 0.01702880859375 Epoch: 193 Loss: 0.0200347900390625 Loss: 0.016815185546875 Epoch: 194 Loss: 0.01248931884765625 Loss: 0.008514404296875 Epoch: 195 Loss: 0.021453857421875 Loss: 0.016387939453125 Epoch: 196 Loss: 0.00821685791015625 Loss: 0.00756072998046875 Epoch: 197 Loss: 0.0170135498046875 Loss: 0.018096923828125 Epoch: 198 Loss: 0.020538330078125 Loss: 0.01502227783203125 Epoch: 199 Loss: 0.016326904296875 Loss: 0.0178070068359375
Inference
Let's check the results on our train dataset
[10]
[11]
Benzema after Real Mardid's win against PSG with Real Mardid's win against PSG
Push to Hub
[13]
Token is valid. Your token has been saved in your configured git credential helpers (store). Your token has been saved to /root/.cache/huggingface/token Login successful
[14]
Upload 1 LFS files: 0%| | 0/1 [00:00<?, ?it/s]
adapter_model.bin: 0%| | 0.00/21.0M [00:00<?, ?B/s]
CommitInfo(commit_url='https://huggingface.co/ybelkada/blip2-opt-2.7b-football-captions-adapters/commit/94febcd3b3278e1dbd3d72f7902b03fedeb4bede', commit_message='Upload model', commit_description='', oid='94febcd3b3278e1dbd3d72f7902b03fedeb4bede', pr_url=None, pr_revision=None, pr_num=None)
Load from the Hub
Once trained you can push the model and processor on the Hub to use them later. Meanwhile you can play with the model that we have fine-tuned! Please restart the runtime to run the cell below!
[ ]
===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run python -m bitsandbytes and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues ================================================================================ bin /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0 CUDA SETUP: Highest compute capability among GPUs detected: 7.5 CUDA SETUP: Detected CUDA version 118 CUDA SETUP: Loading binary /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so...
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: /usr/lib64-nvidia did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('http'), PosixPath('//172.28.0.1'), PosixPath('8013')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('--listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-t4-s-24nx5ih8r1vcm --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//ipykernel.pylab.backend_inline')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
Downloading (…)/adapter_config.json: 0%| | 0.00/351 [00:00<?, ?B/s]
Overriding torch_dtype=None with `torch_dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don't pass this argument at all to remove this warning.
Downloading (…)lve/main/config.json: 0%| | 0.00/7.05k [00:00<?, ?B/s]
Downloading (…)model.bin.index.json: 0%| | 0.00/122k [00:00<?, ?B/s]
Downloading shards: 0%| | 0/8 [00:00<?, ?it/s]
Downloading (…)l-00001-of-00008.bin: 0%| | 0.00/995M [00:00<?, ?B/s]
Downloading (…)l-00002-of-00008.bin: 0%| | 0.00/999M [00:00<?, ?B/s]
Downloading (…)l-00003-of-00008.bin: 0%| | 0.00/985M [00:00<?, ?B/s]
Downloading (…)l-00004-of-00008.bin: 0%| | 0.00/997M [00:00<?, ?B/s]
Downloading (…)l-00005-of-00008.bin: 0%| | 0.00/997M [00:00<?, ?B/s]
Downloading (…)l-00006-of-00008.bin: 0%| | 0.00/997M [00:00<?, ?B/s]
Downloading (…)l-00007-of-00008.bin: 0%| | 0.00/997M [00:00<?, ?B/s]
Downloading (…)l-00008-of-00008.bin: 0%| | 0.00/782M [00:00<?, ?B/s]
Loading checkpoint shards: 0%| | 0/8 [00:00<?, ?it/s]
Downloading adapter_model.bin: 0%| | 0.00/21.0M [00:00<?, ?B/s]
Downloading (…)rocessor_config.json: 0%| | 0.00/432 [00:00<?, ?B/s]
Downloading (…)okenizer_config.json: 0%| | 0.00/904 [00:00<?, ?B/s]
Downloading (…)olve/main/vocab.json: 0%| | 0.00/798k [00:00<?, ?B/s]
Downloading (…)olve/main/merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s]
Downloading (…)/main/tokenizer.json: 0%| | 0.00/2.11M [00:00<?, ?B/s]
Downloading (…)cial_tokens_map.json: 0%| | 0.00/548 [00:00<?, ?B/s]
Let's check the results on our train dataset!
[5]
