01 Text Classification
hf-blogtrainerhacktoberfestnotebooks
Export
Trainer: GLUE MNLI example, the Colab version 🔥
Install transformers from master, and also clone the repo to get some utility files
[0]
[0]
Check that we have a GPU and check its memory size (depending on its RAM size you can change the batch sizes below)
[25]
Thu May 7 16:49:22 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |
| N/A 51C P0 38W / 250W | 2871MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
All imports are here:
[0]
We use dataclass-based configuration objects, let's define the one related to which model we are going to train here:
[0]
Here are all the training parameters we are going to use:
[0]
[0]
We fine-tune on MNLI so let's find out the number of labels:
[30]
3
🤗 Now we can instantiate our config, our tokenizer, and our model
[0]
[0]
We need to define a task-specific way of computing relevant metrics (see more details in the Trainer class):
[0]
We are now ready to initialize our Trainer
[0]
Launching the training is as simple is doing trainer.train() ♥️
[33]
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-33-0c647bc3a8b8> in <module>() ----> 1 get_ipython().run_cell_magic('time', '', 'trainer.train()') /usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell) 2115 magic_arg_s = self.var_expand(line, stack_depth) 2116 with self.builtin_trap: -> 2117 result = fn(magic_arg_s, cell) 2118 return result 2119 <decorator-gen-60> in time(self, line, cell, local_ns) /usr/local/lib/python3.6/dist-packages/IPython/core/magic.py in <lambda>(f, *a, **k) 186 # but it's overkill for just that one bit of state. 187 def magic_deco(arg): --> 188 call = lambda f, *a, **k: f(*a, **k) 189 190 if callable(arg): /usr/local/lib/python3.6/dist-packages/IPython/core/magics/execution.py in time(self, line, cell, local_ns) 1187 if mode=='eval': 1188 st = clock2() -> 1189 out = eval(code, glob, local_ns) 1190 end = clock2() 1191 else: <timed eval> in <module>() /content/transformers/src/transformers/trainer.py in train(self, model_path) 380 continue 381 --> 382 tr_loss += self._training_step(model, inputs, optimizer) 383 384 if (step + 1) % self.args.gradient_accumulation_steps == 0 or ( /content/transformers/src/transformers/trainer.py in _training_step(self, model, inputs, optimizer) 477 scaled_loss.backward() 478 else: --> 479 loss.backward() 480 481 return loss.item() /usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph) 196 products. Defaults to ``False``. 197 """ --> 198 torch.autograd.backward(self, gradient, retain_graph, create_graph) 199 200 def register_hook(self, hook): /usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables) 98 Variable._execution_engine.run_backward( 99 tensors, grad_tensors, retain_graph, create_graph, --> 100 allow_unreachable=True) # allow_unreachable flag 101 102 KeyboardInterrupt:
Check that our training was successful using TensorBoard
[0]
[36]
Reusing TensorBoard on port 6006 (pid 1059), started 0:01:32 ago. (Use '!kill 1059' to kill it.)
<IPython.core.display.Javascript object>
🎉 Yeah it's training!
[0]