Export
标记器（Tokenizer） (PyTorch)
Install the Transformers, Datasets, and Evaluate libraries to run this notebook.
[ ]
[ ]
['Jim', 'Henson', 'was', 'a', 'puppeteer']
[ ]
[ ]
[ ]
{'input_ids': [101, 7993, 170, 11303, 1200, 2443, 1110, 3014, 102],
, 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0],
, 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]} [ ]
[ ]
['Using', 'a', 'transform', '##er', 'network', 'is', 'simple']
[ ]
[7993, 170, 11303, 1200, 2443, 1110, 3014]
[ ]
'Using a Transformer network is simple'