from transformers import pipeline from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained('bert-base-cased') # encoded_input = tokenizer("Hello, I'm a single sentence!") # encoded_input = tokenizer("你好我是你爸爸") batch_sentences = ["Hello I'm a single sentence", "And another sentence", "And the very very last one"] encoded_inputs = tokenizer(batch_sentences) print(encoded_inputs)
{'input_ids': [[101, 8667, 146, 112, 182, 170, 1423, 5650, 102], [101, 1262, 1330, 5650, 102], [101, 1262, 1103, 1304, 1304, 1314, 1141, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1]]}
注意这里tokenizer在encode之后返回了三个部分:
- input_ids: 相当于是分词之后每个token转变成了一个id
- token_type_ids: 我们知道BERT模型允许我们传入两个sequence。而这个token_type_id就表示当前的token究竟是第一个sequence还是第二个sequence
- attention_mask: 表示当前的位置是真正的token还是只是padding而已。