• transformers :bert1


    from transformers import pipeline
    
    
    from transformers import AutoTokenizer
    
    
    tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
    
    # encoded_input = tokenizer("Hello, I'm a single sentence!")
    # encoded_input = tokenizer("你好我是你爸爸")
    
    batch_sentences = ["Hello I'm a single sentence",
                       "And another sentence",
                       "And the very very last one"]
    encoded_inputs = tokenizer(batch_sentences)
    print(encoded_inputs)
    {'input_ids': [[101, 8667, 146, 112, 182, 170, 1423, 5650, 102], [101, 1262, 1330, 5650, 102], [101, 1262, 1103, 1304, 1304, 1314, 1141, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1]]}


    注意这里tokenizer在encode之后返回了三个部分:

      • input_ids: 相当于是分词之后每个token转变成了一个id
      • token_type_ids: 我们知道BERT模型允许我们传入两个sequence。而这个token_type_id就表示当前的token究竟是第一个sequence还是第二个sequence
      • attention_mask: 表示当前的位置是真正的token还是只是padding而已。
  • 相关阅读:
    MongoDB知识
    SELECT INTO 和 INSERT INTO SELECT 两种表复制语句
    SQL Server 触发器
    TSQL游标使用
    python和C#的区别
    bcp和SqlDataAdapter进行批量跟新插入方法
    SQL Server中索引视图用法详解
    hdu 1950 Bridging signals
    UVA 116 Unidirectional TSP
    poj 3230 Travel
  • 原文地址:https://www.cnblogs.com/DDBD/p/14142525.html
Copyright © 2020-2023  润新知