• Junyang Lin's avatar
    Add qwen2 (#28436) · d6ffe74d
    Junyang Lin authored
    
    
    * add config, modeling, and tokenization
    
    * add auto and init
    
    * update readme
    
    * update readme
    
    * update team name
    
    * fixup
    
    * fixup
    
    * update config
    
    * update code style
    
    * update for fixup
    
    * update for fixup
    
    * update for fixup
    
    * update for testing
    
    * update for testing
    
    * fix bug for config and tokenization
    
    * fix bug for bos token
    
    * not doctest
    
    * debug tokenizer
    
    * not doctest
    
    * debug tokenization
    
    * debug init for tokenizer
    
    * fix style
    
    * update init
    
    * delete if in token auto
    
    * add tokenizer doc
    
    * add tokenizer in init
    
    * Update dummy_tokenizers_objects.py
    
    * update
    
    * update
    
    * debug
    
    * Update tokenization_qwen2.py
    
    * debug
    
    * Update convert_slow_tokenizer.py
    
    * add copies
    
    * add copied from and make style
    
    * update files map
    
    * update test
    
    * fix style
    
    * fix merge reading and update tests
    
    * fix tests
    
    * fix tests
    
    * fix style
    
    * debug a variable in readme
    
    * Update src/transformers/models/qwen2/configuration_qwen2.py
    
    Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
    
    * update test and copied from
    
    * fix style
    
    * update qwen2 tokenization  and tests
    
    * Update tokenization_qwen2.py
    
    * delete the copied from after property
    
    * fix style
    
    * update tests
    
    * update tests
    
    * add copied from
    
    * fix bugs
    
    * update doc
    
    * add warning for sliding window attention
    
    * update qwen2 tokenization
    
    * fix style
    
    * Update src/transformers/models/qwen2/modeling_qwen2.py
    
    Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
    
    * fix tokenizer fast
    
    ---------
    
    Co-authored-by: default avatarRen Xuancheng <jklj077@users.noreply.github.com>
    Co-authored-by: default avatarrenxuancheng.rxc <renxuancheng.rxc@alibaba-inc.com>
    Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
    d6ffe74d