Skip to content
GitLab
Commit b6ae24fc authored by Ita Zaporozhets's avatar Ita Zaporozhets
Browse files

check if repo has tokenizer.json file, then load PreTrainedTokenizerFast

parent 88b140e8
Hide whitespace changes
Inline Side-by-side
Showing
with 102 additions and 3 deletions
+102 -3
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment