Skip to content
GitLab
Explore
Projects
Groups
Topics
Snippets
Projects
Groups
Topics
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
某某某
transformers-new
Commits
dcb0e614
Unverified
Commit
dcb0e614
authored
4 years ago
by
Patrick von Platen
Committed by
GitHub
4 years ago
Browse files
Options
Download
Patches
Plain Diff
push (#11667)
parent
05a93067
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
src/transformers/models/auto/tokenization_auto.py
+2
-0
src/transformers/models/auto/tokenization_auto.py
with
2 additions
and
0 deletions
+2
-0
src/transformers/models/auto/tokenization_auto.py
+
2
−
0
View file @
dcb0e614
...
@@ -63,6 +63,7 @@ from .configuration_auto import (
...
@@ -63,6 +63,7 @@ from .configuration_auto import (
BertConfig
,
BertConfig
,
BertGenerationConfig
,
BertGenerationConfig
,
BigBirdConfig
,
BigBirdConfig
,
BigBirdPegasusConfig
,
BlenderbotConfig
,
BlenderbotConfig
,
BlenderbotSmallConfig
,
BlenderbotSmallConfig
,
CamembertConfig
,
CamembertConfig
,
...
@@ -275,6 +276,7 @@ TOKENIZER_MAPPING = OrderedDict(
...
@@ -275,6 +276,7 @@ TOKENIZER_MAPPING = OrderedDict(
(
Wav2Vec2Config
,
(
Wav2Vec2CTCTokenizer
,
None
)),
(
Wav2Vec2Config
,
(
Wav2Vec2CTCTokenizer
,
None
)),
(
GPTNeoConfig
,
(
GPT2Tokenizer
,
GPT2TokenizerFast
)),
(
GPTNeoConfig
,
(
GPT2Tokenizer
,
GPT2TokenizerFast
)),
(
LukeConfig
,
(
LukeTokenizer
,
None
)),
(
LukeConfig
,
(
LukeTokenizer
,
None
)),
(
BigBirdPegasusConfig
,
(
PegasusTokenizer
,
PegasusTokenizerFast
)),
]
]
)
)
...
...
This diff is collapsed.
Click to expand it.
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment
Menu
Explore
Projects
Groups
Topics
Snippets