hf-internal-testing
/

test_dynamic_tokenizer

Model card Files Files and versions

test_dynamic_tokenizer / tokenization_fast.py

sgugger's picture

Initial commit

08a3578 almost 4 years ago

history blame contribute delete

202 Bytes

	from transformers import BertTokenizerFast
	from .tokenization import NewTokenizer

	class NewTokenizerFast(BertTokenizerFast):
	slow_tokenizer_class = NewTokenizer
	special_attribute_present =True