hf-internal-testing
/

test_dynamic_tokenizer

Model card Files Files and versions

test_dynamic_tokenizer / tokenization.py

sgugger's picture

Initial commit

08a3578 almost 4 years ago

history blame contribute delete

112 Bytes

	from transformers import BertTokenizer


	class NewTokenizer(BertTokenizer):
	special_attribute_present =True