Models
Datasets
Spaces
Docs
Enterprise
Pricing
Log In
Sign Up

SaulLu
/

bengali-tokenizer

Model card Files Files and versions

bengali-tokenizer

1.72 MB

1 contributor

History: 4 commits

SaulLu's picture

set normalized to true

524ccfa over 4 years ago

.gitattributes

690 Bytes

initial commit over 4 years ago
special_tokens_map.json

321 Bytes

set normalized to true over 4 years ago
tokenizer.json

1.72 MB

set normalized to true over 4 years ago
tokenizer_config.json

467 Bytes

tokenizer v2- include normalization discussed with Bengali community over 4 years ago