deepseek-ai

/

DeepSeek-V3

like 4.03k

Follow

DeepSeek 120k

Your current tokenizer config:

"unk_token": null

This means there's no defined "unknown token," which is risky — the tokenizer can't handle out-of-vocabulary (OOV) tokens properly.

This update defines the unk_token, enabling the tokenizer to:

1. Prevent crashes or undefined behavior when unknown tokens are encountered.
2. Ensure compatibility with libraries that expect a defined unk_token.

For those asking about API access — I've been using Crazyrouter as a unified gateway. One API key, OpenAI SDK compatible. Works well for testing different models without managing multiple accounts.