huihui-ai/Huihui-Mistral-Small-3.2-24B-Instruct-2506-abliterated
It's queued! :D
mradermacher was faster than me this time and already queued it as highly anticipated model.
You can check for progress at http://hf.tst.eu/status.html or regularly check the model
summary page at https://hf.tst.eu/model#Huihui-Mistral-Small-3.2-24B-Instruct-2506-abliterated-GGUF for quants to appear.
Hey, did something happen with this one? I don't see it in the queue and it's not got quants yet? No worries if I've just misunderstood the status page.
tokenizer.json is missing:
Huihui-Mistral-Small-3.2-24B-Instruct-2506-abliterated FileNotFoundError: [Errno 2] No such file or directory: 'Huihui-Mistral-Small-3.2-24B-Instruct-2506-abliterated/tokenizer.json'
Ah they use tekken.json so someone would need to create a llamacpp fixed version of it that uses a HuggingFace compatible tokenizer instead.
I actually in the past did so for the base model Huihui abliterated for this exact model: https://huggingface.co/nicoboss/Mistral-Small-3.2-24B-Instruct-2506-llamacppfixed
I created a llamacppfixed version of it under https://huggingface.co/nicoboss/Huihui-Mistral-Small-3.2-24B-Instruct-2506-abliterated-llamacppfixed
It's queued! :D
You can check for progress at http://hf.tst.eu/status.html or regularly check the model
summary page at https://hf.tst.eu/model#Huihui-Mistral-Small-3.2-24B-Instruct-2506-abliterated-llamacppfixed-GGUF for quants to appear.
what about regular Mistral-Small-3.2-24B-Instruct-2506?
edit: bartowski already quantized it (static quants): https://huggingface.co/bartowski/mistralai_Mistral-Small-3.2-24B-Instruct-2506-GGUF
bartowski's quants are imatrix quants btw.
what about regular Mistral-Small-3.2-24B-Instruct-2506?
edit: bartowski already quantized it (static quants): https://huggingface.co/bartowski/mistralai_Mistral-Small-3.2-24B-Instruct-2506-GGUF
@Notmebug We did Mistral-Small-3.2-24B-Instruct-2506 20 days ago:
Static quants: https://huggingface.co/mradermacher/Mistral-Small-3.2-24B-Instruct-2506-llamacppfixed-GGUF
Weighted/imatrix quants: https://huggingface.co/mradermacher/Mistral-Small-3.2-24B-Instruct-2506-llamacppfixed-i1-GGUF
bartowski's quants are imatrix quants btw.
They are but use a smaller dataset for imatrix trainin and so might be slightly less optimaly quantized. On the other hand he uses custom quantisation recepies that trade off some size for better quality which results in quite unique non-standard quants with thair own quality/size trade-offs so I see why some prefer them for specific use cases. I have no clue what approach he used to make Mistral-Small-3.2-24B-Instruct-2506 llama.cpp compatible.