Does it work?
Hi! I've downloaded Q8_0 version to test, but I cannot get any real output - it's either ++++++ or something similar. Previous aquif-3.5-8B-Think.Q8_0 works correctly, and other models too.
I use ChatML template, as written in the tokenizer_config.json of the model. It it only for me?
Unfortunately, I can report it is only you. It seems to work fine for me. Are you using llama.cpp or something else? For quite a while now, it seems that some other inference engines have trouble with newer models by ignoring important config parameters (such as the chat template). When in doubt, always use the chat template from the model, don't supply your own unless you know what you are doing.
> Hi!
The user simply greeted me with "Hi!", which is a common opening and requires no deeper analysis or multi-step reasoning to respond appropriately. This is a straightforward interaction that doesn't demand complex thought.
Hello! How can I assist you today? If you have any questions or need help with something, feel free to let me know.
> can you write a limerick about the hardships of modern internet life?
The core task is to compose a limerick on a specific theme, which involves basic creative writing without deep analysis or multi-step reasoning. This is a straightforward task that doesn't require complex thought processes.
Certainly! Here's a limerick about the hardships of modern internet life:
"Modern life's a digital maze,
Where screens glow 'til the break of day.
We scroll 'til we're drained,
By likes and by shares,
Lost in the noise, we can't find our way."
Unfortunately, I can report it is only you. It seems to work fine for me. Are you using llama.cpp or something else? For quite a while now, it seems that some other inference engines have trouble with newer models by ignoring important config parameters (such as the chat template). When in doubt, always use the chat template from the model, don't supply your own unless you know what you are doing.
I use a llama.cpp-based app, and I manually set the chat template. It works with all other models so far, interesting why this one is special. Thanks for the answer - I will look on my side.
The above output was genetrated with llama-cli --jinja -cnv and no other special options, to give you a baseline that works.