Based on Unsloth's original BF16 GGUF.

The real-world test problem is from https://www.mathe-wettbewerbe.de/fileadmin/Mathe-Wettbewerbe/Bundeswettbewerb_Mathematik/Dokumente/Aufgaben_und_Loesungen_BWM/2025/BWM_25_2_Aufgabenblatt.pdf (published in Sept. 2025, the answer is available in Oct. 2025. So I believe the LLM does not take this problem in the training set.)

./build/bin/llama-server -m /media/lovedheart/8EC4693CC46927A3/IQ1_M/Qwen3-VL-235B-A22B-Thinking-IQ1_M-00001-of-00039.gguf --mmproj /media/lovedheart/8EC4693CC46927A3/IQ1_M/mmproj-F16.gguf -c 262144 -ot "emb=Vulkan0"

ggml_vulkan: Found 1 Vulkan devices: ggml_vulkan: 0 = AMD Radeon Graphics (RADV PHOENIX) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat main: setting n_parallel = 4 and kv_unified = true (add -kvu to disable this) build: 6964 (22c8c3c6a) with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu system info: n_threads = 8, n_threads_batch = 8, total_threads = 16 system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

main: binding port with default address family main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 15 main: loading model srv load_model: loading model '/media/lovedheart/8EC4693CC46927A3/IQ1_M/Qwen3-VL-235B-A22B-Thinking-IQ1_M-00001-of-00039.gguf' llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon Graphics (RADV PHOENIX)) (0000:c7:00.0) - 119581 MiB free llama_model_loader: additional 38 GGUFs metadata loaded. llama_model_loader: loaded meta data with 48 key-value pairs and 1131 tensors from /media/lovedheart/8EC4693CC46927A3/IQ1_M/Qwen3-VL-235B-A22B-Thinking-IQ1_M-00001-of-00039.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen3vlmoe llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Qwen3-Vl-235B-A22B-Thinking llama_model_loader: - kv 3: general.finetune str = Thinking llama_model_loader: - kv 4: general.basename str = Qwen3-Vl-235B-A22B-Thinking llama_model_loader: - kv 5: general.quantized_by str = Unsloth llama_model_loader: - kv 6: general.size_label str = 235B-A22B llama_model_loader: - kv 7: general.license str = apache-2.0 llama_model_loader: - kv 8: general.repo_url str = https://huggingface.co/unsloth llama_model_loader: - kv 9: general.base_model.count u32 = 1 llama_model_loader: - kv 10: general.base_model.0.name str = Qwen3 VL 235B A22B Thinking llama_model_loader: - kv 11: general.base_model.0.organization str = Qwen llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3-VL-... llama_model_loader: - kv 13: general.tags arr[str,1] = ["unsloth"] llama_model_loader: - kv 14: qwen3vlmoe.block_count u32 = 94 llama_model_loader: - kv 15: qwen3vlmoe.context_length u32 = 262144 llama_model_loader: - kv 16: qwen3vlmoe.embedding_length u32 = 4096 llama_model_loader: - kv 17: qwen3vlmoe.feed_forward_length u32 = 12288 llama_model_loader: - kv 18: qwen3vlmoe.attention.head_count u32 = 64 llama_model_loader: - kv 19: qwen3vlmoe.attention.head_count_kv u32 = 4 llama_model_loader: - kv 20: qwen3vlmoe.rope.freq_base f32 = 5000000.000000 llama_model_loader: - kv 21: qwen3vlmoe.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 22: qwen3vlmoe.expert_used_count u32 = 8 llama_model_loader: - kv 23: qwen3vlmoe.attention.key_length u32 = 128 llama_model_loader: - kv 24: qwen3vlmoe.attention.value_length u32 = 128 llama_model_loader: - kv 25: qwen3vlmoe.expert_count u32 = 128 llama_model_loader: - kv 26: qwen3vlmoe.expert_feed_forward_length u32 = 1536 llama_model_loader: - kv 27: qwen3vlmoe.rope.dimension_sections arr[i32,4] = [24, 20, 20, 0] llama_model_loader: - kv 28: qwen3vlmoe.n_deepstack_layers u32 = 3 llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 30: tokenizer.ggml.pre str = qwen2 llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,151936] = ["!", """, "#", "$", "%", "&", "'", ... llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 34: tokenizer.ggml.eos_token_id u32 = 151645 llama_model_loader: - kv 35: tokenizer.ggml.padding_token_id u32 = 151654 llama_model_loader: - kv 36: tokenizer.ggml.bos_token_id u32 = 151643 llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 38: tokenizer.chat_template str = {# Unsloth template fixes #}\n{%- set ... llama_model_loader: - kv 39: general.quantization_version u32 = 2 llama_model_loader: - kv 40: general.file_type u32 = 31 llama_model_loader: - kv 41: quantize.imatrix.file str = E:\LLM\Qwen3-VL-235B-A22B-Thinking-GG... llama_model_loader: - kv 42: quantize.imatrix.dataset str = unsloth_calibration_Qwen3-VL-235B-A22... llama_model_loader: - kv 43: quantize.imatrix.entries_count u32 = 752 llama_model_loader: - kv 44: quantize.imatrix.chunks_count u32 = 79 llama_model_loader: - kv 45: split.no u16 = 0 llama_model_loader: - kv 46: split.tensors.count i32 = 1131 llama_model_loader: - kv 47: split.count u16 = 39 llama_model_loader: - type f32: 471 tensors llama_model_loader: - type q2_K: 55 tensors llama_model_loader: - type q4_K: 295 tensors llama_model_loader: - type q6_K: 65 tensors llama_model_loader: - type iq2_xxs: 36 tensors llama_model_loader: - type iq3_xxs: 14 tensors llama_model_loader: - type iq1_m: 160 tensors llama_model_loader: - type bf16: 35 tensors print_info: file format = GGUF V3 (latest) print_info: file type = IQ1_M - 1.75 bpw print_info: file size = 67.02 GiB (2.45 BPW) load: printing all EOG tokens: load: - 151643 ('<|endoftext|>') load: - 151645 ('<|im_end|>') load: - 151662 ('<|fim_pad|>') load: - 151663 ('<|repo_name|>') load: - 151664 ('<|file_sep|>') load: special tokens cache size = 26 load: token to piece cache size = 0.9311 MB print_info: arch = qwen3vlmoe print_info: vocab_only = 0 print_info: n_ctx_train = 262144 print_info: n_embd = 16384 print_info: n_layer = 94 print_info: n_head = 64 print_info: n_head_kv = 4 print_info: n_rot = 128 print_info: n_swa = 0 print_info: is_swa_any = 0 print_info: n_embd_head_k = 128 print_info: n_embd_head_v = 128 print_info: n_gqa = 16 print_info: n_embd_k_gqa = 512 print_info: n_embd_v_gqa = 512 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-06 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: f_attn_scale = 0.0e+00 print_info: n_ff = 12288 print_info: n_expert = 128 print_info: n_expert_used = 8 print_info: n_expert_groups = 0 print_info: n_group_used = 0 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 40 print_info: rope scaling = linear print_info: freq_base_train = 5000000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 262144 print_info: rope_finetuned = unknown print_info: mrope sections = [24, 20, 20, 0] print_info: model type = 235B.A22B print_info: model params = 235.09 B print_info: general.name = Qwen3-Vl-235B-A22B-Thinking print_info: n_ff_exp = 1536 print_info: vocab type = BPE print_info: n_vocab = 151936 print_info: n_merges = 151387 print_info: BOS token = 151643 '<|endoftext|>' print_info: EOS token = 151645 '<|im_end|>' print_info: EOT token = 151645 '<|im_end|>' print_info: PAD token = 151654 '<|vision_pad|>' print_info: LF token = 198 'Ċ' print_info: FIM PRE token = 151659 '<|fim_prefix|>' print_info: FIM SUF token = 151661 '<|fim_suffix|>' print_info: FIM MID token = 151660 '<|fim_middle|>' print_info: FIM PAD token = 151662 '<|fim_pad|>' print_info: FIM REP token = 151663 '<|repo_name|>' print_info: FIM SEP token = 151664 '<|file_sep|>' print_info: EOG token = 151643 '<|endoftext|>' print_info: EOG token = 151645 '<|im_end|>' print_info: EOG token = 151662 '<|fim_pad|>' print_info: EOG token = 151663 '<|repo_name|>' print_info: EOG token = 151664 '<|file_sep|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = true) load_tensors: offloading 94 repeating layers to GPU load_tensors: offloading output layer to GPU load_tensors: offloaded 95/95 layers to GPU load_tensors: Vulkan0 model buffer size = 68627.51 MiB .................................................................................................. llama_context: constructing llama_context llama_context: n_seq_max = 4 llama_context: n_ctx = 262144 llama_context: n_ctx_seq = 262144 llama_context: n_batch = 2048 llama_context: n_ubatch = 512 llama_context: causal_attn = 1 llama_context: flash_attn = auto llama_context: kv_unified = true llama_context: freq_base = 5000000.0 llama_context: freq_scale = 1 llama_context: Vulkan_Host output buffer size = 2.32 MiB llama_kv_cache: Vulkan0 KV buffer size = 48128.00 MiB llama_kv_cache: size = 48128.00 MiB (262144 cells, 94 layers, 4/1 seqs), K (f16): 24064.00 MiB, V (f16): 24064.00 MiB llama_context: Flash Attention was auto, set to enabled llama_context: Vulkan0 compute buffer size = 816.02 MiB llama_context: Vulkan_Host compute buffer size = 512.02 MiB llama_context: graph nodes = 5929 llama_context: graph splits = 1 common_init_from_params: added <|endoftext|> logit bias = -inf common_init_from_params: added <|im_end|> logit bias = -inf common_init_from_params: added <|fim_pad|> logit bias = -inf common_init_from_params: added <|repo_name|> logit bias = -inf common_init_from_params: added <|file_sep|> logit bias = -inf common_init_from_params: setting dry_penalty_last_n to ctx_size = 262144 common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) clip_model_loader: model name: Qwen3-Vl-235B-A22B-Thinking clip_model_loader: description:
clip_model_loader: GGUF version: 3 clip_model_loader: alignment: 32 clip_model_loader: n_tensors: 352 clip_model_loader: n_kv: 31

clip_model_loader: has vision encoder clip_ctx: CLIP using Vulkan0 backend load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024 load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

load_hparams: projector: qwen3vl_merger load_hparams: n_embd: 1152 load_hparams: n_head: 16 load_hparams: n_ff: 4304 load_hparams: n_layer: 27 load_hparams: ffn_op: gelu load_hparams: projection_dim: 4096

--- vision hparams --- load_hparams: image_size: 768 load_hparams: patch_size: 16 load_hparams: has_llava_proj: 0 load_hparams: minicpmv_version: 0 load_hparams: n_merge: 2 load_hparams: n_wa_pattern: 0 load_hparams: image_min_pixels: 8192 load_hparams: image_max_pixels: 4194304

load_hparams: model size: 1105.32 MiB load_hparams: metadata size: 0.12 MiB alloc_compute_meta: warmup with image size = 1472 x 1472 alloc_compute_meta: Vulkan0 compute buffer size = 372.08 MiB alloc_compute_meta: CPU compute buffer size = 24.93 MiB alloc_compute_meta: graph splits = 1, nodes = 853 warmup: flash attention is enabled srv load_model: loaded multimodal model, '/media/lovedheart/8EC4693CC46927A3/IQ1_M/mmproj-F16.gguf' srv init: initializing slots, n_slots = 4 slot init: id 0 | task -1 | new slot, n_ctx = 262144 slot init: id 1 | task -1 | new slot, n_ctx = 262144 slot init: id 2 | task -1 | new slot, n_ctx = 262144 slot init: id 3 | task -1 | new slot, n_ctx = 262144 srv init: prompt cache is enabled, size limit: 8192 MiB srv init: use --cache-ram 0 to disable the prompt cache srv init: for more info see https://github.com/ggml-org/llama.cpp/pull/16391 srv init: thinking = 0 main: model loaded main: chat template, chat_template: {# Unsloth template fixes #} {%- set image_count = namespace(value=0) %} {%- set video_count = namespace(value=0) %} {%- macro render_content(content, do_vision_count) %} {%- if content is string %} {{- content }} {%- else %} {%- for item in content %} {%- if 'image' in item or 'image_url' in item or item.type == 'image' %} {%- if do_vision_count %} {%- set image_count.value = image_count.value + 1 %} {%- endif %} {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%} <|vision_start|><|image_pad|><|vision_end|> {%- elif 'video' in item or item.type == 'video' %} {%- if do_vision_count %} {%- set video_count.value = video_count.value + 1 %} {%- endif %} {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%} <|vision_start|><|video_pad|><|vision_end|> {%- elif 'text' in item %} {{- item.text }} {%- endif %} {%- endfor %} {%- endif %} {%- endmacro %} {%- if tools %} {{- '<|im_start|>system\n' }} {%- if messages[0].role == 'system' %} {{- render_content(messages[0].content, false) + '\n\n' }} {%- endif %} {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} {%- for tool in tools %} {{- "\n" }} {{- tool | tojson }} {%- endfor %} {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{"name": , "arguments": }\n<|im_end|>\n" }} {%- else %} {%- if messages[0].role == 'system' %} {{- '<|im_start|>system\n' + render_content(messages[0].content, false) + '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} {%- for message in messages[::-1] %} {%- set index = (messages|length - 1) - loop.index0 %} {%- if ns.multi_step_tool and message.role == "user" %} {%- set content = render_content(message.content, false) %} {%- if not(content.startswith('') and content.endswith('')) %} {%- set ns.multi_step_tool = false %} {%- set ns.last_query_index = index %} {%- endif %} {%- endif %} {%- endfor %} {%- for message in messages %} {%- set content = render_content(message.content, True) %} {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} {%- elif message.role == "assistant" %} {%- set reasoning_content = '' %} {%- if message.reasoning_content is string %} {%- set reasoning_content = message.reasoning_content %} {%- else %} {%- if '' in content %} {# Unsloth template fixes - must change to for loop since llama.cpp will error out if not #} {%- set parts = content.split('') %} {%- for part in parts %} {%- if loop.index0 == 0 -%} {%- set reasoning_content = part.rstrip('\n') %} {%- set reasoning_content = (reasoning_content.split('')|last) %} {%- set reasoning_content = reasoning_content.lstrip('\n') -%} {%- else -%} {%- set content = part.lstrip('\n') %} {%- endif %} {%- endfor %} {%- endif %} {%- endif %} {%- if loop.index0 > ns.last_query_index %} {%- if loop.last or (not loop.last and reasoning_content) %} {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }} {%- else %} {{- '<|im_start|>' + message.role + '\n' + content }} {%- endif %} {%- else %} {{- '<|im_start|>' + message.role + '\n' + content }} {%- endif %} {%- if message.tool_calls %} {%- for tool_call in message.tool_calls %} {%- if (loop.first and content) or (not loop.first) %} {{- '\n' }} {%- endif %} {%- if tool_call.function %} {%- set tool_call = tool_call.function %} {%- endif %} {{- '\n{"name": "' }} {{- tool_call.name }} {{- '", "arguments": ' }} {%- if tool_call.arguments is string %} {{- tool_call.arguments }} {%- else %} {{- tool_call.arguments | tojson }} {%- endif %} {{- '}\n' }} {%- endfor %} {%- endif %} {{- '<|im_end|>\n' }} {%- elif message.role == "tool" %} {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} {{- '<|im_start|>user' }} {%- endif %} {{- '\n\n' }} {{- content }} {{- '\n' }} {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} {{- '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- endfor %} {%- if add_generation_prompt %} {{- '<|im_start|>assistant\n\n' }} {%- endif %} {# Copyright 2025-present Unsloth. Apache 2.0 License. #}, example_format: '<|im_start|>system You are a helpful assistant<|im_end|> <|im_start|>user Hello<|im_end|> <|im_start|>assistant Hi there<|im_end|> <|im_start|>user How are you?<|im_end|> <|im_start|>assistant ' main: server is listening on http://127.0.0.1:8080 - starting the main loop srv update_slots: all slots are idle srv log_server_r: request: GET / 127.0.0.1 200 srv log_server_r: request: GET /props 127.0.0.1 200 srv log_server_r: request: GET /props 127.0.0.1 200 srv log_server_r: request: GET /props 127.0.0.1 200 srv update_slots: all slots are idle srv log_server_r: request: GET /slots 127.0.0.1 200 srv log_server_r: request: GET /props 127.0.0.1 200 srv log_server_r: request: GET /props 127.0.0.1 200 srv params_from_: Chat format: Content-only slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1 slot launch_slot_: id 3 | task 1 | processing task slot update_slots: id 3 | task 1 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 641 slot update_slots: id 3 | task 1 | n_tokens = 0, memory_seq_rm [0, end) slot update_slots: id 3 | task 1 | prompt processing progress, n_tokens = 11, batch.n_tokens = 11, progress = 0.017161 slot update_slots: id 3 | task 1 | n_tokens = 11, memory_seq_rm [11, end) srv process_chun: processing image... encoding image slice... image slice encoded in 1612 ms decoding image batch 1/1, n_tokens_batch = 624 image decoded (batch 1/1) in 23441 ms srv process_chun: image processed in 25053 ms slot update_slots: id 3 | task 1 | prompt processing progress, n_tokens = 641, batch.n_tokens = 6, progress = 1.000000 slot update_slots: id 3 | task 1 | prompt done, n_tokens = 641, batch.n_tokens = 6 srv log_server_r: request: GET /props 127.0.0.1 200 srv log_server_r: request: GET /slots 127.0.0.1 200 srv log_server_r: request: GET /slots 127.0.0.1 200 slot print_timing: id 3 | task 1 |

prompt eval time = 28184.45 ms / 641 tokens ( 43.97 ms per token, 22.74 tokens per second) eval time = 1887509.92 ms / 10000 tokens ( 188.75 ms per token, 5.30 tokens per second) total time = 1915694.37 ms / 10641 tokens

slot release: id 3 | task 1 | stop processing: n_tokens = 10640, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 127.0.0.1 200

Some screen shots: