ubergarm
/

Qwen3-235B-A22B-GGUF

Text Generation

Model card Files Files and versions

ubergarm commited on Jul 22

Commit

32f2b1c

·

1 Parent(s): f956ece

massage command

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -49,7 +49,7 @@ Final estimate: PPL = 5.4403 +/- 0.03421 (wiki.test.raw, compare to Q8_0 at 5.31
 # This example for 24GB VRAM + 96 GB RAM + 16 physical core CPU
 # Offload first ffn layers  0-9 on GPU VRAM.
 # Leave remaining ffn layers on CPU RAM.
-./build/bin/llama-server
   --model ubergarm/Qwen3-235B-A22B-GGUF/Qwen3-235B-A22B-mix-IQ3_K-00001-of-00003.gguf \
   --alias ubergarm/Qwen3-235B-A22B-mix-IQ3_K \
   -fa -fmoe \
@@ -59,7 +59,7 @@ Final estimate: PPL = 5.4403 +/- 0.03421 (wiki.test.raw, compare to Q8_0 at 5.31
   -ot blk\.[0-9]\.ffn.*=CUDA0 \
   -ot "blk.*\.ffn.*=CPU \
   -ngl 99 \
-  --threads 16
   -ub 4096 -b 4096 \
   --host 127.0.0.1 \
   --port 8080

 # This example for 24GB VRAM + 96 GB RAM + 16 physical core CPU
 # Offload first ffn layers  0-9 on GPU VRAM.
 # Leave remaining ffn layers on CPU RAM.
+./build/bin/llama-server \
   --model ubergarm/Qwen3-235B-A22B-GGUF/Qwen3-235B-A22B-mix-IQ3_K-00001-of-00003.gguf \
   --alias ubergarm/Qwen3-235B-A22B-mix-IQ3_K \
   -fa -fmoe \
   -ot blk\.[0-9]\.ffn.*=CUDA0 \
   -ot "blk.*\.ffn.*=CPU \
   -ngl 99 \
+  --threads 16 \
   -ub 4096 -b 4096 \
   --host 127.0.0.1 \
   --port 8080