ubergarm

/

GLM-4.6-GGUF

like 27

Thanks for sharing your results! The command looks reasonable and cool you can hit 8192 batch sizes (some folks have trouble with higher sizes on their rigs).

CPU power capped at 280 W, GPUs at 450 W

The only tip I can think of off-hand is to consider not power capping your GPUs with nvidia-smi -pl 450 or whatever and instead use the LACT "indirect undervolt" method assuming Linux or something like MSI Afterburner on Windoze.

On Beaver AI Discord a guy has an RTX 6000 PRO and said by locking the boost clock and setting 1000MHz offset (which seems scaled in Blackwell vs older arch GPUs) that it runs at same performance and never goes much over 300W anyway. I have a post on Reddit, l1t, and there is a good github issue thread describing it on the LACT github repo. It works on headless systems as well and can be persisted across boots. @Panchovix helped spread this method and explain how the Blackwell GPUs use like 10x the offset as earlier models for some reason.

https://github.com/ilya-zlobintsev/LACT/issues/486#issuecomment-3303917115

Thank you @ubergarm for the tip about LACT, I'll sure check it out!

I started power-limiting when some people stressed me out here about my 2200W PSU, claiming it is not strong enough for 2x CPU, 2x GPU, 4x HDD and 3x SSD 😀. Although I disagree, I decided that putting some power limits in place might not be a bad idea...

Yeah, higher -ub values work well on my machine. Here I tested -ub 16384 to get over 1000 t/s PP.
What I find quite surprising is that offloading some/more layers has very little effect... but I guess it is just math - I'd need to offload much more to see a noticeable improvement.

Yeah as @anikifoss points out on the linked thread

GLM-4.6 has 92 layers

So each additional routed expert layer offload to VRAM isn't a large overall percentage especially given only 8 or however many are active per token - its very sparse.

You can still add a max power cap in addition to LACT if you're really worried about tripping the power supply, but yeah play around and get your GPUs tuned up one at a time then save the config into /etc/lact/config.yaml and you'll be gucci!

No idea what this was about, it said this already by the time I read this:

This comment has been hidden (marked as Off-Topic)

No idea what this was about (...)

HF is acting strange lately... but this was my mistake ;)

Here are some performance tests of the IQ5_K version running on EPYC 9355, RTX 5090, and RTX Pro 6000.
Sharing these benchmarks for anyone thinking about this kind of setup.

• CPU power capped at 280 W, GPUs at 450 W
• Running inside a Proxmox/QEMU VM, with only 32 real cores assigned (SMT cores reserved for host)
• Testing various combinations of context size and batch size fitting the VRAM

This is really neat! How much RAM do you have? I assume not everything is fitting on the 96GB of VRAM.