Cannot load w/ sglang 0.5.6
#1
by
huggyfaceenjoyer
- opened
AttributeError: 'Glm4vMoeConfig' object has no attribute 'rope_scaling'
4.6v
"rope_parameters": {
"mrope_section": [
8,
12,
12
],
"partial_rotary_factor": 0.5,
"rope_theta": 500000,
"rope_type": "default"
},
4.5 Air
"rope_scaling": null,
"rope_theta": 1000000,
use sglang 0.5.6.post1
use sglang 0.5.6.post1
That got me to be able to get further along, but I run out of vram with same settings as Air FP8 with 4.6V FP8, even drastically reducing context window to minimum. Dual RTX 6000 Pro 96G.
Also having a lot of problems with 0.5.6.post1 and/or prerelease transformers where even GLM 4.5 Air is no longer stable and had to revert.