zai-org

/

GLM-4.7-Flash

like 1.48k

Follow

Z.ai 8.99k

Right, for transformers, I just tried it too, and using DeepseekV3ForCausalLM for simple conversation works, but for sglang and vLLM, they use different hooks, which could cause errors (especially sglang, kernel is different and now still in progress)

As for why even the Todos are the same, it's because the Attention implement in transformers is indeed completely identical to DeepseekV3Attention, and using modular will copy all tools from DeepseekV3.

What if we simply inherit from the DeepseekV3 modular/modeling module rather than duplicating the code?

The Qwen model appears to follow this approach by extending parts of code from its previous version without duplication: https://github.com/huggingface/transformers/blob/9ed801f3ef0029e3733bbd2c9f9f9866912412a2/src/transformers/models/qwen2_5_omni/modular_qwen2_5_omni.py#L2064

I remain skeptical that duplicating the code achieves any meaningful benefit; it strikes me as poor practice to omit acknowledgment for the original author's work.

Additionally, you mentioned this was intended to prevent issues with sglang and vLLM, yet:

• A user has indicated it's incompatible with vLLM: https://huggingface.co/zai-org/GLM-4.7-Flash/discussions/22
• The KV cache consumes excess memory (https://huggingface.co/zai-org/GLM-4.7-Flash/discussions/21) due to the absence of a configuration that recognizes the model as deepseek: https://github.com/vllm-project/vllm/pull/32614
• Support for sglang is still WIP, whereas deepseekv3 runs without any problems

Sorry for the oversight earlier - the modular file does inherit the DSv3 module: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4_moe_lite/modular_glm4_moe_lite.py

And it looks like the only thing added on top of DSv3 is to ignore the MTP layer.