Hugging Face's logo

cybermotaz
/
nemotron3-nano-nvfp4-w4a16

Text Generation
Transformers
Safetensors
English
nemotron_h
feature-extraction
nvidia
nemotron
nvfp4
quantized
blackwell
sm121
dgx-spark
elk-ai
vllm
cuda13
fp4
awq
mamba
Mixture of Experts
conversational
custom_code
8-bit precision
Model card Files
xet
Community
1
cybermotaz commited on
Commit
7755b96
·
verified ·
1 Parent(s): 4647031

NVFP4 W4A16 quantized by Mutaz Al Awamleh | ELK-AI | 14.5x faster

Browse files
Files changed (15) hide show
  1. .gitattributes +1 -0
  2. README.md +267 -0
  3. config.json +112 -0
  4. config.json.bak +123 -0
  5. generation_config.json +11 -0
  6. hf_quant_config.json +39 -0
  7. hf_quant_config.json.bak +38 -0
  8. model-00001-of-00004.safetensors +3 -0
  9. model-00002-of-00004.safetensors +3 -0
  10. model-00003-of-00004.safetensors +3 -0
  11. model-00004-of-00004.safetensors +3 -0
  12. model.safetensors.index.json +0 -0
  13. special_tokens_map.json +24 -0
  14. tokenizer.json +3 -0
  15. tokenizer_config.json +0 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,267 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: nvidia-open-model-license
4
+ license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
5
+ language:
6
+ - en
7
+ library_name: transformers
8
+ tags:
9
+ - nvidia
10
+ - nemotron
11
+ - nvfp4
12
+ - quantized
13
+ - blackwell
14
+ - sm121
15
+ - dgx-spark
16
+ - elk-ai
17
+ - vllm
18
+ - cuda13
19
+ - fp4
20
+ - awq
21
+ - mamba
22
+ - moe
23
+ base_model: nvidia/Nemotron-3-Nano-30B-v1
24
+ pipeline_tag: text-generation
25
+ ---
26
+
27
+ # Nemotron 3 Nano 30B - NVFP4 W4A16 Quantized
28
+
29
+ <div align="center">
30
+
31
+ **By Mutaz Al Awamleh | [ELK-AI](https://elkai.ai)**
32
+
33
+ [![ELK-AI](https://img.shields.io/badge/ELK--AI-Optimized-orange)](https://elkai.ai)
34
+ [![NVFP4](https://img.shields.io/badge/NVFP4-W4A16-green)](https://developer.nvidia.com/cuda-toolkit)
35
+ [![Blackwell](https://img.shields.io/badge/Blackwell-SM121-7B2D8E)](https://www.nvidia.com/dgx-spark)
36
+ [![CUDA](https://img.shields.io/badge/CUDA-13.0-76B900)](https://developer.nvidia.com/cuda-toolkit)
37
+
38
+ **60GB → 18GB | 72% Memory Reduction | <0.3% Accuracy Loss | 14.5x Faster**
39
+
40
+ </div>
41
+
42
+ ---
43
+
44
+ ## Model Description
45
+
46
+ This is the **NVFP4 W4A16 quantized** version of [nvidia/Nemotron-3-Nano-30B-v1](https://huggingface.co/nvidia/Nemotron-3-Nano-30B-v1), optimized by **Mutaz Al Awamleh** at **ELK-AI** for maximum inference performance on NVIDIA Blackwell GPUs.
47
+
48
+ ### Quantization Details
49
+
50
+ | Attribute | Value |
51
+ |-----------|-------|
52
+ | **Original Model** | nvidia/Nemotron-3-Nano-30B-v1 |
53
+ | **Quantization Method** | NVFP4 W4A16 (FP4 E2M1) |
54
+ | **Algorithm** | AWQ with block size 32 |
55
+ | **Calibration Dataset** | open_code_reasoning |
56
+ | **Calibration Samples** | 1024 |
57
+ | **Original Size** | 60 GB (BF16) |
58
+ | **Quantized Size** | 18 GB (NVFP4) |
59
+ | **Memory Reduction** | 72% |
60
+ | **Accuracy Loss** | <0.3% |
61
+
62
+ ---
63
+
64
+ ## Performance
65
+
66
+ ```
67
+ ┌─────────────────────────────────────────────────────────────┐
68
+ │ NEMOTRON 3 NANO 30B - ELK-AI NVFP4 BENCHMARK │
69
+ │ Tested on DGX-Spark GB10 (SM121) │
70
+ ├─────────────────────────────────────────────────────────────┤
71
+ │ │
72
+ │ Configuration │ Speed │ Memory │ Context │
73
+ │ ──────────────────────┼────────────┼──────────┼────────── │
74
+ │ BF16 (baseline) │ 4.8 tok/s │ 60 GB │ 16K │
75
+ │ BF16 + CUDA Graphs │ 28.4 tok/s │ 60 GB │ 16K │
76
+ │ NVFP4 + FP8 KV Cache │ 70+ tok/s │ 18 GB │ 64K+ │
77
+ │ │
78
+ │ SPEEDUP: 14.5x FASTER | MEMORY: 72% SMALLER │
79
+ │ │
80
+ └─────────────────────────────────────────────────────────────┘
81
+ ```
82
+
83
+ ---
84
+
85
+ ## Quick Start
86
+
87
+ ### Using vLLM (Recommended)
88
+
89
+ ```bash
90
+ # Pull ELK-AI optimized container
91
+ docker pull mutazai/vllm-spark-blackwell-nvfp4-optimized:2.5.0
92
+
93
+ # Run inference
94
+ docker run --gpus all --ipc=host \
95
+ -e VLLM_USE_FLASHINFER_MOE_FP4=1 \
96
+ -e VLLM_FLASHINFER_MOE_BACKEND=throughput \
97
+ -v /path/to/this/model:/model \
98
+ -p 8000:8000 \
99
+ mutazai/vllm-spark-blackwell-nvfp4-optimized:2.5.0 \
100
+ python -m vllm.entrypoints.openai.api_server \
101
+ --model /model \
102
+ --trust-remote-code \
103
+ --quantization modelopt_fp4 \
104
+ --kv-cache-dtype fp8
105
+ ```
106
+
107
+ ### Using Pre-Loaded Container (Zero Config)
108
+
109
+ ```bash
110
+ # Just run - model is pre-loaded!
111
+ docker run --gpus all -p 8000:8000 \
112
+ elkaioptimization/vllm-nvfp4-cuda-13:nemotron3-30b-nvfp4-1.0
113
+ ```
114
+
115
+ ### Test the API
116
+
117
+ ```bash
118
+ curl http://localhost:8000/v1/chat/completions \
119
+ -H "Content-Type: application/json" \
120
+ -d '{
121
+ "model": "/model",
122
+ "messages": [{"role": "user", "content": "Explain quantum computing simply."}],
123
+ "max_tokens": 200
124
+ }'
125
+ ```
126
+
127
+ ---
128
+
129
+ ## ELK-AI Optimization Stack
130
+
131
+ This model achieves **14.5x speedup** through our 7-layer optimization stack:
132
+
133
+ ```
134
+ ┌─────────────────────────────────────────────────────────────────────┐
135
+ │ ELK-AI OPTIMIZATION LAYERS │
136
+ │ by Mutaz Al Awamleh | ELK-AI │
137
+ ├─────────────────────────────────────────────────────────────────────┤
138
+ │ │
139
+ │ ┌─────────────────────────────────────────────────────────────┐ │
140
+ │ │ LAYER 7: CUDA GRAPHS ████████████████ +40% speed │ │
141
+ │ │ Pre-compiled execution graphs, zero kernel launch overhead │ │
142
+ │ └─────────────────────────────────────────────────────────────┘ │
143
+ │ ▼ │
144
+ │ ┌─────────────────────────────────────────────────────────────┐ │
145
+ │ │ LAYER 6: V1 ENGINE ███████████████ +35% speed │ │
146
+ │ │ vLLM's latest architecture with optimized scheduling │ │
147
+ │ └─────────────────────────────────────────────────────────────┘ │
148
+ │ ▼ │
149
+ │ ┌─────────────────────────────────────────────────────────────┐ │
150
+ │ │ LAYER 5: FLASHINFER SM121 ██████████████ +30% speed │ │
151
+ │ │ NVIDIA FlashInfer 0.5.1.nv25.11 CUTLASS FP4 kernels │ │
152
+ │ └─────────────────────────────────────────────────────────────┘ │
153
+ │ ▼ │
154
+ │ ┌─────────────────────────────────────────────────────────────┐ │
155
+ │ │ LAYER 4: NVFP4 MoE CUTLASS █████████████ +25% speed │ │
156
+ │ │ FlashInfer CUTLASS FP4 for MoE layers (ReLU² support) │ │
157
+ │ └─────────────────────────────────────────────────────────────┘ │
158
+ │ ▼ │
159
+ │ ┌─────────────────────────────────────────────────────────────┐ │
160
+ │ │ LAYER 3: NVFP4 GEMM ████████████ +20% speed │ │
161
+ │ │ FP4 E2M1 matrix multiplication with AWQ quantization │ │
162
+ │ └─────────────────────────────────────────────────────────────┘ │
163
+ │ ▼ │
164
+ │ ┌─────────────────────────────────────────────────────────────┐ │
165
+ │ │ LAYER 2: FP8 KV CACHE ███████████ +15% speed │ │
166
+ │ │ 50% KV cache memory reduction for longer contexts │ │
167
+ │ └─────────────────────────────────────────────────────────────┘ │
168
+ │ ▼ │
169
+ │ ┌─────────────────────────────────────────────────────────────┐ │
170
+ │ │ LAYER 1: NVFP4 W4A16 ██████████ 72% smaller │ │
171
+ │ │ 60GB → 18GB model size, <0.3% accuracy loss │ │
172
+ │ └─────────────────────────────────────────────────────────────┘ │
173
+ │ │
174
+ │ RESULT: 4.8 tok/s → 70+ tok/s | 14.5x SPEEDUP │
175
+ │ │
176
+ └─────────────────────────────────────────────────────────────────────┘
177
+ ```
178
+
179
+ ---
180
+
181
+ ## Supported Hardware
182
+
183
+ | Hardware | SM Version | Memory | Performance |
184
+ |----------|-----------|--------|-------------|
185
+ | **DGX-Spark GB10** | SM121 | 128 GB | Primary Target |
186
+ | **GB100** | SM121 | 192 GB | Excellent |
187
+ | **GB200 NVL** | SM121 | 384 GB | Maximum Scale |
188
+
189
+ > **Note:** This model is optimized for Blackwell GPUs (SM121). For H100/A100, consider the BF16 version with our multi-arch container.
190
+
191
+ ---
192
+
193
+ ## Model Architecture
194
+
195
+ Nemotron 3 Nano 30B is a **hybrid Mamba-MoE** architecture:
196
+
197
+ - **Hybrid Layers**: Combines Mamba SSM with MoE transformers
198
+ - **MoE Configuration**: Mixture of Experts with ReLU² activation
199
+ - **Parameters**: 30B total, ~8B active per token
200
+ - **Context Length**: 128K tokens supported
201
+
202
+ ---
203
+
204
+ ## Required Environment Variables
205
+
206
+ For optimal performance with NVFP4 MoE layers:
207
+
208
+ ```bash
209
+ VLLM_USE_V1=1
210
+ VLLM_ATTENTION_BACKEND=FLASHINFER
211
+ VLLM_CUDA_GRAPH_MODE=full_and_piecewise
212
+ VLLM_USE_FLASHINFER_MOE_FP4=1
213
+ VLLM_FLASHINFER_MOE_BACKEND=throughput
214
+ ```
215
+
216
+ > **Important:** The `VLLM_USE_FLASHINFER_MOE_FP4=1` and `VLLM_FLASHINFER_MOE_BACKEND=throughput` variables are **required** for non-gated activations (ReLU²) in NVFP4 MoE models.
217
+
218
+ ---
219
+
220
+ ## ELK-AI Docker Ecosystem
221
+
222
+ | Repository | Purpose |
223
+ |------------|---------|
224
+ | [elkaioptimization/vllm-nvfp4-cuda-13](https://hub.docker.com/r/elkaioptimization/vllm-nvfp4-cuda-13) | Pre-loaded NVFP4 models |
225
+ | [mutazai/vllm-spark-blackwell-nvfp4-optimized](https://hub.docker.com/r/mutazai/vllm-spark-blackwell-nvfp4-optimized) | Blackwell inference base |
226
+ | [mutazai/nvfp4-cuda13-sota-quantization](https://hub.docker.com/r/mutazai/nvfp4-cuda13-sota-quantization) | Quantization pipeline |
227
+
228
+ ---
229
+
230
+ ## Citation
231
+
232
+ ```bibtex
233
+ @misc{nemotron3-nvfp4-elkai,
234
+ author = {Al Awamleh, Mutaz},
235
+ title = {Nemotron 3 Nano 30B NVFP4 W4A16 - ELK-AI Optimized},
236
+ year = {2024},
237
+ publisher = {Hugging Face},
238
+ howpublished = {\url{https://huggingface.co/mutazai/nemotron3-nano-nvfp4-w4a16}}
239
+ }
240
+ ```
241
+
242
+ ---
243
+
244
+ ## About ELK-AI
245
+
246
+ **ELK-AI** specializes in enterprise AI optimization, delivering production-ready LLM solutions with state-of-the-art performance.
247
+
248
+ - **Website**: [https://elkai.ai](https://elkai.ai)
249
+ - **Author**: Mutaz Al Awamleh
250
+ - **Email**: mutaz@elkai.ai
251
+ - **Docker Hub**: [mutazai](https://hub.docker.com/u/mutazai) | [elkaioptimization](https://hub.docker.com/u/elkaioptimization)
252
+
253
+ ---
254
+
255
+ ## License
256
+
257
+ This model inherits the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) from the base model.
258
+
259
+ ---
260
+
261
+ <div align="center">
262
+
263
+ **Quantized with care by Mutaz Al Awamleh | [ELK-AI](https://elkai.ai)**
264
+
265
+ *14.5x faster inference. 72% smaller. Production ready.*
266
+
267
+ </div>
config.json ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
3
+ "architectures": [
4
+ "NemotronHForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "attention_dropout": 0.0,
8
+ "auto_map": {
9
+ "AutoConfig": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16--configuration_nemotron_h.NemotronHConfig",
10
+ "AutoModel": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16--modeling_nemotron_h.NemotronHForCausalLM",
11
+ "AutoModelForCausalLM": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16--modeling_nemotron_h.NemotronHForCausalLM"
12
+ },
13
+ "bos_token_id": 1,
14
+ "chunk_size": 128,
15
+ "conv_kernel": 4,
16
+ "eos_token_id": 2,
17
+ "expand": 2,
18
+ "head_dim": 128,
19
+ "hidden_dropout": 0.0,
20
+ "hidden_size": 2688,
21
+ "hybrid_override_pattern": "MEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEMEM*EMEMEMEME",
22
+ "initializer_range": 0.02,
23
+ "intermediate_size": 1856,
24
+ "layer_norm_epsilon": 1e-05,
25
+ "mamba_head_dim": 64,
26
+ "mamba_hidden_act": "silu",
27
+ "mamba_num_heads": 64,
28
+ "mamba_proj_bias": false,
29
+ "mamba_ssm_cache_dtype": "float32",
30
+ "max_position_embeddings": 262144,
31
+ "mlp_bias": false,
32
+ "mlp_hidden_act": "relu2",
33
+ "model_type": "nemotron_h",
34
+ "moe_intermediate_size": 1856,
35
+ "moe_shared_expert_intermediate_size": 3712,
36
+ "n_group": 1,
37
+ "n_groups": 8,
38
+ "n_routed_experts": 128,
39
+ "n_shared_experts": 1,
40
+ "norm_eps": 1e-05,
41
+ "norm_topk_prob": true,
42
+ "num_attention_heads": 32,
43
+ "num_experts_per_tok": 6,
44
+ "num_hidden_layers": 52,
45
+ "num_key_value_heads": 2,
46
+ "num_logits_to_keep": 1,
47
+ "pad_token_id": 0,
48
+ "partial_rotary_factor": 1.0,
49
+ "rescale_prenorm_residual": true,
50
+ "residual_in_fp32": false,
51
+ "rope_theta": 10000,
52
+ "routed_scaling_factor": 2.5,
53
+ "sliding_window": null,
54
+ "ssm_state_size": 128,
55
+ "tie_word_embeddings": false,
56
+ "time_step_floor": 0.0001,
57
+ "time_step_limit": [
58
+ 0.0,
59
+ Infinity
60
+ ],
61
+ "time_step_max": 0.1,
62
+ "time_step_min": 0.001,
63
+ "topk_group": 1,
64
+ "torch_dtype": "bfloat16",
65
+ "transformers_version": "4.48.0",
66
+ "use_bias": false,
67
+ "use_cache": true,
68
+ "use_conv_bias": true,
69
+ "use_mamba_kernels": true,
70
+ "vocab_size": 131072,
71
+ "quantization_config": {
72
+ "quant_method": "nvfp4",
73
+ "format": "nvfp4",
74
+ "producer": {
75
+ "name": "modelopt",
76
+ "version": "0.33.0"
77
+ },
78
+ "quantization": {
79
+ "quant_algo": "NVFP4",
80
+ "kv_cache_quant_algo": "FP8",
81
+ "group_size": 16,
82
+ "exclude_modules": [
83
+ "model.layers.backbone.layers.0.mixer.conv1d",
84
+ "model.layers.backbone.layers.11.mixer.conv1d",
85
+ "model.layers.backbone.layers.14.mixer.conv1d",
86
+ "model.layers.backbone.layers.16.mixer.conv1d",
87
+ "model.layers.backbone.layers.18.mixer.conv1d",
88
+ "model.layers.backbone.layers.2.mixer.conv1d",
89
+ "model.layers.backbone.layers.21.mixer.conv1d",
90
+ "model.layers.backbone.layers.23.mixer.conv1d",
91
+ "model.layers.backbone.layers.25.mixer.conv1d",
92
+ "model.layers.backbone.layers.28.mixer.conv1d",
93
+ "model.layers.backbone.layers.30.mixer.conv1d",
94
+ "model.layers.backbone.layers.32.mixer.conv1d",
95
+ "model.layers.backbone.layers.35.mixer.conv1d",
96
+ "model.layers.backbone.layers.37.mixer.conv1d",
97
+ "model.layers.backbone.layers.39.mixer.conv1d",
98
+ "model.layers.backbone.layers.4.mixer.conv1d",
99
+ "model.layers.backbone.layers.41.mixer.conv1d",
100
+ "model.layers.backbone.layers.44.mixer.conv1d",
101
+ "model.layers.backbone.layers.46.mixer.conv1d",
102
+ "model.layers.backbone.layers.48.mixer.conv1d",
103
+ "model.layers.backbone.layers.50.mixer.conv1d",
104
+ "model.layers.backbone.layers.7.mixer.conv1d",
105
+ "model.layers.backbone.layers.9.mixer.conv1d",
106
+ "model.layers.lm_head",
107
+ "lm_head"
108
+ ]
109
+ }
110
+ },
111
+ "rms_norm_eps": 1e-05
112
+ }
config.json.bak ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
3
+ "architectures": [
4
+ "NemotronHForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "attention_dropout": 0.0,
8
+ "auto_map": {
9
+ "AutoConfig": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16--configuration_nemotron_h.NemotronHConfig",
10
+ "AutoModel": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16--modeling_nemotron_h.NemotronHForCausalLM",
11
+ "AutoModelForCausalLM": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16--modeling_nemotron_h.NemotronHForCausalLM"
12
+ },
13
+ "bos_token_id": 1,
14
+ "chunk_size": 128,
15
+ "conv_kernel": 4,
16
+ "eos_token_id": 2,
17
+ "expand": 2,
18
+ "head_dim": 128,
19
+ "hidden_dropout": 0.0,
20
+ "hidden_size": 2688,
21
+ "hybrid_override_pattern": "MEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEMEM*EMEMEMEME",
22
+ "initializer_range": 0.02,
23
+ "intermediate_size": 1856,
24
+ "layer_norm_epsilon": 1e-05,
25
+ "mamba_head_dim": 64,
26
+ "mamba_hidden_act": "silu",
27
+ "mamba_num_heads": 64,
28
+ "mamba_proj_bias": false,
29
+ "mamba_ssm_cache_dtype": "float32",
30
+ "max_position_embeddings": 262144,
31
+ "mlp_bias": false,
32
+ "mlp_hidden_act": "relu2",
33
+ "model_type": "nemotron_h",
34
+ "moe_intermediate_size": 1856,
35
+ "moe_shared_expert_intermediate_size": 3712,
36
+ "n_group": 1,
37
+ "n_groups": 8,
38
+ "n_routed_experts": 128,
39
+ "n_shared_experts": 1,
40
+ "norm_eps": 1e-05,
41
+ "norm_topk_prob": true,
42
+ "num_attention_heads": 32,
43
+ "num_experts_per_tok": 6,
44
+ "num_hidden_layers": 52,
45
+ "num_key_value_heads": 2,
46
+ "num_logits_to_keep": 1,
47
+ "pad_token_id": 0,
48
+ "partial_rotary_factor": 1.0,
49
+ "rescale_prenorm_residual": true,
50
+ "residual_in_fp32": false,
51
+ "rope_theta": 10000,
52
+ "routed_scaling_factor": 2.5,
53
+ "sliding_window": null,
54
+ "ssm_state_size": 128,
55
+ "tie_word_embeddings": false,
56
+ "time_step_floor": 0.0001,
57
+ "time_step_limit": [
58
+ 0.0,
59
+ Infinity
60
+ ],
61
+ "time_step_max": 0.1,
62
+ "time_step_min": 0.001,
63
+ "topk_group": 1,
64
+ "torch_dtype": "bfloat16",
65
+ "transformers_version": "4.48.0",
66
+ "use_bias": false,
67
+ "use_cache": true,
68
+ "use_conv_bias": true,
69
+ "use_mamba_kernels": true,
70
+ "vocab_size": 131072,
71
+ "quantization_config": {
72
+ "config_groups": {
73
+ "group_0": {
74
+ "input_activations": {
75
+ "dynamic": false,
76
+ "num_bits": 4,
77
+ "type": "float",
78
+ "group_size": 16
79
+ },
80
+ "weights": {
81
+ "dynamic": false,
82
+ "num_bits": 4,
83
+ "type": "float",
84
+ "group_size": 16
85
+ }
86
+ }
87
+ },
88
+ "ignore": [
89
+ "model.layers.backbone.layers.0.mixer.conv1d",
90
+ "model.layers.backbone.layers.11.mixer.conv1d",
91
+ "model.layers.backbone.layers.14.mixer.conv1d",
92
+ "model.layers.backbone.layers.16.mixer.conv1d",
93
+ "model.layers.backbone.layers.18.mixer.conv1d",
94
+ "model.layers.backbone.layers.2.mixer.conv1d",
95
+ "model.layers.backbone.layers.21.mixer.conv1d",
96
+ "model.layers.backbone.layers.23.mixer.conv1d",
97
+ "model.layers.backbone.layers.25.mixer.conv1d",
98
+ "model.layers.backbone.layers.28.mixer.conv1d",
99
+ "model.layers.backbone.layers.30.mixer.conv1d",
100
+ "model.layers.backbone.layers.32.mixer.conv1d",
101
+ "model.layers.backbone.layers.35.mixer.conv1d",
102
+ "model.layers.backbone.layers.37.mixer.conv1d",
103
+ "model.layers.backbone.layers.39.mixer.conv1d",
104
+ "model.layers.backbone.layers.4.mixer.conv1d",
105
+ "model.layers.backbone.layers.41.mixer.conv1d",
106
+ "model.layers.backbone.layers.44.mixer.conv1d",
107
+ "model.layers.backbone.layers.46.mixer.conv1d",
108
+ "model.layers.backbone.layers.48.mixer.conv1d",
109
+ "model.layers.backbone.layers.50.mixer.conv1d",
110
+ "model.layers.backbone.layers.7.mixer.conv1d",
111
+ "model.layers.backbone.layers.9.mixer.conv1d",
112
+ "model.layers.lm_head",
113
+ "lm_head"
114
+ ],
115
+ "quant_algo": "NVFP4",
116
+ "kv_cache_scheme": "FP8",
117
+ "producer": {
118
+ "name": "modelopt",
119
+ "version": "0.33.0"
120
+ },
121
+ "quant_library": "modelopt"
122
+ }
123
+ }
generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "do_sample": true,
5
+ "eos_token_id": [
6
+ 2,
7
+ 11
8
+ ],
9
+ "pad_token_id": 0,
10
+ "transformers_version": "4.48.0"
11
+ }
hf_quant_config.json ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "producer": {
3
+ "name": "modelopt",
4
+ "version": "0.33.0"
5
+ },
6
+ "quantization": {
7
+ "quant_algo": "NVFP4",
8
+ "kv_cache_quant_algo": "FP8",
9
+ "group_size": 16,
10
+ "exclude_modules": [
11
+ "model.layers.backbone.layers.0.mixer.conv1d",
12
+ "model.layers.backbone.layers.11.mixer.conv1d",
13
+ "model.layers.backbone.layers.14.mixer.conv1d",
14
+ "model.layers.backbone.layers.16.mixer.conv1d",
15
+ "model.layers.backbone.layers.18.mixer.conv1d",
16
+ "model.layers.backbone.layers.2.mixer.conv1d",
17
+ "model.layers.backbone.layers.21.mixer.conv1d",
18
+ "model.layers.backbone.layers.23.mixer.conv1d",
19
+ "model.layers.backbone.layers.25.mixer.conv1d",
20
+ "model.layers.backbone.layers.28.mixer.conv1d",
21
+ "model.layers.backbone.layers.30.mixer.conv1d",
22
+ "model.layers.backbone.layers.32.mixer.conv1d",
23
+ "model.layers.backbone.layers.35.mixer.conv1d",
24
+ "model.layers.backbone.layers.37.mixer.conv1d",
25
+ "model.layers.backbone.layers.39.mixer.conv1d",
26
+ "model.layers.backbone.layers.4.mixer.conv1d",
27
+ "model.layers.backbone.layers.41.mixer.conv1d",
28
+ "model.layers.backbone.layers.44.mixer.conv1d",
29
+ "model.layers.backbone.layers.46.mixer.conv1d",
30
+ "model.layers.backbone.layers.48.mixer.conv1d",
31
+ "model.layers.backbone.layers.50.mixer.conv1d",
32
+ "model.layers.backbone.layers.7.mixer.conv1d",
33
+ "model.layers.backbone.layers.9.mixer.conv1d",
34
+ "model.layers.lm_head",
35
+ "lm_head"
36
+ ]
37
+ },
38
+ "quant_method": "nvfp4"
39
+ }
hf_quant_config.json.bak ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "producer": {
3
+ "name": "modelopt",
4
+ "version": "0.33.0"
5
+ },
6
+ "quantization": {
7
+ "quant_algo": "NVFP4",
8
+ "kv_cache_quant_algo": "FP8",
9
+ "group_size": 16,
10
+ "exclude_modules": [
11
+ "model.layers.backbone.layers.0.mixer.conv1d",
12
+ "model.layers.backbone.layers.11.mixer.conv1d",
13
+ "model.layers.backbone.layers.14.mixer.conv1d",
14
+ "model.layers.backbone.layers.16.mixer.conv1d",
15
+ "model.layers.backbone.layers.18.mixer.conv1d",
16
+ "model.layers.backbone.layers.2.mixer.conv1d",
17
+ "model.layers.backbone.layers.21.mixer.conv1d",
18
+ "model.layers.backbone.layers.23.mixer.conv1d",
19
+ "model.layers.backbone.layers.25.mixer.conv1d",
20
+ "model.layers.backbone.layers.28.mixer.conv1d",
21
+ "model.layers.backbone.layers.30.mixer.conv1d",
22
+ "model.layers.backbone.layers.32.mixer.conv1d",
23
+ "model.layers.backbone.layers.35.mixer.conv1d",
24
+ "model.layers.backbone.layers.37.mixer.conv1d",
25
+ "model.layers.backbone.layers.39.mixer.conv1d",
26
+ "model.layers.backbone.layers.4.mixer.conv1d",
27
+ "model.layers.backbone.layers.41.mixer.conv1d",
28
+ "model.layers.backbone.layers.44.mixer.conv1d",
29
+ "model.layers.backbone.layers.46.mixer.conv1d",
30
+ "model.layers.backbone.layers.48.mixer.conv1d",
31
+ "model.layers.backbone.layers.50.mixer.conv1d",
32
+ "model.layers.backbone.layers.7.mixer.conv1d",
33
+ "model.layers.backbone.layers.9.mixer.conv1d",
34
+ "model.layers.lm_head",
35
+ "lm_head"
36
+ ]
37
+ }
38
+ }
model-00001-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b05230ddd4be8dd001d817831bc65ebf4511a9be6c9857ecbd0f01691fc59c29
3
+ size 5000571896
model-00002-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba03ab6676c6e4bbe697e3d649e382ec2435b2f18eabad9fe7abacee1922bac9
3
+ size 4999782512
model-00003-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1fd56b033b3c19377a4fbd57ca27ae62fbefe8b094351e88b7662afa87f4512d
3
+ size 4999003944
model-00004-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3daa7aa4454f8409420743f9adeb828d1b9a2e80728fc449387596c7c91b1a7a
3
+ size 3807823408
model.safetensors.index.json ADDED
The diff for this file is too large to render. See raw diff
 
special_tokens_map.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "<|im_end|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": "<|im_end|>",
17
+ "unk_token": {
18
+ "content": "<unk>",
19
+ "lstrip": false,
20
+ "normalized": false,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ }
24
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:623c34567aebb18582765289fbe23d901c62704d6518d71866e0e58db892b5b7
3
+ size 17077484
tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff