Hugging Face's logo

Sakalti
/
ultiima-72B

Text Generation
Transformers
Safetensors
qwen2
mergekit
Merge
conversational
Eval Results (legacy)
text-generation-inference
Model card Files
xet
Community
3

Improve language tag

#3
by lbourdois - opened
base: refs/heads/main
←
from: refs/pr/3
Discussion Files changed
+168
-184
Files changed (1) hide show
  1. README.md +168 -184
README.md CHANGED
@@ -1,184 +1,168 @@
1
- ---
2
- language:
3
- - en
4
- - zh
5
- - fr
6
- - es
7
- - pt
8
- - de
9
- - it
10
- - ru
11
- - ja
12
- - ko
13
- - vi
14
- - th
15
- - ar
16
- - fa
17
- - he
18
- - tr
19
- - cs
20
- - pl
21
- - hi
22
- - bn
23
- - ur
24
- - id
25
- - ms
26
- - lo
27
- - my
28
- - ceb
29
- - km
30
- - tl
31
- - nl
32
- license: other
33
- library_name: transformers
34
- tags:
35
- - mergekit
36
- - merge
37
- base_model:
38
- - Qwen/Qwen2.5-72B-Instruct
39
- - Qwen/Qwen2.5-72B
40
- license_name: qwen
41
- inference: true
42
- model-index:
43
- - name: ultiima-72B
44
- results:
45
- - task:
46
- type: text-generation
47
- name: Text Generation
48
- dataset:
49
- name: IFEval (0-Shot)
50
- type: HuggingFaceH4/ifeval
51
- args:
52
- num_few_shot: 0
53
- metrics:
54
- - type: inst_level_strict_acc and prompt_level_strict_acc
55
- value: 71.4
56
- name: strict accuracy
57
- source:
58
- url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
59
- name: Open LLM Leaderboard
60
- - task:
61
- type: text-generation
62
- name: Text Generation
63
- dataset:
64
- name: BBH (3-Shot)
65
- type: BBH
66
- args:
67
- num_few_shot: 3
68
- metrics:
69
- - type: acc_norm
70
- value: 61.1
71
- name: normalized accuracy
72
- source:
73
- url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
74
- name: Open LLM Leaderboard
75
- - task:
76
- type: text-generation
77
- name: Text Generation
78
- dataset:
79
- name: MATH Lvl 5 (4-Shot)
80
- type: hendrycks/competition_math
81
- args:
82
- num_few_shot: 4
83
- metrics:
84
- - type: exact_match
85
- value: 52.42
86
- name: exact match
87
- source:
88
- url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
89
- name: Open LLM Leaderboard
90
- - task:
91
- type: text-generation
92
- name: Text Generation
93
- dataset:
94
- name: GPQA (0-shot)
95
- type: Idavidrein/gpqa
96
- args:
97
- num_few_shot: 0
98
- metrics:
99
- - type: acc_norm
100
- value: 21.92
101
- name: acc_norm
102
- source:
103
- url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
104
- name: Open LLM Leaderboard
105
- - task:
106
- type: text-generation
107
- name: Text Generation
108
- dataset:
109
- name: MuSR (0-shot)
110
- type: TAUR-Lab/MuSR
111
- args:
112
- num_few_shot: 0
113
- metrics:
114
- - type: acc_norm
115
- value: 18.12
116
- name: acc_norm
117
- source:
118
- url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
119
- name: Open LLM Leaderboard
120
- - task:
121
- type: text-generation
122
- name: Text Generation
123
- dataset:
124
- name: MMLU-PRO (5-shot)
125
- type: TIGER-Lab/MMLU-Pro
126
- config: main
127
- split: test
128
- args:
129
- num_few_shot: 5
130
- metrics:
131
- - type: acc
132
- value: 54.51
133
- name: accuracy
134
- source:
135
- url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
136
- name: Open LLM Leaderboard
137
- ---
138
- Built With Qwen
139
- # merge
140
-
141
- This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
142
-
143
- ## Merge Details
144
- ### Merge Method
145
-
146
- This model was merged using the [TIES](https://arxiv.org/abs/2306.01708) merge method using [Qwen/Qwen2.5-72B](https://huggingface.co/Qwen/Qwen2.5-72B) as a base.
147
-
148
- ### Models Merged
149
-
150
- The following models were included in the merge:
151
- * [Qwen/Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct)
152
-
153
- ### Configuration
154
-
155
- The following YAML configuration was used to produce this model:
156
-
157
- ```yaml
158
- models:
159
- - model: Qwen/Qwen2.5-72B-Instruct
160
- parameters:
161
- weight: 1
162
- density: 1
163
- merge_method: ties
164
- base_model: Qwen/Qwen2.5-72B
165
- parameters:
166
- weight: 1
167
- density: 1
168
- normalize: true
169
- int8_mask: true
170
- dtype: float16
171
- ```
172
- # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
173
- Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/Sakalti__ultiima-72B-details)
174
-
175
- | Metric |Value|
176
- |-------------------|----:|
177
- |Avg. |46.58|
178
- |IFEval (0-Shot) |71.40|
179
- |BBH (3-Shot) |61.10|
180
- |MATH Lvl 5 (4-Shot)|52.42|
181
- |GPQA (0-shot) |21.92|
182
- |MuSR (0-shot) |18.12|
183
- |MMLU-PRO (5-shot) |54.51|
184
-
 
1
+ ---
2
+ language:
3
+ - zho
4
+ - eng
5
+ - fra
6
+ - spa
7
+ - por
8
+ - deu
9
+ - ita
10
+ - rus
11
+ - jpn
12
+ - kor
13
+ - vie
14
+ - tha
15
+ - ara
16
+ license: other
17
+ library_name: transformers
18
+ tags:
19
+ - mergekit
20
+ - merge
21
+ base_model:
22
+ - Qwen/Qwen2.5-72B-Instruct
23
+ - Qwen/Qwen2.5-72B
24
+ license_name: qwen
25
+ inference: true
26
+ model-index:
27
+ - name: ultiima-72B
28
+ results:
29
+ - task:
30
+ type: text-generation
31
+ name: Text Generation
32
+ dataset:
33
+ name: IFEval (0-Shot)
34
+ type: HuggingFaceH4/ifeval
35
+ args:
36
+ num_few_shot: 0
37
+ metrics:
38
+ - type: inst_level_strict_acc and prompt_level_strict_acc
39
+ value: 71.4
40
+ name: strict accuracy
41
+ source:
42
+ url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
43
+ name: Open LLM Leaderboard
44
+ - task:
45
+ type: text-generation
46
+ name: Text Generation
47
+ dataset:
48
+ name: BBH (3-Shot)
49
+ type: BBH
50
+ args:
51
+ num_few_shot: 3
52
+ metrics:
53
+ - type: acc_norm
54
+ value: 61.1
55
+ name: normalized accuracy
56
+ source:
57
+ url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
58
+ name: Open LLM Leaderboard
59
+ - task:
60
+ type: text-generation
61
+ name: Text Generation
62
+ dataset:
63
+ name: MATH Lvl 5 (4-Shot)
64
+ type: hendrycks/competition_math
65
+ args:
66
+ num_few_shot: 4
67
+ metrics:
68
+ - type: exact_match
69
+ value: 52.42
70
+ name: exact match
71
+ source:
72
+ url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
73
+ name: Open LLM Leaderboard
74
+ - task:
75
+ type: text-generation
76
+ name: Text Generation
77
+ dataset:
78
+ name: GPQA (0-shot)
79
+ type: Idavidrein/gpqa
80
+ args:
81
+ num_few_shot: 0
82
+ metrics:
83
+ - type: acc_norm
84
+ value: 21.92
85
+ name: acc_norm
86
+ source:
87
+ url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
88
+ name: Open LLM Leaderboard
89
+ - task:
90
+ type: text-generation
91
+ name: Text Generation
92
+ dataset:
93
+ name: MuSR (0-shot)
94
+ type: TAUR-Lab/MuSR
95
+ args:
96
+ num_few_shot: 0
97
+ metrics:
98
+ - type: acc_norm
99
+ value: 18.12
100
+ name: acc_norm
101
+ source:
102
+ url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
103
+ name: Open LLM Leaderboard
104
+ - task:
105
+ type: text-generation
106
+ name: Text Generation
107
+ dataset:
108
+ name: MMLU-PRO (5-shot)
109
+ type: TIGER-Lab/MMLU-Pro
110
+ config: main
111
+ split: test
112
+ args:
113
+ num_few_shot: 5
114
+ metrics:
115
+ - type: acc
116
+ value: 54.51
117
+ name: accuracy
118
+ source:
119
+ url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Sakalti/ultiima-72B
120
+ name: Open LLM Leaderboard
121
+ ---
122
+ Built With Qwen
123
+ # merge
124
+
125
+ This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
126
+
127
+ ## Merge Details
128
+ ### Merge Method
129
+
130
+ This model was merged using the [TIES](https://arxiv.org/abs/2306.01708) merge method using [Qwen/Qwen2.5-72B](https://huggingface.co/Qwen/Qwen2.5-72B) as a base.
131
+
132
+ ### Models Merged
133
+
134
+ The following models were included in the merge:
135
+ * [Qwen/Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct)
136
+
137
+ ### Configuration
138
+
139
+ The following YAML configuration was used to produce this model:
140
+
141
+ ```yaml
142
+ models:
143
+ - model: Qwen/Qwen2.5-72B-Instruct
144
+ parameters:
145
+ weight: 1
146
+ density: 1
147
+ merge_method: ties
148
+ base_model: Qwen/Qwen2.5-72B
149
+ parameters:
150
+ weight: 1
151
+ density: 1
152
+ normalize: true
153
+ int8_mask: true
154
+ dtype: float16
155
+ ```
156
+ # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
157
+ Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/Sakalti__ultiima-72B-details)
158
+
159
+ | Metric |Value|
160
+ |-------------------|----:|
161
+ |Avg. |46.58|
162
+ |IFEval (0-Shot) |71.40|
163
+ |BBH (3-Shot) |61.10|
164
+ |MATH Lvl 5 (4-Shot)|52.42|
165
+ |GPQA (0-shot) |21.92|
166
+ |MuSR (0-shot) |18.12|
167
+ |MMLU-PRO (5-shot) |54.51|
168
+