---
library_name: transformers
tags:
- mergekit
- merge

---
# arco 2

This is a passthrough of arco with an experimental model. As you can see, it dramatically improved on arc challenge, only missing 1.2 points to get to the level of modern 3b baseline performance.

| Parameters | Model                          | MMLU  | ARC-C | HellaSwag | PIQA   | Winogrande | Average |
| -----------|--------------------------------|-------|-------|-----------|--------|------------|---------|
| 0.5b       | qwen2                          |44.13| 28.92| 49.05    | 69.31 | 56.99  | 49.68  |
| 0.5b       | arco (original)                |24.41  | 38.23 | 59.21     | 74.27  | 59.59      | 51.14   |
| 0.5b       | qwen2.5                        |**47.29**|31.83|52.17|70.29|57.06|51.72|
| 0.5b       | arco                           |26.17|37.29|62.88|74.37|**62.27**|52.60|
| 0.5b       | arco 2                         |25.51|**38.82**|**63.02**|**74.70**|61.25|**52.66**|