|
|
---
|
|
|
license: mit
|
|
|
language:
|
|
|
- zho
|
|
|
- eng
|
|
|
- fra
|
|
|
- spa
|
|
|
- por
|
|
|
- deu
|
|
|
- ita
|
|
|
- rus
|
|
|
- jpn
|
|
|
- kor
|
|
|
- vie
|
|
|
- tha
|
|
|
- ara
|
|
|
base_model:
|
|
|
- Qwen/Qwen2.5-0.5B-Instruct
|
|
|
pipeline_tag: text-generation
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
# Zeta 1 - First Step of Zeta Project, This is Epoch.
|
|
|
Zeta 1 is the first step that I, an ordinary person and consumer, have taken in pursuit of my dream as an individual.
|
|
|
|
|
|
## About Zeta 1
|
|
|
Zeta 1 is an LLM with about 400 million parameters.
|
|
|
|
|
|
It might be better to call it an SLM.
|
|
|
|
|
|
Zeta 1 is an SLM that was painstakingly created on a consumer computer.
|
|
|
|
|
|
### Computer Spec
|
|
|
|
|
|
**Machine:** *Mac Mini (M2 Pro, 32GB RAM, 2023)*
|
|
|
|
|
|
This is a technical matter, but on a Mac with Apple Silicon you can only choose between CPU or MPS (Apple's proprietary API).
|
|
|
|
|
|
CPU has some compatibility but is too slow.
|
|
|
|
|
|
MPS is somewhat faster but doesn't allow the use of optimization systems such as fp16.
|
|
|
|
|
|
Zeta 1 was built using MPS.
|
|
|
|
|
|
### Arguments of Trainer
|
|
|
- **train epochs:** 3
|
|
|
- **warmup steps:** 100
|
|
|
|
|
|
### Datasets
|
|
|
Details of the dataset used can be found [here](https://github.com/DiamondGotCat/Zeta-Dataset/releases/tag/zeta-1)
|
|
|
|
|
|
## Links
|
|
|
GitHub: [Zeta](https://github.com/DiamondGotCat/Zeta)
|
|
|
|
|
|
---
|
|
|
|
|
|
Zeta is just a small SLM.
|
|
|
But don't forget that it has big dreams inside. |
