metadata
license: mit
language:
- zho
- eng
- fra
- spa
- por
- deu
- ita
- rus
- jpn
- kor
- vie
- tha
- ara
pipeline_tag: text-generation
datasets:
- DiamondGotCat/Zeta-1-Dataset
new_version: DiamondGotCat/Zeta-2
Zeta 1 - First Step of Zeta Project, This is Epoch.
Zeta 1 is the first step that I, an ordinary person and consumer, have taken in pursuit of my dream as an individual.
About Zeta 1
Zeta 1 is an LLM with about 400 million parameters.
It might be better to call it an SLM.
Zeta 1 is an SLM that was painstakingly created on a consumer computer.
Computer Spec
Machine: Mac Mini (M2 Pro, 32GB RAM, 2023)
This is a technical matter, but on a Mac with Apple Silicon you can only choose between CPU or MPS (Apple's proprietary API).
CPU has some compatibility but is too slow.
MPS is somewhat faster but doesn't allow the use of optimization systems such as fp16.
Zeta 1 was built using MPS.
Arguments of Trainer
- train epochs: 3
- warmup steps: 100
Datasets
Details of the dataset used can be found here
Links
GitHub: Zeta
Zeta is just a small SLM. But don't forget that it has big dreams inside.