| | --- |
| | license: apache-2.0 |
| | tags: |
| | - conversational |
| | - efficient |
| | - i3-architecture |
| | - custom_code |
| | datasets: |
| | - starhopp3r/TinyChat |
| | language: |
| | - en |
| | pipeline_tag: text-generation |
| | library_name: transformers |
| | --- |
| | |
| | # i3 Model - Ultra-Efficient Pretraining Language Model |
| |
|
| | ## Model Description |
| |
|
| | The **i3 Model** is designed to optimize **pretraining efficiency** while retaining core language modeling capabilities. |
| | Its architecture allows training on **memory-constrained hardware**, including CPU-only setups, without sacrificing sequence modeling performance. |
| |
|
| | > [!Note] |
| | > The i3 architecture is present within the model for highly efficient pretraining. It is designed to **reduce memory usage**, **speed up training**, and allow pretraining from scratch on tiny hardware. Internal details are abstracted for simplicity. |
| |
|
| | --- |
| |
|
| | ## Use |
| |
|
| | ```python |
| | from transformers import pipeline |
| | |
| | pipe = pipeline("text-generation", model="FlameF0X/i3-12m") |
| | messages = [ |
| | {"role": "user", "content": "Who are you?"}, |
| | ] |
| | pipe(messages) |
| | ```` |
| |
|
| | --- |
| |
|
| | ## Model Statistics |
| |
|
| | * **Vocabulary Size:** 4,466 (variable-length chunks) |
| | * **Hidden Dimension:** 512 |
| | * **Number of Layers:** 12 |
| | * **Max Sequence Length:** 256 |
| | * **Total Parameters:** 12,691,186 |
| | * **Tokenization:** Memory-efficient variable-length chunking (2–3 characters) |
| |
|
| | * **Total tokens:** 334,524,736 |
| |
|
| | --- |
| |
|
| | ## Key Features |
| |
|
| | 1. **Memory-Optimized:** Designed to train on tiny hardware with minimal RAM usage |
| | 2. **Pretraining-Focused Architecture:** i3 layers provide efficient sequence modeling, low-rank linear updates, and factorized attention |
| | 3. **Variable-Length Tokenization:** 2–3 character chunks for compact embeddings |
| | 4. **Conversational Readiness:** Optimized for dialogue and text generation |
| |
|
| | --- |
| |
|
| | ## i3 Architecture (Abstract Overview) |
| |
|
| | ### Design Philosophy |
| |
|
| | The i3 model targets **CPU-friendly, memory-constrained pretraining**, emphasizing: |
| |
|
| | * Long-range sequence modeling |
| | * Low-rank weight updates for memory savings |
| | * Efficient factorized attention |
| | * 4-bit weights and microbatching for minimal memory footprint |
| |
|
| | ## Technologies used in the i3 Architecture that are open-sourced by me: |
| |
|
| | * [Low-Rank Pre-training](https://github.com/FlameF0X/Low-Rank-Pretraining) - LoRa for pre-training. |
| |
|
| | ### Conceptual Layout |
| |
|
| | ``` |
| | Input Tokens |
| | │ |
| | +-----------------+ |
| | | Embedding Layer | |
| | +-----------------+ |
| | │ |
| | +-----------------+ |
| | | i3 Architecture | |
| | +-----------------+ |
| | │ |
| | +------------------------+ |
| | | KQV Low-Rank Attention | |
| | +------------------------+ |
| | │ |
| | +-----------------------+ |
| | | LayerNorm + Residuals | |
| | +-----------------------+ |
| | │ |
| | +-------------------+ |
| | | Output Projection | |
| | +-------------------+ |
| | │ |
| | Predicted Tokens |
| | ``` |
| |
|
| | > Key idea: Every component is optimized for **memory efficiency** and **pretraining speed** on small hardware, while preserving essential transformer dynamics. |
| |
|
| | --- |
| |
|
| | ## Training Details |
| |
|
| | * **Sequence length:** 128–512 tokens |
| | * **Model size:** ~12M parameters (CPU-friendly) |
| | * **Optimizer:** AdamW or Lion (4-bit / mixed precision) |
| | * **Dataset:** TinyChat (~50–200 MB) |
| | * **Training loop:** gradient checkpointing + recomputation |
| | * **Objective:** token prediction / text generation |
| |
|
| |
|
| | ## Citation |
| |
|
| | ```bibtex |
| | @software{lorpt2025, |
| | title={LoRPt: Low-Rank Pretraining for Resource-Efficient Language Models}, |
| | author={[FlameF0X]}, |
| | year={2025}, |
| | url={https://github.com/FlameF0X/Low-Rank-Pretraining} |
| | } |
| | ``` |
