README.md

---
license: apache-2.0
tags:
- conversational
- efficient
- i3-architecture
- custom_code
datasets:
- starhopp3r/TinyChat
language:
- en
pipeline_tag: text-generation
library_name: transformers
---

# i3 Model - Ultra-Efficient Pretraining Language Model

## Model Description

The **i3 Model** is designed to optimize **pretraining efficiency** while retaining core language modeling capabilities.  
Its architecture allows training on **memory-constrained hardware**, including CPU-only setups, without sacrificing sequence modeling performance.  

> [!Note]  
> The i3 architecture is present within the model for highly efficient pretraining. It is designed to **reduce memory usage**, **speed up training**, and allow pretraining from scratch on tiny hardware. Internal details are abstracted for simplicity.

---

## Use

```python
from transformers import pipeline

pipe = pipeline("text-generation", model="FlameF0X/i3-12m")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)
````

---

## Model Statistics

* **Vocabulary Size:** 4,466 (variable-length chunks)
* **Hidden Dimension:** 512
* **Number of Layers:** 12
* **Max Sequence Length:** 256
* **Total Parameters:** 12,691,186
* **Tokenization:** Memory-efficient variable-length chunking (2–3 characters)

  * **Total tokens:** 334,524,736

---

## Key Features

1. **Memory-Optimized:** Designed to train on tiny hardware with minimal RAM usage
2. **Pretraining-Focused Architecture:** i3 layers provide efficient sequence modeling, low-rank linear updates, and factorized attention
3. **Variable-Length Tokenization:** 2–3 character chunks for compact embeddings
4. **Conversational Readiness:** Optimized for dialogue and text generation

---

## i3 Architecture (Abstract Overview)

### Design Philosophy

The i3 model targets **CPU-friendly, memory-constrained pretraining**, emphasizing:

* Long-range sequence modeling
* Low-rank weight updates for memory savings
* Efficient factorized attention
* 4-bit weights and microbatching for minimal memory footprint

## Technologies used in the i3 Architecture that are open-sourced by me:

* [Low-Rank Pre-training](https://github.com/FlameF0X/Low-Rank-Pretraining) - LoRa for pre-training.

### Conceptual Layout

```
Input Tokens
    │
+-----------------+
| Embedding Layer |
+-----------------+
    │
+-----------------+
| i3 Architecture |
+-----------------+
    │
+------------------------+
| KQV Low-Rank Attention |
+------------------------+
    │
+-----------------------+
| LayerNorm + Residuals |
+-----------------------+
    │
+-------------------+
| Output Projection |
+-------------------+
    │
Predicted Tokens
```

> Key idea: Every component is optimized for **memory efficiency** and **pretraining speed** on small hardware, while preserving essential transformer dynamics.

---

## Training Details

* **Sequence length:** 128–512 tokens
* **Model size:** ~12M parameters (CPU-friendly)
* **Optimizer:** AdamW or Lion (4-bit / mixed precision)
* **Dataset:** TinyChat (~50–200 MB)
* **Training loop:** gradient checkpointing + recomputation
* **Objective:** token prediction / text generation


## Citation

```bibtex
@software{lorpt2025,
  title={LoRPt: Low-Rank Pretraining for Resource-Efficient Language Models},
  author={[FlameF0X]},
  year={2025},
  url={https://github.com/FlameF0X/Low-Rank-Pretraining}
}
```