caca-5M-untrained / README.md

Lyon28

Upload README.md with huggingface_hub

3422a99 verified about 9 hours ago

preview code

raw

history blame contribute delete

23.8 kB

metadata

license: apache-2.0
language:
  - id
  - en
tags:
  - text-generation
  - pytorch
  - causal-lm
  - transformer
  - untrained
  - gqa
  - rope
  - swiglu
  - rmsnorm
  - flash-attention
  - indonesian
library_name: transformers
pipeline_tag: text-generation
widget:
  - text: Jakarta adalah ibu kota
    example_title: 🇮🇩 Text Completion (ID)
  - text: |
      Pertanyaan: Apa itu kecerdasan buatan?
      Jawaban:
    example_title: 🇮🇩 Question Answering (ID)
  - text: |
      Tulis cerita pendek tentang robot yang belajar mencintai.
    example_title: 🇮🇩 Creative Writing (ID)
  - text: The capital of Indonesia is
    example_title: 🇬🇧 Text Completion (EN)
  - text: |
      Question: What is artificial intelligence?
      Answer:
    example_title: 🇬🇧 Question Answering (EN)
  - text: |
      def fibonacci(n):
          """Hitung bilangan fibonacci ke-n"""
    example_title: 💻 Code Completion
  - text: |
      def reverse_string(s):
    example_title: 💻 Code Generation
  - text: |
      User: Halo! Siapa kamu?
      Assistant:
    example_title: 💬 Chat Format (ID)
  - text: |
      User: Jelaskan tentang machine learning dalam 2 kalimat.
      Assistant:
    example_title: 💬 Conversational (ID)
inference:
  parameters:
    max_new_tokens: 100
    temperature: 0.7
    top_p: 0.9
    top_k: 50
    do_sample: true
    repetition_penalty: 1.1
    num_beams: 1
datasets: []
metrics:
  - perplexity
model-index:
  - name: caca-5M
    results: []

🚀 CACA-5M

Model Transformer Modern dengan Arsitektur Canggih

24,253,696 parameters • 24.3M • 8 layers

📖 Dokumentasi • 🚀 Quick Start • 💡 Fitur • 🔧 Training • 📊 Spesifikasi

⚠️ PENTING: Model Belum Dilatih (Untrained)

PERHATIAN: Ini adalah model yang belum melalui proses training. Bobot model masih dalam kondisi random initialization. Output yang dihasilkan akan tidak bermakna dan acak.

Status Model:

🔴 Belum dilatih - Bobot masih random
🟡 Hanya untuk riset - Eksperimen arsitektur & training
🟢 Siap dilatih - Arsitektur sudah teruji

Widget di atas hanya menunjukkan format input yang diharapkan. Setelah model dilatih dengan dataset yang tepat, format yang sama akan menghasilkan output berkualitas.

📋 Deskripsi

Caca adalah arsitektur Large Language Model (LLM) generasi terbaru yang menggabungkan berbagai teknik state-of-the-art dalam deep learning. Model ini dirancang dengan fokus pada efisiensi, skalabilitas, dan performa tinggi.

Caca itu eksperimen open-source Indonesian LLM yang dibuat dari nol secara individual dan bertahap. Bukan kompetitor siapa-siapa, cuma pengen eksplorasi apa yang bisa dilakukan dengan budget terbatas, passion unlimited, dan mindset collaborative. Kalau berguna buat orang lain, alhamdulillah. Kalau enggak, ya tetap fun kok.

Ini proyek eksplorasi, jadi kalau gagal ya bagian dari proses belajar. Kalau berhasil, itu bonus.

🎯 Keunggulan Utama

🇮🇩 Bilingual Support: Optimized untuk Bahasa Indonesia & English
⚡ Ultra Fast: Flash Attention 2 untuk inferensi 3x lebih cepat
💾 Memory Efficient: Grouped Query Attention menghemat 75% KV cache
🎯 Long Context: Support hingga 2,048 token
🔧 Modular: Arsitektur fleksibel dengan berbagai opsi konfigurasi

✨ Fitur Utama

🎯 Core Features

✅ Grouped Query Attention (GQA) - Efisiensi memori dan komputasi superior
- Query heads: 4
- KV heads: 2
- Ratio: 2:1 (hemat 75% KV cache)
✅ Rotary Position Embeddings (RoPE) - Generalisasi konteks panjang lebih baik
- Theta: 10000
- Support extrapolation untuk konteks > training length
✅ RMSNorm - Normalisasi lebih stabil dan 50% lebih cepat dari LayerNorm
- Epsilon: 1e-06
✅ SwiGLU Activation - Performa 10-15% lebih baik dari ReLU/GELU
- Intermediate size: 1,024
✅ Flash Attention 2 - Akselerasi hingga 3x dengan memory efficiency
- Otomatis aktif jika tersedia CUDA

🔥 Advanced Features

🎯 Attention Mechanisms

⚡ Flash Attention v2 - 3x faster with IO-aware algorithm
🔑 Grouped Query Attention (GQA) - 4Q : 2KV heads
🚀 xFormers Support - Memory efficient attention fallback
🎯 PyTorch SDPA - Native scaled dot product attention

📍 Position Encodings

🔄 RoPE - Rotary embeddings (θ=10000)

🪟 Long Context Features

🎓 Training Optimizations

💾 Gradient Checkpointing - Memory efficient training
🎯 Mixed Precision - BF16 & FP16 support

📦 Quantization Support

4️⃣ 4-bit Quantization - NF4, FP4 via bitsandbytes
8️⃣ 8-bit Quantization - LLM.int8() support
🔄 Double Quantization - Further compression

🛠️ Optimization Features

💾 KV Cache - Generasi autoregressive 5-10x lebih cepat
🔧 Gradient Checkpointing - Training model besar dengan memory terbatas
📦 Quantization Ready - Support 4-bit & 8-bit quantization
🎯 Mixed Precision Training - BF16 & FP16 support

📊 Spesifikasi Teknis

Spesifikasi	Detail
💎 Total Parameters	24,253,696 (24.3M)
📐 Hidden Size	256
🔢 Intermediate Size	1,024
🏗️ Num Layers	8
🎯 Attention Heads	4
🔑 KV Heads	2 (GQA)
📏 Head Dimension	64
📚 Vocab Size	32,000 tokens
📖 Max Context	2,048 tokens
🏛️ Architecture	Decoder-only Transformer
🎨 Model Type	Causal Language Model

📐 Arsitektur Detail

🔍 Klik untuk lihat struktur lengkap

CacaForCausalLM (24.3M)
│
├─ Embedding Layer
│  └─ Token Embeddings: 32,000 × 256
│     └─ Parameters: 8,192,000
│
├─ Transformer Layers (8x)
│  │
│  ├─ Layer {i} (repeated 8 times)
│  │  │
│  │  ├─ Input LayerNorm (RMSNorm)
│  │  │  └─ Params: 256
│  │  │
│  │  ├─ Self-Attention (Grouped Query Attention)
│  │  │  ├─ Q Projection: 256 → 256
│  │  │  ├─ K Projection: 256 → 128
│  │  │  ├─ V Projection: 256 → 128
│  │  │  ├─ O Projection: 256 → 256
│  │  │  ├─ RoPE Embeddings: θ=10000
│  │  │  └─ Flash Attention 2 (if available)
│  │  │
│  │  ├─ Post-Attention LayerNorm (RMSNorm)
│  │  │  └─ Params: 256
│  │  │
│  │  ├─ MLP (SwiGLU)
│  │  │  ├─ Gate: 256 → 1,024
│  │  │  ├─ Up: 256 → 1,024
│  │  │  ├─ Activation: SiLU (Swish)
│  │  │  └─ Down: 1,024 → 256
│  │  │
│  │  └─ Residual Connections (2x per layer)
│  │
│  └─ Total Layer Params: ~0M per layer
│
├─ Final LayerNorm (RMSNorm)
│  └─ Params: 256
│
└─ LM Head (Output Projection)
   └─ Linear: 256 → 32,000
      └─ Parameters: 8,192,000

Perhitungan Parameter:

Embeddings: 32,000 × 256 = 8,192,000
Layers: 8 layers × ~0M = ~6M
Total: 24,253,696 parameters

🚀 Quick Start

📦 Instalasi

# Dependencies dasar
pip install torch>=2.0.0 transformers>=4.35.0 accelerate safetensors

# Optional: Untuk performa maksimal
pip install flash-attn --no-build-isolation  # Flash Attention 2
pip install xformers                          # Memory efficient attention
pip install bitsandbytes                      # Quantization support

💻 Penggunaan Dasar

1️⃣ Load Model

from transformers import AutoModelForCausalLM, AutoConfig
import torch

# Load configuration
config = AutoConfig.from_pretrained(
    "Lyon28/caca-5M-untrained",
    trust_remote_code=True
)

print(f"Model: {config.model_type}")
print(f"Parameters: 24,253,696")
print(f"Hidden size: {config.hidden_size}")
print(f"Layers: {config.num_hidden_layers}")

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "Lyon28/caca-5M-untrained",
    config=config,
    torch_dtype=torch.bfloat16,  # Gunakan BF16 untuk efisiensi
    device_map="auto",            # Otomatis distribusi ke GPU
    trust_remote_code=True
)

print(f"Model loaded! Device: {model.device}")

2️⃣ Verifikasi Model

# Hitung total parameter
total_params = sum(p.numel() for p in model.parameters())
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f"Total parameters: {total_params:,}")
print(f"Trainable parameters: {trainable_params:,}")
print(f"Model size: {total_params * 2 / 1e9:.2f} GB (BF16)")

# Test forward pass
batch_size, seq_len = 2, 10
input_ids = torch.randint(0, config.vocab_size, (batch_size, seq_len))
input_ids = input_ids.to(model.device)

with torch.no_grad():
    outputs = model(input_ids)

print(f"Output shape: {outputs.logits.shape}")
print("✅ Model berfungsi dengan baik!")

3️⃣ Generate Text (Setelah Training)

from transformers import AutoTokenizer

# Load tokenizer (gunakan tokenizer yang sesuai)
tokenizer = AutoTokenizer.from_pretrained("your-tokenizer-here")

# Prepare input
text = "Jelaskan tentang kecerdasan buatan"
inputs = tokenizer(text, return_tensors="pt").to(model.device)

# Generate
outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    do_sample=True,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.eos_token_id
)

# Decode
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

🔧 Training Guide

📚 Persiapan Dataset

from datasets import load_dataset

# Load dataset (contoh)
dataset = load_dataset("indonesian-nlp/id-wikipedia")

# Atau load dari file lokal
from datasets import Dataset
import pandas as pd

df = pd.read_csv("your_data.csv")
dataset = Dataset.from_pandas(df)

print(f"Dataset size: {len(dataset)}")

🎯 Training Configuration

from transformers import Trainer, TrainingArguments
from transformers import DataCollatorForLanguageModeling

# Training arguments
training_args = TrainingArguments(
    # Output
    output_dir="./caca-caca-5M-trained",
    run_name="caca-caca-5M-v1",
    
    # Training
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,  # Effective batch size = 32
    learning_rate=2e-4,
    weight_decay=0.1,
    warmup_steps=2000,
    
    # Optimization
    bf16=True,                      # Mixed precision training
    gradient_checkpointing=True,     # Hemat memory
    optim="adamw_torch_fused",      # Optimizer tercepat
    max_grad_norm=1.0,
    
    # Logging & Evaluation
    logging_steps=10,
    logging_first_step=True,
    eval_strategy="steps",
    eval_steps=500,
    save_steps=1000,
    save_total_limit=3,
    
    # Hub integration
    push_to_hub=True,
    hub_model_id="your-username/caca-caca-5M-trained",
    hub_strategy="every_save",
    
    # Distributed training
    ddp_find_unused_parameters=False,
    dataloader_num_workers=4,
)

# Data collator
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False  # Causal LM, bukan Masked LM
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
    data_collator=data_collator,
)

# Train!
print("🚀 Starting training...")
trainer.train()

# Save final model
print("💾 Saving model...")
trainer.save_model("./caca-caca-5M-final")
trainer.push_to_hub()

print("✅ Training complete!")

📊 Estimasi Resource

💰 Klik untuk melihat estimasi biaya & waktu training

Hardware Requirements:

GPU	Memory	Batch Size	Speed	Est. Time (100B tokens)
RTX 3090 (24GB)	24GB	1-2	~1K tok/s	~30 hari
A100 (40GB)	40GB	4-8	~5K tok/s	~6 hari
A100 (80GB)	80GB	8-16	~8K tok/s	~4 hari
8×A100 (80GB)	640GB	64+	~50K tok/s	~14 jam

Cloud Costs (approximate):

AWS p4d.24xlarge (8×A100): $32/hour × 24 hours = **$768/day**
GCP a2-ultragpu-8g: $30/hour × 24 hours = **$720/day**
Lambda Labs (8×A100): $15/hour × 24 hours = **$360/day**

Tips menghemat biaya:

Gunakan spot instances (60-70% lebih murah)
Gradient accumulation untuk batch size lebih besar
Mixed precision (BF16) untuk 2x speedup
Gradient checkpointing untuk hemat memory

💬 Format Chat

Model ini mendukung format chat standar:

# Single-turn
messages = [
    {"role": "user", "content": "Halo! Siapa kamu?"},
]

# Multi-turn conversation
messages = [
    {"role": "system", "content": "Kamu adalah asisten AI yang membantu."},
    {"role": "user", "content": "Jelaskan tentang fotosintesis"},
    {"role": "assistant", "content": "Fotosintesis adalah proses..."},
    {"role": "user", "content": "Apa manfaatnya bagi manusia?"},
]

# Apply chat template
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

print(formatted)
# Output:
# System: Kamu adalah asisten AI yang membantu.
#
# User: Jelaskan tentang fotosintesis
# Assistant: Fotosintesis adalah proses...
# User: Apa manfaatnya bagi manusia?
# Assistant:

🎯 Use Cases

✅ Cocok Untuk:

🔬 Penelitian: Eksperimen arsitektur LLM modern
📚 Edukasi: Belajar tentang transformer & training
🎓 Akademis: Paper, thesis, project
🚀 Base Model: Fine-tuning untuk task spesifik
💡 Proof of Concept: Test ide sebelum scale up

❌ Tidak Cocok Untuk:

🚫 Production: Model belum dilatih
🚫 Real-world apps: Output masih random
🚫 Safety-critical: Belum ada safety alignment
🚫 Direct deployment: Perlu training dulu

📖 Dokumentasi

🔗 Links Penting

📚 Hugging Face Docs: transformers.github.io
💻 GitHub: Lyon-28/caca-transformers
💬 Discussions: Model discussions
🐛 Issues: Report bugs

📝 Related Models

Model Size	Parameters	Link
🐣 Tiny	1M - 50M	caca-1M to caca-50M
🐥 Small	75M - 500M	caca-75M to caca-500M
🦅 Medium	600M - 1B	caca-600M to caca-1B
🦁 Large	1.5B - 5B	caca-1.5B to caca-5B
🐉 XL	6B - 10B	caca-6B to caca-10B
🦖 XXL	12B+	caca-12B to caca-70B

🤝 Contributing

Kami sangat terbuka untuk kontribusi! Beberapa cara untuk berkontribusi:

🐛 Report bugs: Temukan bug? Buka issue
💡 Suggest features: Punya ide? Share di discussions
📝 Improve docs: PR welcome untuk dokumentasi
🎓 Share results: Training hasil? Share di model card
⭐ Star & Share: Bantu project ini berkembang

📜 License & Citation

📄 License

Model ini dirilis di bawah Apache License 2.0:

✅ Gratis untuk penggunaan komersial
✅ Gratis untuk penggunaan riset
✅ Boleh modifikasi & distribusi
✅ Tidak ada garansi

📚 Citation

Jika Anda menggunakan model ini dalam penelitian atau project, mohon cite:

@misc{cacacaca-5M2025,
  author = {Lyon},
  title = {Caca-caca-5M: Modern Transformer Architecture with GQA and Advanced Features},
  year = {2025},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{https://huggingface.co/Lyon28/caca-5M-untrained}},
}

🙏 Acknowledgments

Model ini terinspirasi dan mengimplementasikan berbagai penelitian terkini:

🏗️ Core Architecture

LLaMA (Meta AI, 2023) - Base decoder-only architecture, RMSNorm, SwiGLU
- Paper: LLaMA: Open and Efficient Foundation Language Models
GPT-3 (OpenAI, 2020) - Transformer language modeling paradigm
PaLM (Google, 2022) - SwiGLU activation function

🎯 Attention Mechanisms

Flash Attention v2 (Tri Dao et al., 2023) - Efficient attention with IO-awareness
- Paper: FlashAttention-2: Faster Attention with Better Parallelism
Grouped Query Attention (GQA) (Ainslie et al., Google, 2023) - Memory-efficient attention
- Paper: GQA: Training Generalized Multi-Query Transformer
Multi-Query Attention (MQA) (Shazeer, Google, 2019) - Fast decoding
xFormers (Meta AI, 2022) - Memory efficient attention implementations
PyTorch SDPA (PyTorch Team, 2023) - Built-in scaled dot product attention

📍 Position Encodings

RoPE (Su et al., EleutherAI, 2021) - Rotary Position Embeddings
- Paper: RoFormer: Enhanced Transformer with Rotary Position Embedding
ALiBI (Press et al., 2022) - Attention with Linear Biases for extrapolation
- Paper: Train Short, Test Long: Attention with Linear Biases
YaRN (Peng et al., 2023) - Yet another RoPE extensioN for long context
- Paper: YaRN: Efficient Context Window Extension

🪟 Long Context & Efficiency

Sliding Window Attention (Mistral AI, 2023) - Local attention patterns
- Paper: Mistral 7B
StreamingLLM / Attention Sink (Xiao et al., MIT, 2023) - Infinite sequence lengths
- Paper: Efficient Streaming Language Models with Attention Sinks
Logit Softcapping (Google Gemma, 2024) - Prevent attention overflow
- Paper: Gemma: Open Models Based on Gemini

🧠 Mixture of Experts (MoE)

Mixtral 8x7B (Mistral AI, 2024) - Sparse MoE architecture
- Paper: Mixtral of Experts
Switch Transformers (Fedus et al., Google, 2021) - Scaling with expert choice
- Paper: Switch Transformers: Scaling to Trillion Parameter Models
GLaM (Du et al., Google, 2021) - Generalist Language Model with MoE
Expert Choice Routing (Zhou et al., Google, 2022) - Improved load balancing

🎓 Training Optimizations

Layer Scale (Touvron et al., Meta, 2021) - Training stability for deep networks
- Paper: Going Deeper with Image Transformers (CaiT)
Stochastic Depth (Huang et al., 2016) - Regularization via random layer dropping
- Paper: Deep Networks with Stochastic Depth
Mixture of Depths (MoD) (Raposo et al., Google DeepMind, 2024) - Dynamic compute allocation
- Paper: Mixture-of-Depths: Dynamically allocating compute in transformer-based models
Gradient Checkpointing (Chen et al., 2016) - Memory-efficient training

📦 Quantization

LLM.int8() (Dettmers et al., 2022) - 8-bit matrix multiplication
- Paper: LLM.int8(): 8-bit Matrix Multiplication for Transformers
QLoRA (Dettmers et al., 2023) - 4-bit quantized LoRA fine-tuning
- Paper: QLoRA: Efficient Finetuning of Quantized LLMs
GPTQ (Frantar et al., 2022) - Post-training quantization
bitsandbytes (Dettmers) - Efficient quantization library

🎨 Multimodal Components

Vision Transformer (ViT) (Dosovitskiy et al., Google, 2020) - Image encoding
- Paper: An Image is Worth 16x16 Words
Perceiver Resampler (Alayrac et al., DeepMind, 2022) - Multimodal fusion
- Paper: Flamingo: a Visual Language Model
Q-Former (Li et al., Salesforce, 2023) - Query-based multimodal alignment
- Paper: BLIP-2: Bootstrapping Language-Image Pre-training
Whisper (Radford et al., OpenAI, 2022) - Audio encoding inspiration

🛠️ Normalization & Activations

RMSNorm (Zhang & Sennrich, 2019) - Root Mean Square Layer Normalization
- Paper: Root Mean Square Layer Normalization
SwiGLU (Shazeer, Google, 2020) - GLU activation variant
- Paper: GLU Variants Improve Transformer

🔧 Implementation & Tools

Hugging Face Transformers - Model implementation framework
PyTorch - Deep learning framework
Safetensors - Secure tensor serialization format
Accelerate - Distributed training utilities

Special Thanks to:

🇮🇩 Indonesian NLP Community
🤗 Hugging Face Team
🔬 Open source AI research community

⚠️ Limitations & Bias

Keterbatasan

🔴 Untrained: Model belum dilatih, output random
🟡 No Tokenizer: Perlu prepare tokenizer sendiri
🟡 No Safety: Belum ada content filtering/alignment
🟠 Memory Intensive: Training butuh GPU besar

Potential Biases

Model ini akan mewarisi bias dari data training yang digunakan. Mohon perhatikan:

Bahasa: Bias terhadap bahasa mayoritas di dataset
Kultur: Bias terhadap perspektif kultur tertentu
Gender & Demografis: Potential stereotypes
Faktual: Bisa generate informasi tidak akurat

Rekomendasi: Lakukan evaluation & filtering sebelum deployment.

📞 Support & Contact

💬 Community

Discussions: HF Discussions

📧 Contact

Untuk pertanyaan atau kolaborasi:

Email: cacatransformers@gmail.com
HF Profile: @Lyon28

🌟 Star History

💝 Dibuat dengan ❤️ untuk komunitas AI Indonesia

Terima kasih telah menggunakan Caca!

Jika project ini bermanfaat, consider untuk:

⭐ Star repository ini
🔗 Share ke teman-teman
💬 Join discussions
🤝 Contribute ke project

Quote Dari caca