Create README.md

README.md

@@ -0,0 +1,105 @@
+---
+license: mit
+language:
+- en
+base_model:
+- meta-llama/Llama-3.1-70B-Instruct
+---
+
+# Cakrawala-70B
+
+## Model Description
+
+Cakrawala-70B is a fine-tuned variant of the Llama-3.1-70B-Instruct model, specifically optimized for generating rich roleplaying conversations and character interactions. The model uses QLoRA (Quantized Low-Rank Adaptation) fine-tuning techniques to efficiently adapt the large language model for this specialized use case.
+
+## Intended Use
+
+### Primary Use Case
+Cakrawala-70B is designed specifically for generating high-quality roleplaying conversations with the following key characteristics:
+- Rich, descriptive character interactions
+- Consistent character voice and emotional development
+- Show-don't-tell emotional states
+- Clear separation between character perspectives
+- Structured turn-taking in conversations
+- Detailed physical descriptions and environmental awareness
+
+### Target Audience
+- Game developers creating interactive narratives
+- Writers seeking AI assistance in character development
+- RPG platforms and applications
+- Interactive fiction developers
+- Educational platforms teaching creative writing or character development
+
+## Training Data
+
+### Dataset Composition
+- Total examples: 5,867 conversation pairs
+- Format: JSON Lines (.jsonl)
+- Structure: Conversations field containing alternating messages between participants
+- Validation split: 5% of total data
+
+### Data Characteristics
+Each training example consists of:
+1. Character establishment prompts
+2. Multi-turn conversations (12-13 turns minimum)
+3. Rich descriptive elements including:
+   - Physical actions
+   - Facial expressions
+   - Tone indicators
+   - Environmental details
+   - Character reactions
+
+### Data Processing
+- Messages are structured with distinct role and content fields
+- Training focuses exclusively on completion tokens (train_on_inputs: false)
+- Input loss is excluded from calculations
+- Sequence length is set to 2048 tokens
+- Sample packing is enabled for efficient training
+
+## Training Details
+
+### Base Model
+- Architecture: meta-llama/Llama-3.1-70B-Instruct
+- Model Type: LlamaForCausalLM
+- Tokenizer: AutoTokenizer
+
+### Fine-tuning Approach
+- Method: QLoRA (Quantized Low-Rank Adaptation)
+- Quantization: 4-bit precision
+- Sequence Length: 2048 tokens
+- Training Duration: 3 epochs
+
+### LoRA Configuration
+- Rank (r): 32
+- Alpha: 64
+- Dropout: 0.1
+- Target Modules:
+  - Query Projection (q_proj)
+  - Key Projection (k_proj)
+  - Value Projection (v_proj)
+  - Output Projection (o_proj)
+
+### Training Parameters
+- Gradient Accumulation Steps: 16
+- Micro Batch Size: 4
+- Learning Rate: 0.0003
+- Optimizer: AdamW
+- Scheduler: Cosine
+- Mixed Precision: BF16 & FP16 with TF32 support
+
+## Performance Characteristics
+
+## Limitations
+
+Content Limitations:
+   - Training data size (5,867 examples) may limit variety in some scenarios
+   - Specialized for roleplaying conversations, may not generalize well to other tasks
+
+## Additional Information
+
+Special Tokens:
+- Pad Token: <|end_of_text|>
+
+Infrastructure:
+- Supports 8 x H100 NVL configuration
+- Utilizes 128 vCPU and 1509 GB RAM