Update README.md

README.md

@@ -1,18 +1,21 @@
-# Mineral Nano 1
+# Mineral Nano 1 Vision
 
-Mineral Nano 1 is a compact, efficient language model designed for fast inference and low-resource environments.
+Mineral Nano 1 Vision is a compact, efficient vision-language model designed for fast inference and low-resource environments with multimodal capabilities.
 
 ## Model Details
 
 - **Model Name:** mineral-nano-1
-- **Model Type:** Causal Language Model
-- **Parameters:** ~85M parameters
+- **Model Type:** Vision-Language Model (VLM)
+- **Parameters:** ~110M parameters
 - **Context Length:** 2048 tokens
-- **Architecture:** Transformer-based decoder with 12 layers
+- **Architecture:** Transformer-based decoder with vision encoder (12 layers)
 - **Precision:** BFloat16
+- **Image Resolution:** 224x224
+- **Modalities:** Text + Images
 
 ## Architecture
 
+### Language Model
 - **Hidden Size:** 768
 - **Intermediate Size:** 3072
 - **Attention Heads:** 12
@@ -20,59 +23,148 @@ Mineral Nano 1 is a compact, efficient language model designed for fast inferenc
 - **Vocabulary Size:** 32,000 tokens
 - **Positional Encoding:** RoPE (Rotary Position Embeddings)
 
+### Vision Encoder
+- **Image Size:** 224x224
+- **Patch Size:** 16x16
+- **Hidden Size:** 768
+- **Layers:** 12
+- **Image Tokens:** 196 per image
+- **Architecture:** ViT-style encoder
+
 ## Usage
 
-### Basic Text Generation
+### Installation
+
+```bash
+pip install transformers pillow torch
+```
+
+### Basic Image Understanding
 
 ```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers import AutoProcessor, AutoModelForVision2Seq
+from PIL import Image
+import requests
 
 model_name = "your-username/mineral-nano-1"
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(model_name)
+processor = AutoProcessor.from_pretrained(model_name)
+model = AutoModelForVision2Seq.from_pretrained(model_name)
+
+# Load an image
+url = "https://example.com/image.jpg"
+image = Image.open(requests.get(url, stream=True).raw)
+
+# Prepare inputs
+prompt = "<image>What is in this image?"
+inputs = processor(text=prompt, images=image, return_tensors="pt")
 
-prompt = "Once upon a time"
-inputs = tokenizer(prompt, return_tensors="pt")
+# Generate response
 outputs = model.generate(**inputs, max_new_tokens=100)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+response = processor.decode(outputs[0], skip_special_tokens=True)
+print(response)
+```
+
+### Multiple Images
+
+```python
+from PIL import Image
+
+images = [
+    Image.open("image1.jpg"),
+    Image.open("image2.jpg")
+]
+
+prompt = "<image>Describe the first image. <image>Now describe the second image."
+inputs = processor(text=prompt, images=images, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=200)
+print(processor.decode(outputs[0], skip_special_tokens=True))
 ```
 
-### Chat Format
+### Chat with Images
 
 ```python
 messages = [
-    {"role": "system", "content": "You are a helpful assistant."},
-    {"role": "user", "content": "What is machine learning?"}
+    {
+        "role": "user",
+        "content": [
+            {"type": "image"},
+            {"type": "text", "text": "What objects are in this image?"}
+        ]
+    }
 ]
 
-input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
-inputs = tokenizer(input_text, return_tensors="pt")
-outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+# Apply chat template
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = processor(text=text, images=image, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=150, temperature=0.7)
+print(processor.decode(outputs[0], skip_special_tokens=True))
+```
+
+### Local Images
+
+```python
+from PIL import Image
+
+# Load local image
+image = Image.open("path/to/your/image.jpg")
+
+prompt = "<image>Describe what you see in detail."
+inputs = processor(text=prompt, images=image, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=256)
+print(processor.decode(outputs[0], skip_special_tokens=True))
 ```
 
 ## Training Details
 
 - **Framework:** PyTorch with Transformers
-- **Training Data:** [Specify your training dataset]
+- **Training Data:** Text + Image pairs
 - **Training Duration:** [Specify training time]
 - **Hardware:** [Specify GPUs used]
+- **Vision Encoder:** Pretrained ViT encoder fine-tuned with language model
+
+## Capabilities
+
+✅ Image description and captioning
+✅ Visual question answering
+✅ Object detection and recognition
+✅ Scene understanding
+✅ Multi-image reasoning
+✅ OCR and text extraction from images
 
 ## Limitations
 
-- Limited context window (2048 tokens)
-- May produce inconsistent outputs on complex reasoning tasks
-- Best suited for short-form text generation
-- Compact size means reduced capabilities compared to larger models
+- Limited to 224x224 resolution images
+- Context window of 2048 tokens including image tokens
+- May struggle with fine-grained details
+- Best for general image understanding tasks
+- Compact size means reduced capabilities compared to larger VLMs
+- Limited multilingual vision capabilities
 
 ## Intended Use
 
 This model is designed for:
-- Educational purposes
-- Prototyping and experimentation
+- Educational purposes and learning VLM architectures
+- Prototyping multimodal applications
 - Low-resource deployment scenarios
-- Fast inference applications
-- Personal projects
+- Fast inference with vision capabilities
+- Mobile and edge device applications
+- Personal projects requiring image understanding
+
+## Image Preprocessing
+
+Images are automatically:
+- Resized to 224x224
+- Normalized with CLIP-style statistics
+- Converted to RGB
+- Split into 16x16 patches (196 total patches)
+
+## Performance Tips
+
+- Use square images when possible for best results
+- Ensure images are clear and well-lit
+- Keep prompts concise and specific
+- Use batch processing for multiple images
+- Enable `use_cache=True` for faster generation
 
 ## License
 
@@ -81,9 +173,9 @@ This model is designed for:
 ## Citation
 
 ```bibtex
-@misc{mineral-nano-1,
+@misc{mineral-nano-1-vision,
   author = {Your Name},
-  title = {Mineral Nano 1: A Compact Language Model},
+  title = {Mineral Nano 1 Vision: A Compact Vision-Language Model},
   year = {2025},
   publisher = {HuggingFace},
   url = {https://huggingface.co/your-username/mineral-nano-1}
@@ -92,4 +184,8 @@ This model is designed for:
 
 ## Contact
 
-For questions or issues, please open an issue on the model repository.
+For questions or issues, please open an issue on the model repository.
+
+## Acknowledgments
+
+This model builds upon research in vision transformers and multimodal learning.