File size: 2,196 Bytes
90497a2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
---
language:
- en
- multilingual
license: gemma
library_name: transformers
tags:
- vision-language
- retrieval
- colbert
- late-interaction
pipeline_tag: image-text-to-text
---
# Merged ColGemma3 Model
This model is a merged version of multiple ColGemma3 models using the **linear** merging technique.
## Source Models
1. [Nayana-cognitivelab/NayanaEmbed-ColGemma3-Modal-1848-colbert](https://huggingface.co/Nayana-cognitivelab/NayanaEmbed-ColGemma3-Modal-1848-colbert)
2. [Nayana-cognitivelab/NayanaEmbed-ColGemma3-MultiGPU-merged-1610-22-colbert](https://huggingface.co/Nayana-cognitivelab/NayanaEmbed-ColGemma3-MultiGPU-merged-1610-22-colbert)
## Merge Method: LINEAR
Linear interpolation: Weighted average of model parameters.
## Model Architecture
ColGemma3 is a vision-language model for late interaction retrieval:
- **Base**: Gemma3 vision-language model
- **Vision Encoder**: Processes images into patch embeddings
- **Custom Projection**: Projects embeddings to 128 dimensions
- **Retrieval**: Uses MaxSim scoring for multi-vector retrieval
## Usage
```python
from colpali_engine.models.gemma3.colgemma3 import ColGemma3, ColGemmaProcessor3
from PIL import Image
import torch
# Load model and processor
model = ColGemma3.from_pretrained("Nayana-cognitivelab/NayanaEmbed-ColGemma3-Merge-Colbert-base-nayana-linear-v1", torch_dtype=torch.bfloat16, device_map="auto")
processor = ColGemmaProcessor3.from_pretrained("Nayana-cognitivelab/NayanaEmbed-ColGemma3-Merge-Colbert-base-nayana-linear-v1")
# Process images
images = [Image.open("document.png")]
batch_images = processor.process_images(images).to(model.device)
# Process queries
queries = ["What is this document about?"]
batch_queries = processor.process_queries(queries).to(model.device)
# Generate embeddings
with torch.no_grad():
img_embeddings = model(**batch_images)
query_embeddings = model(**batch_queries)
# Compute similarity scores
scores = processor.score([query_embeddings[0]], [img_embeddings[0]])
```
## Citation
If you use this model, please cite the original ColGemma3 work and the source models.
---
*This model was automatically merged using [Modal](https://modal.com) infrastructure.*
|