|
|
--- |
|
|
language: |
|
|
- en |
|
|
- multilingual |
|
|
license: gemma |
|
|
library_name: transformers |
|
|
tags: |
|
|
- vision-language |
|
|
- retrieval |
|
|
- colbert |
|
|
- late-interaction |
|
|
pipeline_tag: image-text-to-text |
|
|
--- |
|
|
|
|
|
# Merged ColGemma3 Model |
|
|
|
|
|
This model is a merged version of multiple ColGemma3 models using the **linear** merging technique. |
|
|
|
|
|
## Source Models |
|
|
|
|
|
1. [Nayana-cognitivelab/NayanaEmbed-ColGemma3-Modal-1848-colbert](https://huggingface.co/Nayana-cognitivelab/NayanaEmbed-ColGemma3-Modal-1848-colbert) |
|
|
2. [Nayana-cognitivelab/NayanaEmbed-ColGemma3-MultiGPU-merged-1610-22-colbert](https://huggingface.co/Nayana-cognitivelab/NayanaEmbed-ColGemma3-MultiGPU-merged-1610-22-colbert) |
|
|
|
|
|
|
|
|
## Merge Method: LINEAR |
|
|
|
|
|
Linear interpolation: Weighted average of model parameters. |
|
|
|
|
|
## Model Architecture |
|
|
|
|
|
ColGemma3 is a vision-language model for late interaction retrieval: |
|
|
- **Base**: Gemma3 vision-language model |
|
|
- **Vision Encoder**: Processes images into patch embeddings |
|
|
- **Custom Projection**: Projects embeddings to 128 dimensions |
|
|
- **Retrieval**: Uses MaxSim scoring for multi-vector retrieval |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from colpali_engine.models.gemma3.colgemma3 import ColGemma3, ColGemmaProcessor3 |
|
|
from PIL import Image |
|
|
import torch |
|
|
|
|
|
# Load model and processor |
|
|
model = ColGemma3.from_pretrained("Nayana-cognitivelab/NayanaEmbed-ColGemma3-Merge-Colbert-base-nayana-linear-v1", torch_dtype=torch.bfloat16, device_map="auto") |
|
|
processor = ColGemmaProcessor3.from_pretrained("Nayana-cognitivelab/NayanaEmbed-ColGemma3-Merge-Colbert-base-nayana-linear-v1") |
|
|
|
|
|
# Process images |
|
|
images = [Image.open("document.png")] |
|
|
batch_images = processor.process_images(images).to(model.device) |
|
|
|
|
|
# Process queries |
|
|
queries = ["What is this document about?"] |
|
|
batch_queries = processor.process_queries(queries).to(model.device) |
|
|
|
|
|
# Generate embeddings |
|
|
with torch.no_grad(): |
|
|
img_embeddings = model(**batch_images) |
|
|
query_embeddings = model(**batch_queries) |
|
|
|
|
|
# Compute similarity scores |
|
|
scores = processor.score([query_embeddings[0]], [img_embeddings[0]]) |
|
|
``` |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model, please cite the original ColGemma3 work and the source models. |
|
|
|
|
|
--- |
|
|
|
|
|
*This model was automatically merged using [Modal](https://modal.com) infrastructure.* |
|
|
|
