Update README.md
Browse files
README.md
CHANGED
|
@@ -19,19 +19,19 @@ The model only takes images as document-side inputs and produce vectors represen
|
|
| 19 |
|
| 20 |
# News
|
| 21 |
|
| 22 |
-
- 2024-07-14: We released **huggingface demo**! Try our [online demo](https://huggingface.co/spaces/bokesyo/minicpm-visual-embeeding-v0-demo)!
|
| 23 |
|
| 24 |
-
- 2024-07-14: We released a **Gradio demo** of `miniCPM-visual-embedding-v0`, take a look at [pipeline_gradio.py](https://huggingface.co/RhapsodyAI/minicpm-visual-embedding-v0/blob/main/pipeline_gradio.py). You can run `pipeline_gradio.py` to build a demo on your PC.
|
| 25 |
|
| 26 |
-
- 2024-07-13: We released a **command-line based demo** of `miniCPM-visual-embedding-v0` for users to retireve most relavant pages from a given PDF file (could be very long), take a look at [pipeline.py](https://huggingface.co/RhapsodyAI/minicpm-visual-embedding-v0/blob/main/pipeline.py).
|
| 27 |
|
| 28 |
- 2024-06-27: π We released our first visual embedding model checkpoint minicpm-visual-embedding-v0 on [huggingface](https://huggingface.co/RhapsodyAI/minicpm-visual-embedding-v0).
|
| 29 |
|
| 30 |
- 2024-05-08: π We [open-sourced](https://github.com/RhapsodyAILab/minicpm-visual-embedding-v0) our training code (full-parameter tuning with GradCache and DeepSpeed, supports large batch size across multiple GPUs with zero-stage1) and eval code.
|
| 31 |
|
| 32 |
-
#
|
| 33 |
|
| 34 |
-
Pip install all dependencies:
|
| 35 |
|
| 36 |
```
|
| 37 |
Pillow==10.1.0
|
|
@@ -43,27 +43,32 @@ sentencepiece==0.1.99
|
|
| 43 |
numpy==1.26.0
|
| 44 |
```
|
| 45 |
|
| 46 |
-
|
|
|
|
|
|
|
| 47 |
|
| 48 |
```bash
|
| 49 |
git lfs install
|
| 50 |
git clone https://huggingface.co/RhapsodyAI/minicpm-visual-embedding-v0
|
| 51 |
```
|
| 52 |
|
| 53 |
-
|
| 54 |
|
| 55 |
```bash
|
|
|
|
| 56 |
huggingface-cli download --resume-download RhapsodyAI/minicpm-visual-embedding-v0 --local-dir minicpm-visual-embedding-v0 --local-dir-use-symlinks False
|
| 57 |
```
|
| 58 |
|
| 59 |
-
|
| 60 |
|
| 61 |
```bash
|
| 62 |
pip install gradio
|
| 63 |
python pipeline_gradio.py
|
| 64 |
```
|
| 65 |
|
| 66 |
-
|
|
|
|
|
|
|
| 67 |
|
| 68 |
```python
|
| 69 |
from transformers import AutoModel
|
|
@@ -105,11 +110,11 @@ print(scores)
|
|
| 105 |
|
| 106 |
# Todos
|
| 107 |
|
| 108 |
-
|
| 109 |
|
| 110 |
-
|
| 111 |
|
| 112 |
-
|
| 113 |
|
| 114 |
# Limitations
|
| 115 |
|
|
|
|
| 19 |
|
| 20 |
# News
|
| 21 |
|
| 22 |
+
- 2024-07-14: We released **online huggingface demo**! Try our [online demo](https://huggingface.co/spaces/bokesyo/minicpm-visual-embeeding-v0-demo)!
|
| 23 |
|
| 24 |
+
- 2024-07-14: We released a **locally deployable Gradio demo** of `miniCPM-visual-embedding-v0`, take a look at [pipeline_gradio.py](https://huggingface.co/RhapsodyAI/minicpm-visual-embedding-v0/blob/main/pipeline_gradio.py). You can run `pipeline_gradio.py` to build a demo on your PC.
|
| 25 |
|
| 26 |
+
- 2024-07-13: We released a **locally deployable command-line based demo** of `miniCPM-visual-embedding-v0` for users to retireve most relavant pages from a given PDF file (could be very long), take a look at [pipeline.py](https://huggingface.co/RhapsodyAI/minicpm-visual-embedding-v0/blob/main/pipeline.py).
|
| 27 |
|
| 28 |
- 2024-06-27: π We released our first visual embedding model checkpoint minicpm-visual-embedding-v0 on [huggingface](https://huggingface.co/RhapsodyAI/minicpm-visual-embedding-v0).
|
| 29 |
|
| 30 |
- 2024-05-08: π We [open-sourced](https://github.com/RhapsodyAILab/minicpm-visual-embedding-v0) our training code (full-parameter tuning with GradCache and DeepSpeed, supports large batch size across multiple GPUs with zero-stage1) and eval code.
|
| 31 |
|
| 32 |
+
# Deploy on your PC
|
| 33 |
|
| 34 |
+
1. Pip install all dependencies:
|
| 35 |
|
| 36 |
```
|
| 37 |
Pillow==10.1.0
|
|
|
|
| 43 |
numpy==1.26.0
|
| 44 |
```
|
| 45 |
|
| 46 |
+
2. Download the model weights and modeling file, choose one of the following:
|
| 47 |
+
|
| 48 |
+
- Download with git clone.
|
| 49 |
|
| 50 |
```bash
|
| 51 |
git lfs install
|
| 52 |
git clone https://huggingface.co/RhapsodyAI/minicpm-visual-embedding-v0
|
| 53 |
```
|
| 54 |
|
| 55 |
+
- Download with huggingface-hub.
|
| 56 |
|
| 57 |
```bash
|
| 58 |
+
pip install huggingface-hub
|
| 59 |
huggingface-cli download --resume-download RhapsodyAI/minicpm-visual-embedding-v0 --local-dir minicpm-visual-embedding-v0 --local-dir-use-symlinks False
|
| 60 |
```
|
| 61 |
|
| 62 |
+
3. To deploy a local demo, first check `pipeline_gradio.py`, change `model_path` to your local path and change `device` to your device (for users with Nvidia card, use `cuda`, for users with apple silicon, use `mps`, for users with only x86 cpu, please use `cpu`). then launch the demo:
|
| 63 |
|
| 64 |
```bash
|
| 65 |
pip install gradio
|
| 66 |
python pipeline_gradio.py
|
| 67 |
```
|
| 68 |
|
| 69 |
+
# For research purpose
|
| 70 |
+
|
| 71 |
+
To run the model for research purpose, please refer the following code:
|
| 72 |
|
| 73 |
```python
|
| 74 |
from transformers import AutoModel
|
|
|
|
| 110 |
|
| 111 |
# Todos
|
| 112 |
|
| 113 |
+
[x] Release huggingface space demo.
|
| 114 |
|
| 115 |
+
[] Release the evaluation results.
|
| 116 |
|
| 117 |
+
[] Release technical report.
|
| 118 |
|
| 119 |
# Limitations
|
| 120 |
|