File size: 1,787 Bytes
bcf2c1d 568f9ab d3952e6 568f9ab |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
---
tags:
- code
---
# SWE Bench Verified (Compressed)
<picture>
<img src="./plot.png" alt="SWE-Bench Verified Total Image Size" style="width:100%">
</picture>
Setting up all the SWE-Bench Verified images used to take over 200 GiB of storage and 100+ GiB of transfer.
Now it’s just:
- 31 GiB total storage (down from 206 GiB)
- 5 GiB network transfer (down from 100 GiB)
- ~ 5 minutes setup
## 🚀 Getting the Images
Images follow the naming convention:
```
logicstar/sweb.eval.x86_64.<repo>_1776_<instance>
```
### Docker
```bash
curl -L -# https://huggingface.co/LogicStar/SWE-Bench-Verified-Compressed/resolve/main/saved.tar.zst?download=true | zstd -d --long=31 --stdout | docker load
```
### Podman
⚠️ Podman cannot load docker-archives with manifests larger than 1 MiB.
We split the archive into two parts:
```bash
curl -L -# https://huggingface.co/LogicStar/SWE-Bench-Verified-Compressed/resolve/main/saved.1.tar.zst?download=true | zstd -d --long=31 --stdout | podman load
curl -L -# https://huggingface.co/LogicStar/SWE-Bench-Verified-Compressed/resolve/main/saved.2.tar.zst?download=true | zstd -d --long=31 --stdout | podman load
```
For faster downloads and parallelized loading, use the Hugging Face CLI to download the compressed OCI Layout and our load.py script to load the images in parallel:
```bash
# Clone the repo and cd into it
hf download LogicStar/SWE-Bench-Verified-Compressed layout.tar.zst --local-dir .
zstd -d --long=31 --stdout layout.tar.zst | tar -x -f -
python3 load.py
```
## 🛠 Using the Images
Just pass --namespace logicstar to the SWE-Bench harness. Example:
```bash
python -m swebench.harness.run_evaluation \
--predictions_path gold \
--max_workers 1 \
--run_id validate-gold \
--namespace logicstar
``` |