runtime error

Exit code: 1. Reason: 004.safetensors: 100%|██████████| 1.17G/1.17G [00:04<00:00, 289MB/s] Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s] Loading checkpoint shards: 25%|██▌ | 1/4 [03:16<09:48, 196.12s/it] Loading checkpoint shards: 50%|█████ | 2/4 [07:24<07:34, 227.02s/it] Loading checkpoint shards: 75%|███████▌ | 3/4 [10:38<03:31, 211.93s/it] Loading checkpoint shards: 100%|██████████| 4/4 [10:46<00:00, 131.35s/it] Loading checkpoint shards: 100%|██████████| 4/4 [10:46<00:00, 161.64s/it] generation_config.json: 0%| | 0.00/184 [00:00<?, ?B/s] generation_config.json: 100%|██████████| 184/184 [00:00<00:00, 1.26MB/s] adapter_config.json: 0%| | 0.00/881 [00:00<?, ?B/s] adapter_config.json: 100%|██████████| 881/881 [00:00<00:00, 2.50MB/s] adapter_model.safetensors: 0%| | 0.00/151M [00:00<?, ?B/s] adapter_model.safetensors: 11%|█ | 16.8M/151M [00:01<00:09, 14.5MB/s] adapter_model.safetensors: 100%|██████████| 151M/151M [00:02<00:00, 70.8MB/s] Traceback (most recent call last): File "/app/app.py", line 42, in <module> model = PeftModel.from_pretrained(base_model, model_name) File "/usr/local/lib/python3.10/site-packages/peft/peft_model.py", line 568, in from_pretrained load_result = model.load_adapter( File "/usr/local/lib/python3.10/site-packages/peft/peft_model.py", line 1430, in load_adapter dispatch_model( File "/usr/local/lib/python3.10/site-packages/accelerate/big_modeling.py", line 383, in dispatch_model raise ValueError( ValueError: We need an `offload_dir` to dispatch this model according to this `device_map`, the following submodules need to be offloaded: base_model.model.model.layers.29, base_model.model.model.layers.30, base_model.model.model.layers.31, base_model.model.model.norm, base_model.model.model.rotary_emb, base_model.model.lm_head.

Container logs: