KETI_b1_s4_e1_training_log.log

03/28/2025 03:26:22 - INFO  - Output Directory: output/gemma-3-27b-pt/20250328_032606_gemma-3-27b-pt_LoRA
03/28/2025 03:26:22 - INFO  - Experiment name: KETI_b1_s4_e1
03/28/2025 03:26:22 - INFO  - Using 6 GPU(s): NVIDIA A100-SXM4-80GB
03/28/2025 03:26:22 - INFO  - torch_dtype: torch.bfloat16
03/28/2025 03:26:22 - INFO  - ✅ FFT or LoRA 모드로 학습합니다.
03/28/2025 03:26:45 - INFO  - Initializing LORA model...
03/28/2025 03:27:05 - INFO  - gcc -pthread -B /root/pai/envs/llm-finetuning/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/pai/envs/llm-finetuning/include -fPIC -O2 -isystem /root/pai/envs/llm-finetuning/include -fPIC -c /tmp/tmpvhsu4dwj/test.c -o /tmp/tmpvhsu4dwj/test.o
03/28/2025 03:27:06 - INFO  - gcc -pthread -B /root/pai/envs/llm-finetuning/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/pai/envs/llm-finetuning/include -fPIC -O2 -isystem /root/pai/envs/llm-finetuning/include -fPIC -c /tmp/tmp_hmdziqb/test.c -o /tmp/tmp_hmdziqb/test.o
03/28/2025 03:27:07 - INFO  - Start Training !
03/28/2025 03:27:39 - INFO  - [Epoch 0.11] [Step 10] loss: 3.5848
03/28/2025 03:28:04 - INFO  - [Epoch 0.22] [Step 20] loss: 3.0443
03/28/2025 03:28:30 - INFO  - [Epoch 0.33] [Step 30] loss: 2.9392
03/28/2025 03:28:55 - INFO  - [Epoch 0.44] [Step 40] loss: 2.9002
03/28/2025 03:29:18 - INFO  - [Epoch 0.55] [Step 50] loss: 2.8820
03/28/2025 03:29:42 - INFO  - [Epoch 0.66] [Step 60] loss: 2.8730
03/28/2025 03:30:06 - INFO  - [Epoch 0.77] [Step 70] loss: 2.8597
03/28/2025 03:30:29 - INFO  - [Epoch 0.88] [Step 80] loss: 2.8434
03/28/2025 03:30:52 - INFO  - [Epoch 0.99] [Step 90] loss: 2.8448
03/28/2025 03:31:18 - INFO  - ✅ Training complete. Logging system usage...
03/28/2025 03:31:18 - INFO  - >> System Usage - CPU: 3.3%, RAM: 3.6%, SSD: 72.60GB / 1888.43GB
03/28/2025 03:31:18 - INFO  - >> GPU 0: 78.23 GB used
03/28/2025 03:31:18 - INFO  - >> GPU 1: 78.33 GB used
03/28/2025 03:31:18 - INFO  - >> GPU 2: 77.85 GB used
03/28/2025 03:31:18 - INFO  - >> GPU 3: 78.40 GB used
03/28/2025 03:31:18 - INFO  - >> GPU 4: 79.06 GB used
03/28/2025 03:31:18 - INFO  - >> GPU 5: 77.98 GB used
03/28/2025 03:31:18 - INFO  - >> Total GPU Memory Used: 469.84 GB
03/28/2025 03:31:18 - INFO  - >> Total GPU Power Consumption: 550.97 W