2025/03/23 17:26:26 - mmengine - DEBUG - An `DeepSpeedStrategy` instance is built from registry, and its implementation can be found in xtuner.engine._strategy.deepspeed 2025/03/23 17:26:27 - mmengine - INFO - ------------------------------------------------------------ System environment: sys.platform: linux Python: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] CUDA available: True MUSA available: False numpy_random_seed: 346959425 GPU 0,1,2,3,4,5,6,7: NVIDIA GeForce RTX 4090 CUDA_HOME: /usr/local/cuda-12.4 NVCC: Cuda compilation tools, release 12.4, V12.4.99 GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 PyTorch: 2.5.1+cu124 PyTorch compiling details: PyTorch built with: - GCC 9.3 - C++ Version: 201703 - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) - OpenMP 201511 (a.k.a. OpenMP 4.5) - LAPACK is enabled (usually provided by MKL) - NNPACK is enabled - CPU capability usage: AVX2 - CUDA Runtime 12.4 - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90 - CuDNN 90.1 - Magma 2.6.1 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, TorchVision: 0.20.1+cu124 OpenCV: 4.9.0 MMEngine: 0.10.7 Runtime environment: launcher: none randomness: {'seed': None, 'deterministic': False} cudnn_benchmark: False mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: None deterministic: False Distributed launcher: none Distributed training: False GPU number: 1 ------------------------------------------------------------ 2025/03/23 17:26:27 - mmengine - INFO - Config: accumulative_counts = 2 batch_size = 1 betas = ( 0.9, 0.999, ) custom_hooks = [ dict( tokenizer=dict( pretrained_model_name_or_path='/data/wangqun/models/InternVL2_5-2B', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.engine.hooks.DatasetInfoHook'), ] data_path = '/home/wangqun/data/layout_ocr_multi.json' dataloader_num_workers = 4 default_hooks = dict( checkpoint=dict( by_epoch=False, interval=1000, max_keep_ckpts=-1, save_optimizer=False, type='mmengine.hooks.CheckpointHook'), logger=dict( interval=10, log_metric_by_epoch=False, type='mmengine.hooks.LoggerHook'), param_scheduler=dict(type='mmengine.hooks.ParamSchedulerHook'), sampler_seed=dict(type='mmengine.hooks.DistSamplerSeedHook'), timer=dict(type='mmengine.hooks.IterTimerHook')) env_cfg = dict( cudnn_benchmark=False, dist_cfg=dict(backend='nccl'), mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) image_folder = '/' launcher = 'none' llava_dataset = dict( data_paths='/home/wangqun/data/layout_ocr_multi.json', image_folders='/', max_length=8192, model_path='/data/wangqun/models/InternVL2_5-2B', template='xtuner.utils.PROMPT_TEMPLATE.internlm2_chat', type='xtuner.dataset.InternVL_V1_5_Dataset') load_from = None log_level = 'DEBUG' log_processor = dict(by_epoch=False) lr = 2e-05 max_epochs = 4 max_length = 8192 max_norm = 1 model = dict( freeze_llm=True, freeze_visual_encoder=True, llm_lora=dict( lora_alpha=256, lora_dropout=0.05, r=128, target_modules=None, task_type='CAUSAL_LM', type='peft.LoraConfig'), model_path='/data/wangqun/models/InternVL2_5-2B', quantization_llm=True, quantization_vit=False, type='xtuner.model.InternVL_V1_5') optim_type = 'torch.optim.AdamW' optim_wrapper = dict( optimizer=dict( betas=( 0.9, 0.999, ), lr=2e-05, type='torch.optim.AdamW', weight_decay=0.05), type='DeepSpeedOptimWrapper') param_scheduler = [ dict( begin=0, by_epoch=True, convert_to_iter_based=True, end=0.12, start_factor=1e-05, type='mmengine.optim.LinearLR'), dict( begin=0.12, by_epoch=True, convert_to_iter_based=True, end=4, eta_min=0.0, type='mmengine.optim.CosineAnnealingLR'), ] path = '/data/wangqun/models/InternVL2_5-2B' prompt_template = 'xtuner.utils.PROMPT_TEMPLATE.internlm2_chat' randomness = dict(deterministic=False, seed=None) resume = False runner_type = 'FlexibleRunner' save_steps = 1000 save_total_limit = -1 strategy = dict( config=dict( bf16=dict(enabled=True), fp16=dict(enabled=False, initial_scale_power=16), gradient_accumulation_steps='auto', gradient_clipping='auto', train_micro_batch_size_per_gpu='auto', zero_allow_untested_optimizer=True, zero_force_ds_cpu_optimizer=False, zero_optimization=dict(overlap_comm=True, stage=2)), exclude_frozen_parameters=True, gradient_accumulation_steps=2, gradient_clipping=1, sequence_parallel_size=1, train_micro_batch_size_per_gpu=1, type='xtuner.engine.DeepSpeedStrategy') tokenizer = dict( pretrained_model_name_or_path='/data/wangqun/models/InternVL2_5-2B', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained') train_cfg = dict(max_epochs=4, type='xtuner.engine.runner.TrainLoop') train_dataloader = dict( batch_size=1, collate_fn=dict(type='xtuner.dataset.collate_fns.default_collate_fn'), dataset=dict( data_paths='/home/wangqun/data/layout_ocr_multi.json', image_folders='/', max_length=8192, model_path='/data/wangqun/models/InternVL2_5-2B', template='xtuner.utils.PROMPT_TEMPLATE.internlm2_chat', type='xtuner.dataset.InternVL_V1_5_Dataset'), num_workers=4, sampler=dict( length_property='modality_length', per_device_batch_size=2, type='xtuner.dataset.samplers.LengthGroupedSampler')) visualizer = dict( type='mmengine.visualization.Visualizer', vis_backends=[ dict(type='mmengine.visualization.TensorboardVisBackend'), ]) warmup_ratio = 0.03 weight_decay = 0.05 work_dir = '/home/wangqun/work_dirs/internvl_ft_run_14_filter' 2025/03/23 17:26:27 - mmengine - DEBUG - An `TensorboardVisBackend` instance is built from registry, and its implementation can be found in mmengine.visualization.vis_backend 2025/03/23 17:26:27 - mmengine - DEBUG - An `Visualizer` instance is built from registry, and its implementation can be found in mmengine.visualization.visualizer 2025/03/23 17:26:27 - mmengine - DEBUG - Attribute `_env_initialized` is not defined in or `._env_initialized is False, `_init_env` will be called and ._env_initialized will be set to True 2025/03/23 17:26:27 - mmengine - DEBUG - Get class `RuntimeInfoHook` from "hook" registry in "mmengine" 2025/03/23 17:26:27 - mmengine - DEBUG - An `RuntimeInfoHook` instance is built from registry, and its implementation can be found in mmengine.hooks.runtime_info_hook 2025/03/23 17:26:27 - mmengine - DEBUG - An `IterTimerHook` instance is built from registry, and its implementation can be found in mmengine.hooks.iter_timer_hook 2025/03/23 17:26:27 - mmengine - DEBUG - An `DistSamplerSeedHook` instance is built from registry, and its implementation can be found in mmengine.hooks.sampler_seed_hook 2025/03/23 17:26:27 - mmengine - DEBUG - An `LoggerHook` instance is built from registry, and its implementation can be found in mmengine.hooks.logger_hook 2025/03/23 17:26:27 - mmengine - DEBUG - An `ParamSchedulerHook` instance is built from registry, and its implementation can be found in mmengine.hooks.param_scheduler_hook 2025/03/23 17:26:27 - mmengine - DEBUG - An `CheckpointHook` instance is built from registry, and its implementation can be found in mmengine.hooks.checkpoint_hook 2025/03/23 17:26:27 - mmengine - WARNING - Failed to search registry with scope "mmengine" in the "builder" registry tree. As a workaround, the current "builder" registry in "xtuner" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmengine" is a correct scope, or whether the registry is initialized. 2025/03/23 17:26:27 - mmengine - DEBUG - An `from_pretrained` instance is built from registry, and its implementation can be found in transformers.models.auto.tokenization_auto 2025/03/23 17:26:27 - mmengine - DEBUG - An `DatasetInfoHook` instance is built from registry, and its implementation can be found in xtuner.engine.hooks.dataset_info_hook 2025/03/23 17:26:27 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook (BELOW_NORMAL) LoggerHook -------------------- before_train: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DatasetInfoHook (VERY_LOW ) CheckpointHook -------------------- before_train_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DistSamplerSeedHook -------------------- before_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook -------------------- after_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_train_epoch: (NORMAL ) IterTimerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- before_val: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) DatasetInfoHook -------------------- before_val_epoch: (NORMAL ) IterTimerHook -------------------- before_val_iter: (NORMAL ) IterTimerHook -------------------- after_val_iter: (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook -------------------- after_val_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_val: (VERY_HIGH ) RuntimeInfoHook -------------------- after_train: (VERY_HIGH ) RuntimeInfoHook (VERY_LOW ) CheckpointHook -------------------- before_test: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) DatasetInfoHook -------------------- before_test_epoch: (NORMAL ) IterTimerHook -------------------- before_test_iter: (NORMAL ) IterTimerHook -------------------- after_test_iter: (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook -------------------- after_test_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook -------------------- after_test: (VERY_HIGH ) RuntimeInfoHook -------------------- after_run: (BELOW_NORMAL) LoggerHook -------------------- 2025/03/23 17:26:27 - mmengine - DEBUG - An `FlexibleRunner` instance is built from registry, its implementation can be found inmmengine.runner._flexible_runner 2025/03/23 17:26:27 - mmengine - INFO - Starting to loading data and calc length 2025/03/23 17:26:27 - mmengine - INFO - =======Starting to process /home/wangqun/data/layout_ocr_multi.json ======= 2025/03/23 17:26:34 - mmengine - INFO - =======total 4794 samples of /home/wangqun/data/layout_ocr_multi.json======= 2025/03/23 17:26:34 - mmengine - INFO - end loading data and calc length 2025/03/23 17:26:34 - mmengine - INFO - =======total 4794 samples======= 2025/03/23 17:26:34 - mmengine - DEBUG - An `InternVL_V1_5_Dataset` instance is built from registry, and its implementation can be found in xtuner.dataset.internvl_dataset 2025/03/23 17:26:34 - mmengine - INFO - LengthGroupedSampler is used. 2025/03/23 17:26:34 - mmengine - INFO - LengthGroupedSampler construction is complete, and the selected attribute is modality_length 2025/03/23 17:26:34 - mmengine - DEBUG - An `LengthGroupedSampler` instance is built from registry, and its implementation can be found in xtuner.dataset.samplers.length_grouped 2025/03/23 17:26:34 - mmengine - WARNING - Dataset InternVL_V1_5_Dataset has no metainfo. ``dataset_meta`` in visualizer will be None. 2025/03/23 17:26:34 - mmengine - DEBUG - An `TrainLoop` instance is built from registry, and its implementation can be found in xtuner.engine.runner.loops 2025/03/23 17:26:34 - mmengine - INFO - Start to load InternVL_V1_5 model. 2025/03/23 17:26:34 - mmengine - DEBUG - Get class `BaseDataPreprocessor` from "model" registry in "mmengine" 2025/03/23 17:26:34 - mmengine - DEBUG - An `BaseDataPreprocessor` instance is built from registry, and its implementation can be found in mmengine.model.base_model.data_preprocessor 2025/03/23 17:26:36 - mmengine - DEBUG - An `LoraConfig` instance is built from registry, and its implementation can be found in peft.tuners.lora.config 2025/03/23 17:26:37 - mmengine - INFO - InternVL_V1_5( (data_preprocessor): BaseDataPreprocessor() (model): InternVLChatModel( (vision_model): InternVisionModel( (embeddings): InternVisionEmbeddings( (patch_embedding): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14)) ) (encoder): InternVisionEncoder( (layers): ModuleList( (0-23): 24 x InternVisionEncoderLayer( (attn): InternAttention( (qkv): Linear(in_features=1024, out_features=3072, bias=True) (attn_drop): Dropout(p=0.0, inplace=False) (proj_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=1024, out_features=1024, bias=True) ) (mlp): InternMLP( (act): GELUActivation() (fc1): Linear(in_features=1024, out_features=4096, bias=True) (fc2): Linear(in_features=4096, out_features=1024, bias=True) ) (norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True) (norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True) (drop_path1): Identity() (drop_path2): Identity() ) ) ) ) (language_model): PeftModelForCausalLM( (base_model): LoraModel( (model): InternLM2ForCausalLM( (model): InternLM2Model( (tok_embeddings): Embedding(92553, 2048, padding_idx=2) (layers): ModuleList( (0-23): 24 x InternLM2DecoderLayer( (attention): InternLM2Attention( (wqkv): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=4096, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=4096, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (wo): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (rotary_emb): InternLM2DynamicNTKScalingRotaryEmbedding() ) (feed_forward): InternLM2MLP( (w1): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=8192, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=8192, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (w3): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=8192, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=8192, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (w2): lora.Linear( (base_layer): Linear4bit(in_features=8192, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=8192, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) (act_fn): SiLU() ) (attention_norm): InternLM2RMSNorm() (ffn_norm): InternLM2RMSNorm() ) ) (norm): InternLM2RMSNorm() ) (output): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=92553, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=92553, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() (lora_magnitude_vector): ModuleDict() ) ) ) ) (mlp1): Sequential( (0): LayerNorm((4096,), eps=1e-05, elementwise_affine=True) (1): Linear(in_features=4096, out_features=2048, bias=True) (2): GELU(approximate='none') (3): Linear(in_features=2048, out_features=2048, bias=True) ) ) ) 2025/03/23 17:26:37 - mmengine - INFO - InternVL_V1_5 construction is complete 2025/03/23 17:26:37 - mmengine - DEBUG - An `InternVL_V1_5` instance is built from registry, and its implementation can be found in xtuner.model.internvl 2025/03/23 17:26:37 - mmengine - DEBUG - Get class `DefaultOptimWrapperConstructor` from "optimizer wrapper constructor" registry in "mmengine" 2025/03/23 17:26:37 - mmengine - DEBUG - An `DefaultOptimWrapperConstructor` instance is built from registry, and its implementation can be found in mmengine.optim.optimizer.default_constructor 2025/03/23 17:26:37 - mmengine - DEBUG - An `AdamW` instance is built from registry, and its implementation can be found in torch.optim.adamw 2025/03/23 17:26:37 - mmengine - DEBUG - Get class `DeepSpeedOptimWrapper` from "optim_wrapper" registry in "mmengine" 2025/03/23 17:26:37 - mmengine - DEBUG - An `DeepSpeedOptimWrapper` instance is built from registry, and its implementation can be found in mmengine._strategy.deepspeed 2025/03/23 17:26:38 - mmengine - DEBUG - The `end` of is not set. Use the max epochs/iters of train loop as default. 2025/03/23 17:26:38 - mmengine - DEBUG - The `end` of is not set. Use the max epochs/iters of train loop as default. 2025/03/23 17:26:38 - mmengine - INFO - Num train samples 4794 2025/03/23 17:26:38 - mmengine - INFO - train example: 2025/03/23 17:26:39 - mmengine - INFO - <|im_start|> system You are an AI assistant whose name is InternLM (书生·浦语).<|im_end|><|im_start|>user 请从这张聊天截图中提取结构化信息<|im_end|><|im_start|> assistant { "dialog_name": "<对方正在输入...", "conversation": [ { "timestamp": "", "speaker": "avator_0", "content": "不是", "message_bbox": { "min_x": 917, "max_x": 989, "min_y": 253, "max_y": 289 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "avator_0", "content": "在淘宝里", "message_bbox": { "min_x": 839, "max_x": 987, "min_y": 370, "max_y": 404 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "avator_0", "content": "不能发微信", "message_bbox": { "min_x": 801, "max_x": 989, "min_y": 485, "max_y": 521 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "avator_0", "content": "两字", "message_bbox": { "min_x": 915, "max_x": 988, "min_y": 601, "max_y": 637 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "avator_0", "content": "微信", "message_bbox": { "min_x": 916, "max_x": 990, "min_y": 718, "max_y": 753 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "avator_0", "content": "微信", "message_bbox": { "min_x": 845, "max_x": 988, "min_y": 833, "max_y": 869 }, "image": "", "transfer": [], "file": [] } ] }<|im_end|> 2025/03/23 17:26:39 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io 2025/03/23 17:26:39 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future. 2025/03/23 17:26:39 - mmengine - INFO - Checkpoints will be saved to /home/wangqun/work_dirs/internvl_ft_run_14_filter. 2025/03/23 17:27:00 - mmengine - INFO - Iter(train) [ 10/19176] lr: 3.1379e-07 eta: 11:11:42 time: 2.1028 data_time: 0.0148 memory: 18049 loss: 0.4455 2025/03/23 17:27:17 - mmengine - INFO - Iter(train) [ 20/19176] lr: 6.6221e-07 eta: 10:10:12 time: 1.7198 data_time: 0.0165 memory: 11847 loss: 0.5091 2025/03/23 17:27:33 - mmengine - INFO - Iter(train) [ 30/19176] lr: 1.0106e-06 eta: 9:44:05 time: 1.6688 data_time: 0.0156 memory: 11636 loss: 0.5198 2025/03/23 17:27:50 - mmengine - INFO - Iter(train) [ 40/19176] lr: 1.3591e-06 eta: 9:26:28 time: 1.6133 data_time: 0.0158 memory: 11436 loss: 0.5484 2025/03/23 17:28:05 - mmengine - INFO - Iter(train) [ 50/19176] lr: 1.7075e-06 eta: 9:14:04 time: 1.5861 data_time: 0.0157 memory: 11347 loss: 0.5906 2025/03/23 17:28:20 - mmengine - INFO - Iter(train) [ 60/19176] lr: 2.0559e-06 eta: 9:01:16 time: 1.5026 data_time: 0.0148 memory: 11257 loss: 0.6143 2025/03/23 17:28:35 - mmengine - INFO - Iter(train) [ 70/19176] lr: 2.4044e-06 eta: 8:48:11 time: 1.4175 data_time: 0.0141 memory: 11015 loss: 0.6713 2025/03/23 17:28:47 - mmengine - INFO - Iter(train) [ 80/19176] lr: 2.7528e-06 eta: 8:29:28 time: 1.1954 data_time: 0.0137 memory: 10728 loss: 0.6002 2025/03/23 17:28:57 - mmengine - INFO - Iter(train) [ 90/19176] lr: 3.1012e-06 eta: 8:09:07 time: 1.0327 data_time: 0.0132 memory: 10132 loss: 0.4988 2025/03/23 17:29:05 - mmengine - INFO - Iter(train) [ 100/19176] lr: 3.4496e-06 eta: 7:45:58 time: 0.8173 data_time: 0.0123 memory: 9982 loss: 0.5125 2025/03/23 17:29:24 - mmengine - INFO - Iter(train) [ 110/19176] lr: 3.7981e-06 eta: 7:58:27 time: 1.9066 data_time: 0.0144 memory: 13146 loss: 0.3397 2025/03/23 17:29:42 - mmengine - INFO - Iter(train) [ 120/19176] lr: 4.1465e-06 eta: 8:04:26 time: 1.7411 data_time: 0.0148 memory: 11958 loss: 0.3173 2025/03/23 17:29:58 - mmengine - INFO - Iter(train) [ 130/19176] lr: 4.4949e-06 eta: 8:07:31 time: 1.6617 data_time: 0.0146 memory: 11765 loss: 0.3304 2025/03/23 17:30:14 - mmengine - INFO - Iter(train) [ 140/19176] lr: 4.8434e-06 eta: 8:08:42 time: 1.5997 data_time: 0.0144 memory: 11375 loss: 0.3162 2025/03/23 17:30:30 - mmengine - INFO - Iter(train) [ 150/19176] lr: 5.1918e-06 eta: 8:08:54 time: 1.5622 data_time: 0.0145 memory: 11337 loss: 0.3057 2025/03/23 17:30:45 - mmengine - INFO - Iter(train) [ 160/19176] lr: 5.5402e-06 eta: 8:07:42 time: 1.4935 data_time: 0.0143 memory: 11146 loss: 0.3779 2025/03/23 17:30:59 - mmengine - INFO - Iter(train) [ 170/19176] lr: 5.8886e-06 eta: 8:05:16 time: 1.4225 data_time: 0.0144 memory: 10991 loss: 0.3915 2025/03/23 17:31:12 - mmengine - INFO - Iter(train) [ 180/19176] lr: 6.2371e-06 eta: 8:01:16 time: 1.3193 data_time: 0.0140 memory: 10821 loss: 0.3762 2025/03/23 17:31:23 - mmengine - INFO - Iter(train) [ 190/19176] lr: 6.5855e-06 eta: 7:54:24 time: 1.1231 data_time: 0.0128 memory: 10431 loss: 0.3685 2025/03/23 17:31:33 - mmengine - INFO - Iter(train) [ 200/19176] lr: 6.9339e-06 eta: 7:44:50 time: 0.9100 data_time: 0.0123 memory: 10047 loss: 0.3958 2025/03/23 17:31:53 - mmengine - INFO - Iter(train) [ 210/19176] lr: 7.2824e-06 eta: 7:53:17 time: 2.0470 data_time: 0.0143 memory: 13932 loss: 0.3002 2025/03/23 17:32:10 - mmengine - INFO - Iter(train) [ 220/19176] lr: 7.6308e-06 eta: 7:56:28 time: 1.7362 data_time: 0.0144 memory: 11949 loss: 0.2741 2025/03/23 17:32:27 - mmengine - INFO - Iter(train) [ 230/19176] lr: 7.9792e-06 eta: 7:58:07 time: 1.6468 data_time: 0.0147 memory: 11568 loss: 0.2694 2025/03/23 17:32:43 - mmengine - INFO - Iter(train) [ 240/19176] lr: 8.3276e-06 eta: 7:59:17 time: 1.6217 data_time: 0.0142 memory: 11425 loss: 0.2683 2025/03/23 17:32:59 - mmengine - INFO - Iter(train) [ 250/19176] lr: 8.6761e-06 eta: 7:59:34 time: 1.5614 data_time: 0.0143 memory: 11277 loss: 0.3413 2025/03/23 17:33:14 - mmengine - INFO - Iter(train) [ 260/19176] lr: 9.0245e-06 eta: 7:59:02 time: 1.4971 data_time: 0.0148 memory: 11133 loss: 0.3109 2025/03/23 17:33:28 - mmengine - INFO - Iter(train) [ 270/19176] lr: 9.3729e-06 eta: 7:58:05 time: 1.4606 data_time: 0.0144 memory: 11136 loss: 0.3284 2025/03/23 17:33:42 - mmengine - INFO - Iter(train) [ 280/19176] lr: 9.7214e-06 eta: 7:56:08 time: 1.3655 data_time: 0.0146 memory: 10891 loss: 0.4090 2025/03/23 17:33:52 - mmengine - INFO - Iter(train) [ 290/19176] lr: 1.0070e-05 eta: 7:50:53 time: 1.0521 data_time: 0.0125 memory: 10254 loss: 0.3708 2025/03/23 17:33:59 - mmengine - INFO - Iter(train) [ 300/19176] lr: 1.0418e-05 eta: 7:41:25 time: 0.6169 data_time: 0.0108 memory: 9586 loss: 0.3747 2025/03/23 17:34:17 - mmengine - INFO - Iter(train) [ 310/19176] lr: 1.0767e-05 eta: 7:45:05 time: 1.8527 data_time: 0.0139 memory: 13092 loss: 0.2860 2025/03/23 17:34:34 - mmengine - INFO - Iter(train) [ 320/19176] lr: 1.1115e-05 eta: 7:46:52 time: 1.6862 data_time: 0.0148 memory: 11753 loss: 0.2605 2025/03/23 17:34:50 - mmengine - INFO - Iter(train) [ 330/19176] lr: 1.1463e-05 eta: 7:48:08 time: 1.6449 data_time: 0.0147 memory: 11608 loss: 0.2517 2025/03/23 17:35:06 - mmengine - INFO - Iter(train) [ 340/19176] lr: 1.1812e-05 eta: 7:48:52 time: 1.5954 data_time: 0.0150 memory: 11347 loss: 0.2566 2025/03/23 17:35:22 - mmengine - INFO - Iter(train) [ 350/19176] lr: 1.2160e-05 eta: 7:49:08 time: 1.5513 data_time: 0.0146 memory: 11277 loss: 0.2947 2025/03/23 17:35:37 - mmengine - INFO - Iter(train) [ 360/19176] lr: 1.2509e-05 eta: 7:48:44 time: 1.4785 data_time: 0.0147 memory: 11137 loss: 0.2906 2025/03/23 17:35:51 - mmengine - INFO - Iter(train) [ 370/19176] lr: 1.2857e-05 eta: 7:47:57 time: 1.4317 data_time: 0.0150 memory: 11021 loss: 0.3224 2025/03/23 17:36:04 - mmengine - INFO - Iter(train) [ 380/19176] lr: 1.3206e-05 eta: 7:45:47 time: 1.2603 data_time: 0.0146 memory: 10788 loss: 0.3348 2025/03/23 17:36:14 - mmengine - INFO - Iter(train) [ 390/19176] lr: 1.3554e-05 eta: 7:41:40 time: 1.0045 data_time: 0.0126 memory: 10249 loss: 0.2911 2025/03/23 17:36:19 - mmengine - INFO - Iter(train) [ 400/19176] lr: 1.3902e-05 eta: 7:34:21 time: 0.5710 data_time: 0.0108 memory: 9347 loss: 0.3113 2025/03/23 17:36:39 - mmengine - INFO - Iter(train) [ 410/19176] lr: 1.4251e-05 eta: 7:38:20 time: 2.0058 data_time: 0.0143 memory: 13843 loss: 0.2411 2025/03/23 17:36:57 - mmengine - INFO - Iter(train) [ 420/19176] lr: 1.4599e-05 eta: 7:40:16 time: 1.7579 data_time: 0.0149 memory: 11942 loss: 0.2354 2025/03/23 17:37:14 - mmengine - INFO - Iter(train) [ 430/19176] lr: 1.4948e-05 eta: 7:41:48 time: 1.7172 data_time: 0.0153 memory: 11794 loss: 0.2294 2025/03/23 17:37:31 - mmengine - INFO - Iter(train) [ 440/19176] lr: 1.5296e-05 eta: 7:42:56 time: 1.6735 data_time: 0.0151 memory: 11524 loss: 0.2491 2025/03/23 17:37:47 - mmengine - INFO - Iter(train) [ 450/19176] lr: 1.5645e-05 eta: 7:43:25 time: 1.5879 data_time: 0.0150 memory: 11401 loss: 0.2577 2025/03/23 17:38:02 - mmengine - INFO - Iter(train) [ 460/19176] lr: 1.5993e-05 eta: 7:43:13 time: 1.4911 data_time: 0.0151 memory: 11152 loss: 0.2931 2025/03/23 17:38:15 - mmengine - INFO - Iter(train) [ 470/19176] lr: 1.6341e-05 eta: 7:42:06 time: 1.3528 data_time: 0.0157 memory: 10855 loss: 0.6254 2025/03/23 17:38:26 - mmengine - INFO - Iter(train) [ 480/19176] lr: 1.6690e-05 eta: 7:39:32 time: 1.1258 data_time: 0.0134 memory: 10574 loss: 0.3271 2025/03/23 17:38:35 - mmengine - INFO - Iter(train) [ 490/19176] lr: 1.7038e-05 eta: 7:35:38 time: 0.8994 data_time: 0.0123 memory: 9989 loss: 0.3018 2025/03/23 17:38:42 - mmengine - INFO - Iter(train) [ 500/19176] lr: 1.7387e-05 eta: 7:30:10 time: 0.6233 data_time: 0.0114 memory: 9142 loss: 0.2995 2025/03/23 17:39:02 - mmengine - INFO - Iter(train) [ 510/19176] lr: 1.7735e-05 eta: 7:33:30 time: 2.0329 data_time: 0.0139 memory: 15882 loss: 0.2967 2025/03/23 17:39:20 - mmengine - INFO - Iter(train) [ 520/19176] lr: 1.8084e-05 eta: 7:35:23 time: 1.8152 data_time: 0.0146 memory: 12284 loss: 0.2300 2025/03/23 17:39:37 - mmengine - INFO - Iter(train) [ 530/19176] lr: 1.8432e-05 eta: 7:36:36 time: 1.7117 data_time: 0.0153 memory: 11794 loss: 0.2415 2025/03/23 17:39:54 - mmengine - INFO - Iter(train) [ 540/19176] lr: 1.8780e-05 eta: 7:37:25 time: 1.6559 data_time: 0.0153 memory: 11695 loss: 0.2374 2025/03/23 17:40:10 - mmengine - INFO - Iter(train) [ 550/19176] lr: 1.9129e-05 eta: 7:37:51 time: 1.5925 data_time: 0.0154 memory: 11355 loss: 0.2777 2025/03/23 17:40:25 - mmengine - INFO - Iter(train) [ 560/19176] lr: 1.9477e-05 eta: 7:37:54 time: 1.5278 data_time: 0.0145 memory: 11204 loss: 0.2843 2025/03/23 17:40:39 - mmengine - INFO - Iter(train) [ 570/19176] lr: 1.9826e-05 eta: 7:37:16 time: 1.4048 data_time: 0.0145 memory: 10995 loss: 0.3306 2025/03/23 17:40:51 - mmengine - INFO - Iter(train) [ 580/19176] lr: 2.0000e-05 eta: 7:35:38 time: 1.2134 data_time: 0.0137 memory: 10638 loss: 0.2871 2025/03/23 17:41:02 - mmengine - INFO - Iter(train) [ 590/19176] lr: 2.0000e-05 eta: 7:33:07 time: 1.0375 data_time: 0.0127 memory: 10217 loss: 0.2966 2025/03/23 17:41:10 - mmengine - INFO - Iter(train) [ 600/19176] lr: 2.0000e-05 eta: 7:29:34 time: 0.8221 data_time: 0.0125 memory: 9724 loss: 0.2867 2025/03/23 17:41:31 - mmengine - INFO - Iter(train) [ 610/19176] lr: 2.0000e-05 eta: 7:32:40 time: 2.1131 data_time: 0.0141 memory: 18682 loss: 0.2123 2025/03/23 17:41:48 - mmengine - INFO - Iter(train) [ 620/19176] lr: 2.0000e-05 eta: 7:33:40 time: 1.7104 data_time: 0.0143 memory: 11803 loss: 0.3217 2025/03/23 17:42:05 - mmengine - INFO - Iter(train) [ 630/19176] lr: 2.0000e-05 eta: 7:34:28 time: 1.6794 data_time: 0.0143 memory: 11645 loss: 0.2600 2025/03/23 17:42:21 - mmengine - INFO - Iter(train) [ 640/19176] lr: 1.9999e-05 eta: 7:34:52 time: 1.6055 data_time: 0.0154 memory: 11435 loss: 0.2610 2025/03/23 17:42:36 - mmengine - INFO - Iter(train) [ 650/19176] lr: 1.9999e-05 eta: 7:35:00 time: 1.5531 data_time: 0.0146 memory: 11260 loss: 0.2936 2025/03/23 17:42:51 - mmengine - INFO - Iter(train) [ 660/19176] lr: 1.9999e-05 eta: 7:34:50 time: 1.4891 data_time: 0.0149 memory: 11163 loss: 0.2975 2025/03/23 17:43:06 - mmengine - INFO - Iter(train) [ 670/19176] lr: 1.9999e-05 eta: 7:34:29 time: 1.4503 data_time: 0.0147 memory: 11101 loss: 0.3365 2025/03/23 17:43:19 - mmengine - INFO - Iter(train) [ 680/19176] lr: 1.9998e-05 eta: 7:33:32 time: 1.3206 data_time: 0.0145 memory: 10808 loss: 0.4375 2025/03/23 17:43:31 - mmengine - INFO - Iter(train) [ 690/19176] lr: 1.9998e-05 eta: 7:31:53 time: 1.1557 data_time: 0.0142 memory: 10434 loss: 0.3143 2025/03/23 17:43:38 - mmengine - INFO - Iter(train) [ 700/19176] lr: 1.9998e-05 eta: 7:28:24 time: 0.7281 data_time: 0.0118 memory: 10294 loss: 0.2746 2025/03/23 17:43:56 - mmengine - INFO - Iter(train) [ 710/19176] lr: 1.9997e-05 eta: 7:29:47 time: 1.8332 data_time: 0.0141 memory: 13058 loss: 0.2253 2025/03/23 17:44:13 - mmengine - INFO - Iter(train) [ 720/19176] lr: 1.9997e-05 eta: 7:30:33 time: 1.6962 data_time: 0.0151 memory: 11806 loss: 0.2433 2025/03/23 17:44:30 - mmengine - INFO - Iter(train) [ 730/19176] lr: 1.9997e-05 eta: 7:31:02 time: 1.6403 data_time: 0.0145 memory: 11471 loss: 0.2261 2025/03/23 17:44:45 - mmengine - INFO - Iter(train) [ 740/19176] lr: 1.9996e-05 eta: 7:31:17 time: 1.5844 data_time: 0.0145 memory: 11482 loss: 0.2169 2025/03/23 17:45:01 - mmengine - INFO - Iter(train) [ 750/19176] lr: 1.9996e-05 eta: 7:31:25 time: 1.5608 data_time: 0.0143 memory: 11352 loss: 0.2845 2025/03/23 17:45:16 - mmengine - INFO - Iter(train) [ 760/19176] lr: 1.9995e-05 eta: 7:31:17 time: 1.4978 data_time: 0.0150 memory: 11192 loss: 0.3136 2025/03/23 17:45:30 - mmengine - INFO - Iter(train) [ 770/19176] lr: 1.9995e-05 eta: 7:30:44 time: 1.3928 data_time: 0.0142 memory: 10917 loss: 0.3118 2025/03/23 17:45:42 - mmengine - INFO - Iter(train) [ 780/19176] lr: 1.9994e-05 eta: 7:29:20 time: 1.1766 data_time: 0.0128 memory: 10609 loss: 0.2793 2025/03/23 17:45:52 - mmengine - INFO - Iter(train) [ 790/19176] lr: 1.9993e-05 eta: 7:27:31 time: 1.0588 data_time: 0.0122 memory: 10224 loss: 0.2468 2025/03/23 17:46:00 - mmengine - INFO - Iter(train) [ 800/19176] lr: 1.9993e-05 eta: 7:24:46 time: 0.8077 data_time: 0.0113 memory: 9991 loss: 0.2583 2025/03/23 17:46:21 - mmengine - INFO - Iter(train) [ 810/19176] lr: 1.9992e-05 eta: 7:26:47 time: 2.0485 data_time: 0.0142 memory: 18041 loss: 0.2401 2025/03/23 17:46:38 - mmengine - INFO - Iter(train) [ 820/19176] lr: 1.9992e-05 eta: 7:27:28 time: 1.7114 data_time: 0.0138 memory: 11875 loss: 0.2507 2025/03/23 17:46:55 - mmengine - INFO - Iter(train) [ 830/19176] lr: 1.9991e-05 eta: 7:27:56 time: 1.6552 data_time: 0.0141 memory: 11506 loss: 0.2577 2025/03/23 17:47:11 - mmengine - INFO - Iter(train) [ 840/19176] lr: 1.9990e-05 eta: 7:28:13 time: 1.6073 data_time: 0.0142 memory: 11420 loss: 0.2300 2025/03/23 17:47:26 - mmengine - INFO - Iter(train) [ 850/19176] lr: 1.9989e-05 eta: 7:28:17 time: 1.5552 data_time: 0.0143 memory: 11304 loss: 0.2143 2025/03/23 17:47:41 - mmengine - INFO - Iter(train) [ 860/19176] lr: 1.9988e-05 eta: 7:28:09 time: 1.5003 data_time: 0.0142 memory: 11218 loss: 0.3643 2025/03/23 17:47:55 - mmengine - INFO - Iter(train) [ 870/19176] lr: 1.9988e-05 eta: 7:27:47 time: 1.4308 data_time: 0.0139 memory: 11028 loss: 0.2697 2025/03/23 17:48:09 - mmengine - INFO - Iter(train) [ 880/19176] lr: 1.9987e-05 eta: 7:27:00 time: 1.3124 data_time: 0.0139 memory: 10833 loss: 0.3730 2025/03/23 17:48:19 - mmengine - INFO - Iter(train) [ 890/19176] lr: 1.9986e-05 eta: 7:25:25 time: 1.0736 data_time: 0.0127 memory: 10367 loss: 0.2579 2025/03/23 17:48:28 - mmengine - INFO - Iter(train) [ 900/19176] lr: 1.9985e-05 eta: 7:23:10 time: 0.8691 data_time: 0.0122 memory: 9979 loss: 0.2640 2025/03/23 17:48:48 - mmengine - INFO - Iter(train) [ 910/19176] lr: 1.9984e-05 eta: 7:24:41 time: 1.9831 data_time: 0.0145 memory: 15455 loss: 0.2264 2025/03/23 17:49:05 - mmengine - INFO - Iter(train) [ 920/19176] lr: 1.9983e-05 eta: 7:25:14 time: 1.6970 data_time: 0.0148 memory: 11740 loss: 0.2537 2025/03/23 17:49:21 - mmengine - INFO - Iter(train) [ 930/19176] lr: 1.9982e-05 eta: 7:25:38 time: 1.6649 data_time: 0.0149 memory: 11539 loss: 0.2056 2025/03/23 17:49:37 - mmengine - INFO - Iter(train) [ 940/19176] lr: 1.9981e-05 eta: 7:25:48 time: 1.5923 data_time: 0.0154 memory: 11384 loss: 0.2133 2025/03/23 17:49:53 - mmengine - INFO - Iter(train) [ 950/19176] lr: 1.9980e-05 eta: 7:25:44 time: 1.5218 data_time: 0.0144 memory: 11239 loss: 0.2538 2025/03/23 17:50:07 - mmengine - INFO - Iter(train) [ 960/19176] lr: 1.9979e-05 eta: 7:25:27 time: 1.4525 data_time: 0.0145 memory: 11114 loss: 0.2533 2025/03/23 17:50:20 - mmengine - INFO - Iter(train) [ 970/19176] lr: 1.9978e-05 eta: 7:24:43 time: 1.3107 data_time: 0.0141 memory: 10815 loss: 0.3180 2025/03/23 17:50:32 - mmengine - INFO - Iter(train) [ 980/19176] lr: 1.9977e-05 eta: 7:23:29 time: 1.1484 data_time: 0.0133 memory: 10478 loss: 0.2918 2025/03/23 17:50:42 - mmengine - INFO - Iter(train) [ 990/19176] lr: 1.9976e-05 eta: 7:21:53 time: 1.0202 data_time: 0.0132 memory: 10229 loss: 0.2564 2025/03/23 17:50:49 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 17:50:49 - mmengine - INFO - Iter(train) [ 1000/19176] lr: 1.9974e-05 eta: 7:19:23 time: 0.7108 data_time: 0.0114 memory: 9630 loss: 0.3066 2025/03/23 17:50:49 - mmengine - INFO - Saving checkpoint at 1000 iterations 2025/03/23 17:51:10 - mmengine - INFO - Iter(train) [ 1010/19176] lr: 1.9973e-05 eta: 7:21:04 time: 2.0912 data_time: 0.0963 memory: 15218 loss: 0.2329 2025/03/23 17:51:27 - mmengine - INFO - Iter(train) [ 1020/19176] lr: 1.9972e-05 eta: 7:21:33 time: 1.7045 data_time: 0.0150 memory: 11942 loss: 0.2476 2025/03/23 17:51:43 - mmengine - INFO - Iter(train) [ 1030/19176] lr: 1.9971e-05 eta: 7:21:50 time: 1.6357 data_time: 0.0146 memory: 11481 loss: 0.2792 2025/03/23 17:51:59 - mmengine - INFO - Iter(train) [ 1040/19176] lr: 1.9969e-05 eta: 7:21:56 time: 1.5795 data_time: 0.0150 memory: 11355 loss: 0.2327 2025/03/23 17:52:14 - mmengine - INFO - Iter(train) [ 1050/19176] lr: 1.9968e-05 eta: 7:21:53 time: 1.5324 data_time: 0.0145 memory: 11234 loss: 0.2398 2025/03/23 17:52:29 - mmengine - INFO - Iter(train) [ 1060/19176] lr: 1.9967e-05 eta: 7:21:34 time: 1.4334 data_time: 0.0147 memory: 11088 loss: 0.2607 2025/03/23 17:52:42 - mmengine - INFO - Iter(train) [ 1070/19176] lr: 1.9965e-05 eta: 7:20:54 time: 1.3158 data_time: 0.0139 memory: 10917 loss: 0.2984 2025/03/23 17:52:53 - mmengine - INFO - Iter(train) [ 1080/19176] lr: 1.9964e-05 eta: 7:19:48 time: 1.1513 data_time: 0.0128 memory: 10518 loss: 0.2940 2025/03/23 17:53:03 - mmengine - INFO - Iter(train) [ 1090/19176] lr: 1.9962e-05 eta: 7:18:16 time: 0.9934 data_time: 0.0121 memory: 10101 loss: 0.3552 2025/03/23 17:53:11 - mmengine - INFO - Iter(train) [ 1100/19176] lr: 1.9961e-05 eta: 7:16:10 time: 0.7772 data_time: 0.0125 memory: 9869 loss: 0.3410 2025/03/23 17:53:31 - mmengine - INFO - Iter(train) [ 1110/19176] lr: 1.9959e-05 eta: 7:17:15 time: 1.9348 data_time: 0.0144 memory: 13186 loss: 0.2733 2025/03/23 17:53:48 - mmengine - INFO - Iter(train) [ 1120/19176] lr: 1.9958e-05 eta: 7:17:48 time: 1.7487 data_time: 0.0144 memory: 11937 loss: 0.2070 2025/03/23 17:54:05 - mmengine - INFO - Iter(train) [ 1130/19176] lr: 1.9956e-05 eta: 7:18:08 time: 1.6684 data_time: 0.0146 memory: 11631 loss: 0.2622 2025/03/23 17:54:21 - mmengine - INFO - Iter(train) [ 1140/19176] lr: 1.9955e-05 eta: 7:18:20 time: 1.6227 data_time: 0.0145 memory: 11408 loss: 0.2499 2025/03/23 17:54:36 - mmengine - INFO - Iter(train) [ 1150/19176] lr: 1.9953e-05 eta: 7:18:20 time: 1.5524 data_time: 0.0143 memory: 11287 loss: 0.2663 2025/03/23 17:54:51 - mmengine - INFO - Iter(train) [ 1160/19176] lr: 1.9951e-05 eta: 7:18:12 time: 1.5023 data_time: 0.0146 memory: 11127 loss: 0.2508 2025/03/23 17:55:06 - mmengine - INFO - Iter(train) [ 1170/19176] lr: 1.9950e-05 eta: 7:17:59 time: 1.4686 data_time: 0.0142 memory: 11040 loss: 0.3209 2025/03/23 17:55:20 - mmengine - INFO - Iter(train) [ 1180/19176] lr: 1.9948e-05 eta: 7:17:30 time: 1.3648 data_time: 0.0137 memory: 10950 loss: 0.2595 2025/03/23 17:55:31 - mmengine - INFO - Iter(train) [ 1190/19176] lr: 1.9946e-05 eta: 7:16:24 time: 1.1213 data_time: 0.0125 memory: 10506 loss: 0.3508 2025/03/23 17:55:39 - mmengine - INFO - Iter(train) [ 1200/19176] lr: 1.9945e-05 eta: 7:14:27 time: 0.7725 data_time: 0.0113 memory: 9925 loss: 0.2854 2025/03/23 17:55:58 - mmengine - INFO - Iter(train) [ 1210/19176] lr: 1.9943e-05 eta: 7:15:30 time: 1.9679 data_time: 0.0135 memory: 15756 loss: 0.2232 2025/03/23 17:56:16 - mmengine - INFO - Iter(train) [ 1220/19176] lr: 1.9941e-05 eta: 7:16:00 time: 1.7582 data_time: 0.0147 memory: 12009 loss: 0.1975 2025/03/23 17:56:33 - mmengine - INFO - Iter(train) [ 1230/19176] lr: 1.9939e-05 eta: 7:16:20 time: 1.6962 data_time: 0.0141 memory: 11664 loss: 0.2038 2025/03/23 17:56:50 - mmengine - INFO - Iter(train) [ 1240/19176] lr: 1.9937e-05 eta: 7:16:35 time: 1.6633 data_time: 0.0144 memory: 11582 loss: 0.2101 2025/03/23 17:57:06 - mmengine - INFO - Iter(train) [ 1250/19176] lr: 1.9935e-05 eta: 7:16:42 time: 1.6138 data_time: 0.0143 memory: 11381 loss: 0.2669 2025/03/23 17:57:21 - mmengine - INFO - Iter(train) [ 1260/19176] lr: 1.9933e-05 eta: 7:16:37 time: 1.5287 data_time: 0.0142 memory: 11259 loss: 0.2793 2025/03/23 17:57:36 - mmengine - INFO - Iter(train) [ 1270/19176] lr: 1.9931e-05 eta: 7:16:25 time: 1.4783 data_time: 0.0156 memory: 11076 loss: 0.2476 2025/03/23 17:57:49 - mmengine - INFO - Iter(train) [ 1280/19176] lr: 1.9929e-05 eta: 7:15:57 time: 1.3673 data_time: 0.0142 memory: 10922 loss: 0.4272 2025/03/23 17:58:01 - mmengine - INFO - Iter(train) [ 1290/19176] lr: 1.9927e-05 eta: 7:14:59 time: 1.1515 data_time: 0.0151 memory: 10582 loss: 0.2696 2025/03/23 17:58:07 - mmengine - INFO - Iter(train) [ 1300/19176] lr: 1.9925e-05 eta: 7:12:53 time: 0.6471 data_time: 0.0111 memory: 9670 loss: 0.2259 2025/03/23 17:58:32 - mmengine - INFO - Iter(train) [ 1310/19176] lr: 1.9923e-05 eta: 7:14:55 time: 2.4567 data_time: 0.0147 memory: 18869 loss: 0.2356 2025/03/23 17:58:50 - mmengine - INFO - Iter(train) [ 1320/19176] lr: 1.9921e-05 eta: 7:15:33 time: 1.8447 data_time: 0.0149 memory: 12741 loss: 0.2319 2025/03/23 17:59:07 - mmengine - INFO - Iter(train) [ 1330/19176] lr: 1.9919e-05 eta: 7:15:50 time: 1.6991 data_time: 0.0142 memory: 11808 loss: 0.1959 2025/03/23 17:59:24 - mmengine - INFO - Iter(train) [ 1340/19176] lr: 1.9917e-05 eta: 7:15:58 time: 1.6368 data_time: 0.0159 memory: 11418 loss: 0.2477 2025/03/23 17:59:40 - mmengine - INFO - Iter(train) [ 1350/19176] lr: 1.9915e-05 eta: 7:16:02 time: 1.6065 data_time: 0.0155 memory: 11384 loss: 0.2273 2025/03/23 17:59:55 - mmengine - INFO - Iter(train) [ 1360/19176] lr: 1.9912e-05 eta: 7:15:58 time: 1.5495 data_time: 0.0143 memory: 11283 loss: 0.2185 2025/03/23 18:00:11 - mmengine - INFO - Iter(train) [ 1370/19176] lr: 1.9910e-05 eta: 7:15:49 time: 1.5158 data_time: 0.0142 memory: 11173 loss: 0.2882 2025/03/23 18:00:25 - mmengine - INFO - Iter(train) [ 1380/19176] lr: 1.9908e-05 eta: 7:15:27 time: 1.4066 data_time: 0.0138 memory: 11054 loss: 0.2235 2025/03/23 18:00:37 - mmengine - INFO - Iter(train) [ 1390/19176] lr: 1.9906e-05 eta: 7:14:44 time: 1.2528 data_time: 0.0139 memory: 10808 loss: 0.3344 2025/03/23 18:00:46 - mmengine - INFO - Iter(train) [ 1400/19176] lr: 1.9903e-05 eta: 7:13:22 time: 0.9296 data_time: 0.0130 memory: 10093 loss: 0.3046 2025/03/23 18:01:08 - mmengine - INFO - Iter(train) [ 1410/19176] lr: 1.9901e-05 eta: 7:14:31 time: 2.1333 data_time: 0.0136 memory: 18234 loss: 0.2258 2025/03/23 18:01:25 - mmengine - INFO - Iter(train) [ 1420/19176] lr: 1.9899e-05 eta: 7:14:50 time: 1.7361 data_time: 0.0148 memory: 12105 loss: 0.2180 2025/03/23 18:01:42 - mmengine - INFO - Iter(train) [ 1430/19176] lr: 1.9896e-05 eta: 7:15:05 time: 1.7024 data_time: 0.0142 memory: 12229 loss: 0.2172 2025/03/23 18:01:59 - mmengine - INFO - Iter(train) [ 1440/19176] lr: 1.9894e-05 eta: 7:15:11 time: 1.6394 data_time: 0.0144 memory: 11451 loss: 0.2168 2025/03/23 18:02:14 - mmengine - INFO - Iter(train) [ 1450/19176] lr: 1.9891e-05 eta: 7:15:03 time: 1.5296 data_time: 0.0140 memory: 11744 loss: 0.2197 2025/03/23 18:02:28 - mmengine - INFO - Iter(train) [ 1460/19176] lr: 1.9889e-05 eta: 7:14:42 time: 1.4245 data_time: 0.0136 memory: 11064 loss: 0.2527 2025/03/23 18:02:42 - mmengine - INFO - Iter(train) [ 1470/19176] lr: 1.9886e-05 eta: 7:14:15 time: 1.3653 data_time: 0.0137 memory: 10919 loss: 0.3316 2025/03/23 18:02:54 - mmengine - INFO - Iter(train) [ 1480/19176] lr: 1.9884e-05 eta: 7:13:25 time: 1.1824 data_time: 0.0133 memory: 10484 loss: 0.8494 2025/03/23 18:03:04 - mmengine - INFO - Iter(train) [ 1490/19176] lr: 1.9881e-05 eta: 7:12:19 time: 1.0323 data_time: 0.0124 memory: 10183 loss: 0.2806 2025/03/23 18:03:12 - mmengine - INFO - Iter(train) [ 1500/19176] lr: 1.9878e-05 eta: 7:10:45 time: 0.7962 data_time: 0.0116 memory: 9734 loss: 0.2482 2025/03/23 18:03:32 - mmengine - INFO - Iter(train) [ 1510/19176] lr: 1.9876e-05 eta: 7:11:31 time: 1.9789 data_time: 0.0139 memory: 13542 loss: 0.2599 2025/03/23 18:03:49 - mmengine - INFO - Iter(train) [ 1520/19176] lr: 1.9873e-05 eta: 7:11:50 time: 1.7573 data_time: 0.0147 memory: 12047 loss: 0.1948 2025/03/23 18:04:06 - mmengine - INFO - Iter(train) [ 1530/19176] lr: 1.9870e-05 eta: 7:12:01 time: 1.6860 data_time: 0.0144 memory: 11642 loss: 0.2274 2025/03/23 18:04:22 - mmengine - INFO - Iter(train) [ 1540/19176] lr: 1.9868e-05 eta: 7:12:05 time: 1.6353 data_time: 0.0144 memory: 11493 loss: 0.2366 2025/03/23 18:04:38 - mmengine - INFO - Iter(train) [ 1550/19176] lr: 1.9865e-05 eta: 7:11:57 time: 1.5283 data_time: 0.0141 memory: 11253 loss: 0.2432 2025/03/23 18:04:52 - mmengine - INFO - Iter(train) [ 1560/19176] lr: 1.9862e-05 eta: 7:11:42 time: 1.4720 data_time: 0.0147 memory: 11088 loss: 0.2871 2025/03/23 18:05:06 - mmengine - INFO - Iter(train) [ 1570/19176] lr: 1.9859e-05 eta: 7:11:19 time: 1.3916 data_time: 0.0147 memory: 11031 loss: 0.2507 2025/03/23 18:05:19 - mmengine - INFO - Iter(train) [ 1580/19176] lr: 1.9857e-05 eta: 7:10:41 time: 1.2610 data_time: 0.0134 memory: 10676 loss: 0.3365 2025/03/23 18:05:29 - mmengine - INFO - Iter(train) [ 1590/19176] lr: 1.9854e-05 eta: 7:09:40 time: 1.0543 data_time: 0.0126 memory: 10275 loss: 0.2148 2025/03/23 18:05:37 - mmengine - INFO - Iter(train) [ 1600/19176] lr: 1.9851e-05 eta: 7:08:03 time: 0.7147 data_time: 0.0116 memory: 9593 loss: 0.2822 2025/03/23 18:05:57 - mmengine - INFO - Iter(train) [ 1610/19176] lr: 1.9848e-05 eta: 7:08:53 time: 2.0551 data_time: 0.0140 memory: 15096 loss: 0.2284 2025/03/23 18:06:15 - mmengine - INFO - Iter(train) [ 1620/19176] lr: 1.9845e-05 eta: 7:09:13 time: 1.7807 data_time: 0.0145 memory: 12093 loss: 0.2211 2025/03/23 18:06:32 - mmengine - INFO - Iter(train) [ 1630/19176] lr: 1.9842e-05 eta: 7:09:24 time: 1.7091 data_time: 0.0149 memory: 11659 loss: 0.2040 2025/03/23 18:06:48 - mmengine - INFO - Iter(train) [ 1640/19176] lr: 1.9839e-05 eta: 7:09:27 time: 1.6309 data_time: 0.0144 memory: 11533 loss: 0.2278 2025/03/23 18:07:04 - mmengine - INFO - Iter(train) [ 1650/19176] lr: 1.9836e-05 eta: 7:09:20 time: 1.5367 data_time: 0.0143 memory: 11268 loss: 0.2665 2025/03/23 18:07:19 - mmengine - INFO - Iter(train) [ 1660/19176] lr: 1.9833e-05 eta: 7:09:10 time: 1.5151 data_time: 0.0147 memory: 11166 loss: 0.2292 2025/03/23 18:07:33 - mmengine - INFO - Iter(train) [ 1670/19176] lr: 1.9830e-05 eta: 7:08:51 time: 1.4354 data_time: 0.0137 memory: 11033 loss: 0.2983 2025/03/23 18:07:47 - mmengine - INFO - Iter(train) [ 1680/19176] lr: 1.9827e-05 eta: 7:08:26 time: 1.3711 data_time: 0.0143 memory: 10864 loss: 0.2688 2025/03/23 18:07:58 - mmengine - INFO - Iter(train) [ 1690/19176] lr: 1.9824e-05 eta: 7:07:33 time: 1.1004 data_time: 0.0124 memory: 10415 loss: 0.2469 2025/03/23 18:08:07 - mmengine - INFO - Iter(train) [ 1700/19176] lr: 1.9820e-05 eta: 7:06:18 time: 0.8723 data_time: 0.0117 memory: 9887 loss: 0.2547 2025/03/23 18:08:26 - mmengine - INFO - Iter(train) [ 1710/19176] lr: 1.9817e-05 eta: 7:06:47 time: 1.8917 data_time: 0.0141 memory: 14151 loss: 0.2258 2025/03/23 18:08:43 - mmengine - INFO - Iter(train) [ 1720/19176] lr: 1.9814e-05 eta: 7:07:00 time: 1.7447 data_time: 0.0143 memory: 11896 loss: 0.2414 2025/03/23 18:09:00 - mmengine - INFO - Iter(train) [ 1730/19176] lr: 1.9811e-05 eta: 7:07:07 time: 1.6804 data_time: 0.0150 memory: 11668 loss: 0.1854 2025/03/23 18:09:16 - mmengine - INFO - Iter(train) [ 1740/19176] lr: 1.9807e-05 eta: 7:07:09 time: 1.6300 data_time: 0.0144 memory: 11465 loss: 0.2724 2025/03/23 18:09:32 - mmengine - INFO - Iter(train) [ 1750/19176] lr: 1.9804e-05 eta: 7:07:04 time: 1.5746 data_time: 0.0144 memory: 11376 loss: 0.2200 2025/03/23 18:09:47 - mmengine - INFO - Iter(train) [ 1760/19176] lr: 1.9801e-05 eta: 7:06:53 time: 1.5036 data_time: 0.0142 memory: 11149 loss: 0.2320 2025/03/23 18:10:01 - mmengine - INFO - Iter(train) [ 1770/19176] lr: 1.9797e-05 eta: 7:06:35 time: 1.4358 data_time: 0.0141 memory: 11057 loss: 0.2676 2025/03/23 18:10:13 - mmengine - INFO - Iter(train) [ 1780/19176] lr: 1.9794e-05 eta: 7:05:52 time: 1.1874 data_time: 0.0136 memory: 10558 loss: 0.2459 2025/03/23 18:10:24 - mmengine - INFO - Iter(train) [ 1790/19176] lr: 1.9791e-05 eta: 7:04:58 time: 1.0563 data_time: 0.0120 memory: 10250 loss: 0.2466 2025/03/23 18:10:32 - mmengine - INFO - Iter(train) [ 1800/19176] lr: 1.9787e-05 eta: 7:03:38 time: 0.7910 data_time: 0.0117 memory: 9944 loss: 0.2661 2025/03/23 18:10:50 - mmengine - INFO - Iter(train) [ 1810/19176] lr: 1.9784e-05 eta: 7:04:01 time: 1.8617 data_time: 0.0140 memory: 13422 loss: 0.2270 2025/03/23 18:11:07 - mmengine - INFO - Iter(train) [ 1820/19176] lr: 1.9780e-05 eta: 7:04:07 time: 1.6766 data_time: 0.0143 memory: 11664 loss: 0.2148 2025/03/23 18:11:23 - mmengine - INFO - Iter(train) [ 1830/19176] lr: 1.9777e-05 eta: 7:04:06 time: 1.6140 data_time: 0.0141 memory: 11413 loss: 0.2289 2025/03/23 18:11:39 - mmengine - INFO - Iter(train) [ 1840/19176] lr: 1.9773e-05 eta: 7:04:01 time: 1.5629 data_time: 0.0146 memory: 11311 loss: 0.2306 2025/03/23 18:11:54 - mmengine - INFO - Iter(train) [ 1850/19176] lr: 1.9769e-05 eta: 7:03:51 time: 1.5242 data_time: 0.0143 memory: 11281 loss: 0.2408 2025/03/23 18:12:09 - mmengine - INFO - Iter(train) [ 1860/19176] lr: 1.9766e-05 eta: 7:03:40 time: 1.5055 data_time: 0.0145 memory: 11192 loss: 0.2400 2025/03/23 18:12:24 - mmengine - INFO - Iter(train) [ 1870/19176] lr: 1.9762e-05 eta: 7:03:23 time: 1.4479 data_time: 0.0143 memory: 11049 loss: 0.3084 2025/03/23 18:12:37 - mmengine - INFO - Iter(train) [ 1880/19176] lr: 1.9758e-05 eta: 7:02:56 time: 1.3312 data_time: 0.0139 memory: 10914 loss: 0.2431 2025/03/23 18:12:48 - mmengine - INFO - Iter(train) [ 1890/19176] lr: 1.9755e-05 eta: 7:02:08 time: 1.1036 data_time: 0.0125 memory: 10404 loss: 0.2468 2025/03/23 18:12:56 - mmengine - INFO - Iter(train) [ 1900/19176] lr: 1.9751e-05 eta: 7:00:54 time: 0.8108 data_time: 0.0116 memory: 9974 loss: 0.2184 2025/03/23 18:13:15 - mmengine - INFO - Iter(train) [ 1910/19176] lr: 1.9747e-05 eta: 7:01:16 time: 1.8628 data_time: 0.0143 memory: 12954 loss: 0.2253 2025/03/23 18:13:32 - mmengine - INFO - Iter(train) [ 1920/19176] lr: 1.9743e-05 eta: 7:01:26 time: 1.7421 data_time: 0.0156 memory: 12068 loss: 0.2404 2025/03/23 18:13:49 - mmengine - INFO - Iter(train) [ 1930/19176] lr: 1.9740e-05 eta: 7:01:28 time: 1.6545 data_time: 0.0150 memory: 11559 loss: 0.2101 2025/03/23 18:14:05 - mmengine - INFO - Iter(train) [ 1940/19176] lr: 1.9736e-05 eta: 7:01:26 time: 1.6040 data_time: 0.0149 memory: 11352 loss: 0.2067 2025/03/23 18:14:20 - mmengine - INFO - Iter(train) [ 1950/19176] lr: 1.9732e-05 eta: 7:01:18 time: 1.5391 data_time: 0.0146 memory: 11262 loss: 0.2517 2025/03/23 18:14:35 - mmengine - INFO - Iter(train) [ 1960/19176] lr: 1.9728e-05 eta: 7:01:05 time: 1.4873 data_time: 0.0146 memory: 11104 loss: 0.2548 2025/03/23 18:14:48 - mmengine - INFO - Iter(train) [ 1970/19176] lr: 1.9724e-05 eta: 7:00:38 time: 1.3291 data_time: 0.0134 memory: 10916 loss: 0.2257 2025/03/23 18:15:00 - mmengine - INFO - Iter(train) [ 1980/19176] lr: 1.9720e-05 eta: 6:59:56 time: 1.1576 data_time: 0.0128 memory: 10479 loss: 0.2082 2025/03/23 18:15:10 - mmengine - INFO - Iter(train) [ 1990/19176] lr: 1.9716e-05 eta: 6:59:03 time: 1.0149 data_time: 0.0128 memory: 10157 loss: 0.2919 2025/03/23 18:15:18 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 18:15:18 - mmengine - INFO - Iter(train) [ 2000/19176] lr: 1.9712e-05 eta: 6:57:49 time: 0.7695 data_time: 0.0120 memory: 9866 loss: 0.2672 2025/03/23 18:15:18 - mmengine - INFO - Saving checkpoint at 2000 iterations 2025/03/23 18:15:37 - mmengine - INFO - Iter(train) [ 2010/19176] lr: 1.9708e-05 eta: 6:58:13 time: 1.9170 data_time: 0.0929 memory: 12505 loss: 0.2222 2025/03/23 18:15:54 - mmengine - INFO - Iter(train) [ 2020/19176] lr: 1.9704e-05 eta: 6:58:20 time: 1.7151 data_time: 0.0147 memory: 11769 loss: 0.2144 2025/03/23 18:16:10 - mmengine - INFO - Iter(train) [ 2030/19176] lr: 1.9700e-05 eta: 6:58:19 time: 1.6228 data_time: 0.0144 memory: 11434 loss: 0.2251 2025/03/23 18:16:26 - mmengine - INFO - Iter(train) [ 2040/19176] lr: 1.9696e-05 eta: 6:58:15 time: 1.5913 data_time: 0.0145 memory: 11337 loss: 0.2326 2025/03/23 18:16:41 - mmengine - INFO - Iter(train) [ 2050/19176] lr: 1.9692e-05 eta: 6:58:04 time: 1.5124 data_time: 0.0148 memory: 11185 loss: 0.2559 2025/03/23 18:16:56 - mmengine - INFO - Iter(train) [ 2060/19176] lr: 1.9688e-05 eta: 6:57:47 time: 1.4343 data_time: 0.0144 memory: 11015 loss: 0.2841 2025/03/23 18:17:08 - mmengine - INFO - Iter(train) [ 2070/19176] lr: 1.9683e-05 eta: 6:57:17 time: 1.2715 data_time: 0.0143 memory: 10800 loss: 0.2747 2025/03/23 18:17:19 - mmengine - INFO - Iter(train) [ 2080/19176] lr: 1.9679e-05 eta: 6:56:31 time: 1.0885 data_time: 0.0129 memory: 10270 loss: 0.2460 2025/03/23 18:17:29 - mmengine - INFO - Iter(train) [ 2090/19176] lr: 1.9675e-05 eta: 6:55:40 time: 1.0202 data_time: 0.0122 memory: 10151 loss: 0.2499 2025/03/23 18:17:37 - mmengine - INFO - Iter(train) [ 2100/19176] lr: 1.9671e-05 eta: 6:54:27 time: 0.7356 data_time: 0.0113 memory: 9876 loss: 0.2314 2025/03/23 18:17:57 - mmengine - INFO - Iter(train) [ 2110/19176] lr: 1.9666e-05 eta: 6:55:00 time: 2.0440 data_time: 0.0141 memory: 16526 loss: 0.2093 2025/03/23 18:18:14 - mmengine - INFO - Iter(train) [ 2120/19176] lr: 1.9662e-05 eta: 6:55:07 time: 1.7265 data_time: 0.0141 memory: 11815 loss: 0.2233 2025/03/23 18:18:31 - mmengine - INFO - Iter(train) [ 2130/19176] lr: 1.9658e-05 eta: 6:55:09 time: 1.6699 data_time: 0.0148 memory: 11691 loss: 0.2388 2025/03/23 18:18:47 - mmengine - INFO - Iter(train) [ 2140/19176] lr: 1.9653e-05 eta: 6:55:07 time: 1.6155 data_time: 0.0153 memory: 11425 loss: 0.2317 2025/03/23 18:19:03 - mmengine - INFO - Iter(train) [ 2150/19176] lr: 1.9649e-05 eta: 6:54:59 time: 1.5500 data_time: 0.0148 memory: 11249 loss: 0.2901 2025/03/23 18:19:18 - mmengine - INFO - Iter(train) [ 2160/19176] lr: 1.9644e-05 eta: 6:54:47 time: 1.4984 data_time: 0.0145 memory: 11156 loss: 0.2281 2025/03/23 18:19:32 - mmengine - INFO - Iter(train) [ 2170/19176] lr: 1.9640e-05 eta: 6:54:28 time: 1.4015 data_time: 0.0135 memory: 10988 loss: 0.2334 2025/03/23 18:19:45 - mmengine - INFO - Iter(train) [ 2180/19176] lr: 1.9635e-05 eta: 6:53:59 time: 1.2826 data_time: 0.0136 memory: 10729 loss: 0.3141 2025/03/23 18:19:56 - mmengine - INFO - Iter(train) [ 2190/19176] lr: 1.9631e-05 eta: 6:53:16 time: 1.0969 data_time: 0.0124 memory: 10359 loss: 0.2398 2025/03/23 18:20:04 - mmengine - INFO - Iter(train) [ 2200/19176] lr: 1.9626e-05 eta: 6:52:12 time: 0.8167 data_time: 0.0115 memory: 10056 loss: 0.3273 2025/03/23 18:20:23 - mmengine - INFO - Iter(train) [ 2210/19176] lr: 1.9622e-05 eta: 6:52:36 time: 1.9536 data_time: 0.0135 memory: 13545 loss: 0.1984 2025/03/23 18:20:41 - mmengine - INFO - Iter(train) [ 2220/19176] lr: 1.9617e-05 eta: 6:52:45 time: 1.7763 data_time: 0.0139 memory: 12141 loss: 0.2070 2025/03/23 18:20:58 - mmengine - INFO - Iter(train) [ 2230/19176] lr: 1.9612e-05 eta: 6:52:49 time: 1.6987 data_time: 0.0140 memory: 11751 loss: 0.2284 2025/03/23 18:21:14 - mmengine - INFO - Iter(train) [ 2240/19176] lr: 1.9608e-05 eta: 6:52:47 time: 1.6267 data_time: 0.0140 memory: 11447 loss: 0.2287 2025/03/23 18:21:30 - mmengine - INFO - Iter(train) [ 2250/19176] lr: 1.9603e-05 eta: 6:52:41 time: 1.5813 data_time: 0.0142 memory: 11613 loss: 0.2740 2025/03/23 18:21:45 - mmengine - INFO - Iter(train) [ 2260/19176] lr: 1.9598e-05 eta: 6:52:29 time: 1.5001 data_time: 0.0139 memory: 11176 loss: 0.2200 2025/03/23 18:21:59 - mmengine - INFO - Iter(train) [ 2270/19176] lr: 1.9594e-05 eta: 6:52:11 time: 1.4163 data_time: 0.0137 memory: 10991 loss: 0.2560 2025/03/23 18:22:11 - mmengine - INFO - Iter(train) [ 2280/19176] lr: 1.9589e-05 eta: 6:51:37 time: 1.2029 data_time: 0.0126 memory: 10622 loss: 0.2087 2025/03/23 18:22:22 - mmengine - INFO - Iter(train) [ 2290/19176] lr: 1.9584e-05 eta: 6:50:51 time: 1.0370 data_time: 0.0130 memory: 10218 loss: 0.2501 2025/03/23 18:22:29 - mmengine - INFO - Iter(train) [ 2300/19176] lr: 1.9579e-05 eta: 6:49:45 time: 0.7619 data_time: 0.0119 memory: 9652 loss: 0.2053 2025/03/23 18:22:50 - mmengine - INFO - Iter(train) [ 2310/19176] lr: 1.9574e-05 eta: 6:50:12 time: 2.0294 data_time: 0.0135 memory: 16945 loss: 0.1960 2025/03/23 18:23:07 - mmengine - INFO - Iter(train) [ 2320/19176] lr: 1.9569e-05 eta: 6:50:16 time: 1.7133 data_time: 0.0135 memory: 11813 loss: 0.2929 2025/03/23 18:23:23 - mmengine - INFO - Iter(train) [ 2330/19176] lr: 1.9564e-05 eta: 6:50:13 time: 1.6176 data_time: 0.0133 memory: 11490 loss: 0.2126 2025/03/23 18:23:38 - mmengine - INFO - Iter(train) [ 2340/19176] lr: 1.9559e-05 eta: 6:50:03 time: 1.5169 data_time: 0.0137 memory: 11220 loss: 0.2288 2025/03/23 18:23:53 - mmengine - INFO - Iter(train) [ 2350/19176] lr: 1.9554e-05 eta: 6:49:49 time: 1.4804 data_time: 0.0142 memory: 11129 loss: 0.2358 2025/03/23 18:24:07 - mmengine - INFO - Iter(train) [ 2360/19176] lr: 1.9549e-05 eta: 6:49:33 time: 1.4393 data_time: 0.0141 memory: 11014 loss: 0.2560 2025/03/23 18:24:20 - mmengine - INFO - Iter(train) [ 2370/19176] lr: 1.9544e-05 eta: 6:49:08 time: 1.3081 data_time: 0.0136 memory: 10815 loss: 0.2994 2025/03/23 18:24:32 - mmengine - INFO - Iter(train) [ 2380/19176] lr: 1.9539e-05 eta: 6:48:29 time: 1.1240 data_time: 0.0121 memory: 10405 loss: 0.1891 2025/03/23 18:24:42 - mmengine - INFO - Iter(train) [ 2390/19176] lr: 1.9534e-05 eta: 6:47:47 time: 1.0680 data_time: 0.0120 memory: 10301 loss: 0.2197 2025/03/23 18:24:50 - mmengine - INFO - Iter(train) [ 2400/19176] lr: 1.9529e-05 eta: 6:46:42 time: 0.7413 data_time: 0.0109 memory: 9953 loss: 0.2365 2025/03/23 18:25:10 - mmengine - INFO - Iter(train) [ 2410/19176] lr: 1.9524e-05 eta: 6:47:05 time: 1.9916 data_time: 0.0131 memory: 14188 loss: 0.2048 2025/03/23 18:25:27 - mmengine - INFO - Iter(train) [ 2420/19176] lr: 1.9519e-05 eta: 6:47:09 time: 1.7178 data_time: 0.0142 memory: 11869 loss: 0.2133 2025/03/23 18:25:43 - mmengine - INFO - Iter(train) [ 2430/19176] lr: 1.9514e-05 eta: 6:47:07 time: 1.6485 data_time: 0.0144 memory: 11470 loss: 0.2103 2025/03/23 18:25:59 - mmengine - INFO - Iter(train) [ 2440/19176] lr: 1.9509e-05 eta: 6:47:02 time: 1.5999 data_time: 0.0142 memory: 11374 loss: 0.2245 2025/03/23 18:26:15 - mmengine - INFO - Iter(train) [ 2450/19176] lr: 1.9503e-05 eta: 6:46:54 time: 1.5529 data_time: 0.0150 memory: 11273 loss: 0.2322 2025/03/23 18:26:30 - mmengine - INFO - Iter(train) [ 2460/19176] lr: 1.9498e-05 eta: 6:46:42 time: 1.4940 data_time: 0.0144 memory: 11195 loss: 0.2589 2025/03/23 18:26:45 - mmengine - INFO - Iter(train) [ 2470/19176] lr: 1.9493e-05 eta: 6:46:28 time: 1.4750 data_time: 0.0149 memory: 11108 loss: 0.2673 2025/03/23 18:26:58 - mmengine - INFO - Iter(train) [ 2480/19176] lr: 1.9487e-05 eta: 6:46:07 time: 1.3560 data_time: 0.0144 memory: 10944 loss: 0.2192 2025/03/23 18:27:09 - mmengine - INFO - Iter(train) [ 2490/19176] lr: 1.9482e-05 eta: 6:45:27 time: 1.0889 data_time: 0.0124 memory: 10310 loss: 0.2204 2025/03/23 18:27:17 - mmengine - INFO - Iter(train) [ 2500/19176] lr: 1.9477e-05 eta: 6:44:31 time: 0.8323 data_time: 0.0118 memory: 9859 loss: 0.3069 2025/03/23 18:27:37 - mmengine - INFO - Iter(train) [ 2510/19176] lr: 1.9471e-05 eta: 6:44:50 time: 1.9660 data_time: 0.0141 memory: 14641 loss: 0.2020 2025/03/23 18:27:54 - mmengine - INFO - Iter(train) [ 2520/19176] lr: 1.9466e-05 eta: 6:44:52 time: 1.7071 data_time: 0.0142 memory: 11722 loss: 0.2041 2025/03/23 18:28:11 - mmengine - INFO - Iter(train) [ 2530/19176] lr: 1.9460e-05 eta: 6:44:51 time: 1.6550 data_time: 0.0148 memory: 11479 loss: 0.1981 2025/03/23 18:28:27 - mmengine - INFO - Iter(train) [ 2540/19176] lr: 1.9455e-05 eta: 6:44:47 time: 1.6227 data_time: 0.0152 memory: 11396 loss: 0.1941 2025/03/23 18:28:43 - mmengine - INFO - Iter(train) [ 2550/19176] lr: 1.9449e-05 eta: 6:44:40 time: 1.5742 data_time: 0.0148 memory: 11314 loss: 0.2492 2025/03/23 18:28:58 - mmengine - INFO - Iter(train) [ 2560/19176] lr: 1.9444e-05 eta: 6:44:29 time: 1.5282 data_time: 0.0145 memory: 11206 loss: 0.2484 2025/03/23 18:29:12 - mmengine - INFO - Iter(train) [ 2570/19176] lr: 1.9438e-05 eta: 6:44:13 time: 1.4378 data_time: 0.0146 memory: 11047 loss: 0.3122 2025/03/23 18:29:25 - mmengine - INFO - Iter(train) [ 2580/19176] lr: 1.9433e-05 eta: 6:43:45 time: 1.2446 data_time: 0.0138 memory: 10717 loss: 0.2697 2025/03/23 18:29:35 - mmengine - INFO - Iter(train) [ 2590/19176] lr: 1.9427e-05 eta: 6:43:03 time: 1.0354 data_time: 0.0126 memory: 10214 loss: 0.2483 2025/03/23 18:29:42 - mmengine - INFO - Iter(train) [ 2600/19176] lr: 1.9421e-05 eta: 6:42:03 time: 0.7433 data_time: 0.0116 memory: 9991 loss: 0.2309 2025/03/23 18:30:02 - mmengine - INFO - Iter(train) [ 2610/19176] lr: 1.9416e-05 eta: 6:42:22 time: 1.9826 data_time: 0.0138 memory: 15361 loss: 0.2684 2025/03/23 18:30:20 - mmengine - INFO - Iter(train) [ 2620/19176] lr: 1.9410e-05 eta: 6:42:24 time: 1.7260 data_time: 0.0146 memory: 11921 loss: 0.1807 2025/03/23 18:30:36 - mmengine - INFO - Iter(train) [ 2630/19176] lr: 1.9404e-05 eta: 6:42:25 time: 1.6944 data_time: 0.0146 memory: 11733 loss: 0.1839 2025/03/23 18:30:53 - mmengine - INFO - Iter(train) [ 2640/19176] lr: 1.9399e-05 eta: 6:42:21 time: 1.6424 data_time: 0.0144 memory: 11501 loss: 0.2245 2025/03/23 18:31:09 - mmengine - INFO - Iter(train) [ 2650/19176] lr: 1.9393e-05 eta: 6:42:15 time: 1.5940 data_time: 0.0144 memory: 11372 loss: 0.2130 2025/03/23 18:31:24 - mmengine - INFO - Iter(train) [ 2660/19176] lr: 1.9387e-05 eta: 6:42:05 time: 1.5248 data_time: 0.0143 memory: 11189 loss: 0.2178 2025/03/23 18:31:38 - mmengine - INFO - Iter(train) [ 2670/19176] lr: 1.9381e-05 eta: 6:41:48 time: 1.4289 data_time: 0.0140 memory: 11065 loss: 0.2391 2025/03/23 18:31:51 - mmengine - INFO - Iter(train) [ 2680/19176] lr: 1.9375e-05 eta: 6:41:22 time: 1.2771 data_time: 0.0149 memory: 10700 loss: 0.2788 2025/03/23 18:32:02 - mmengine - INFO - Iter(train) [ 2690/19176] lr: 1.9369e-05 eta: 6:40:45 time: 1.0928 data_time: 0.0132 memory: 10382 loss: 0.2144 2025/03/23 18:32:10 - mmengine - INFO - Iter(train) [ 2700/19176] lr: 1.9363e-05 eta: 6:39:50 time: 0.7894 data_time: 0.0123 memory: 9892 loss: 0.3463 2025/03/23 18:32:29 - mmengine - INFO - Iter(train) [ 2710/19176] lr: 1.9357e-05 eta: 6:40:05 time: 1.9446 data_time: 0.0140 memory: 14557 loss: 0.2810 2025/03/23 18:32:46 - mmengine - INFO - Iter(train) [ 2720/19176] lr: 1.9352e-05 eta: 6:40:05 time: 1.7086 data_time: 0.0145 memory: 11765 loss: 0.1945 2025/03/23 18:33:03 - mmengine - INFO - Iter(train) [ 2730/19176] lr: 1.9346e-05 eta: 6:40:04 time: 1.6720 data_time: 0.0147 memory: 11744 loss: 0.2408 2025/03/23 18:33:19 - mmengine - INFO - Iter(train) [ 2740/19176] lr: 1.9340e-05 eta: 6:39:57 time: 1.5911 data_time: 0.0143 memory: 11371 loss: 0.2183 2025/03/23 18:33:34 - mmengine - INFO - Iter(train) [ 2750/19176] lr: 1.9333e-05 eta: 6:39:47 time: 1.5332 data_time: 0.0151 memory: 11253 loss: 0.2293 2025/03/23 18:33:49 - mmengine - INFO - Iter(train) [ 2760/19176] lr: 1.9327e-05 eta: 6:39:33 time: 1.4784 data_time: 0.0145 memory: 11121 loss: 0.2070 2025/03/23 18:34:03 - mmengine - INFO - Iter(train) [ 2770/19176] lr: 1.9321e-05 eta: 6:39:12 time: 1.3485 data_time: 0.0137 memory: 10900 loss: 0.2134 2025/03/23 18:34:14 - mmengine - INFO - Iter(train) [ 2780/19176] lr: 1.9315e-05 eta: 6:38:38 time: 1.1389 data_time: 0.0131 memory: 10413 loss: 0.2344 2025/03/23 18:34:24 - mmengine - INFO - Iter(train) [ 2790/19176] lr: 1.9309e-05 eta: 6:37:58 time: 1.0174 data_time: 0.0123 memory: 10154 loss: 0.2113 2025/03/23 18:34:31 - mmengine - INFO - Iter(train) [ 2800/19176] lr: 1.9303e-05 eta: 6:36:59 time: 0.6949 data_time: 0.0112 memory: 9645 loss: 0.2470 2025/03/23 18:34:49 - mmengine - INFO - Iter(train) [ 2810/19176] lr: 1.9297e-05 eta: 6:37:05 time: 1.8180 data_time: 0.0138 memory: 12445 loss: 0.2011 2025/03/23 18:35:06 - mmengine - INFO - Iter(train) [ 2820/19176] lr: 1.9290e-05 eta: 6:37:05 time: 1.6942 data_time: 0.0141 memory: 11615 loss: 0.2120 2025/03/23 18:35:23 - mmengine - INFO - Iter(train) [ 2830/19176] lr: 1.9284e-05 eta: 6:37:00 time: 1.6300 data_time: 0.0147 memory: 11472 loss: 0.2231 2025/03/23 18:35:38 - mmengine - INFO - Iter(train) [ 2840/19176] lr: 1.9278e-05 eta: 6:36:52 time: 1.5723 data_time: 0.0142 memory: 11394 loss: 0.2412 2025/03/23 18:35:54 - mmengine - INFO - Iter(train) [ 2850/19176] lr: 1.9271e-05 eta: 6:36:41 time: 1.5261 data_time: 0.0145 memory: 11219 loss: 0.2082 2025/03/23 18:36:08 - mmengine - INFO - Iter(train) [ 2860/19176] lr: 1.9265e-05 eta: 6:36:27 time: 1.4668 data_time: 0.0140 memory: 11140 loss: 0.2258 2025/03/23 18:36:23 - mmengine - INFO - Iter(train) [ 2870/19176] lr: 1.9259e-05 eta: 6:36:11 time: 1.4304 data_time: 0.0140 memory: 11034 loss: 0.2155 2025/03/23 18:36:36 - mmengine - INFO - Iter(train) [ 2880/19176] lr: 1.9252e-05 eta: 6:35:51 time: 1.3604 data_time: 0.0137 memory: 10909 loss: 0.2833 2025/03/23 18:36:47 - mmengine - INFO - Iter(train) [ 2890/19176] lr: 1.9246e-05 eta: 6:35:18 time: 1.1257 data_time: 0.0132 memory: 10463 loss: 0.2775 2025/03/23 18:36:56 - mmengine - INFO - Iter(train) [ 2900/19176] lr: 1.9240e-05 eta: 6:34:27 time: 0.8084 data_time: 0.0118 memory: 10012 loss: 0.2866 2025/03/23 18:37:15 - mmengine - INFO - Iter(train) [ 2910/19176] lr: 1.9233e-05 eta: 6:34:37 time: 1.9008 data_time: 0.0142 memory: 13524 loss: 0.1947 2025/03/23 18:37:32 - mmengine - INFO - Iter(train) [ 2920/19176] lr: 1.9227e-05 eta: 6:34:38 time: 1.7272 data_time: 0.0150 memory: 11860 loss: 0.2089 2025/03/23 18:37:49 - mmengine - INFO - Iter(train) [ 2930/19176] lr: 1.9220e-05 eta: 6:34:38 time: 1.7223 data_time: 0.0149 memory: 12317 loss: 0.1847 2025/03/23 18:38:05 - mmengine - INFO - Iter(train) [ 2940/19176] lr: 1.9213e-05 eta: 6:34:34 time: 1.6432 data_time: 0.0145 memory: 11459 loss: 0.2054 2025/03/23 18:38:22 - mmengine - INFO - Iter(train) [ 2950/19176] lr: 1.9207e-05 eta: 6:34:28 time: 1.6260 data_time: 0.0155 memory: 11436 loss: 0.2310 2025/03/23 18:38:37 - mmengine - INFO - Iter(train) [ 2960/19176] lr: 1.9200e-05 eta: 6:34:19 time: 1.5485 data_time: 0.0142 memory: 11334 loss: 0.2281 2025/03/23 18:38:52 - mmengine - INFO - Iter(train) [ 2970/19176] lr: 1.9194e-05 eta: 6:34:03 time: 1.4338 data_time: 0.0146 memory: 11069 loss: 0.2528 2025/03/23 18:39:04 - mmengine - INFO - Iter(train) [ 2980/19176] lr: 1.9187e-05 eta: 6:33:38 time: 1.2784 data_time: 0.0132 memory: 10788 loss: 0.2646 2025/03/23 18:39:15 - mmengine - INFO - Iter(train) [ 2990/19176] lr: 1.9180e-05 eta: 6:33:02 time: 1.0539 data_time: 0.0126 memory: 10282 loss: 0.2162 2025/03/23 18:39:23 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 18:39:23 - mmengine - INFO - Iter(train) [ 3000/19176] lr: 1.9174e-05 eta: 6:32:11 time: 0.7851 data_time: 0.0114 memory: 9842 loss: 0.2026 2025/03/23 18:39:23 - mmengine - INFO - Saving checkpoint at 3000 iterations 2025/03/23 18:39:45 - mmengine - INFO - Iter(train) [ 3010/19176] lr: 1.9167e-05 eta: 6:32:37 time: 2.2073 data_time: 0.0916 memory: 18682 loss: 0.2438 2025/03/23 18:40:02 - mmengine - INFO - Iter(train) [ 3020/19176] lr: 1.9160e-05 eta: 6:32:36 time: 1.7181 data_time: 0.0145 memory: 11882 loss: 0.1876 2025/03/23 18:40:19 - mmengine - INFO - Iter(train) [ 3030/19176] lr: 1.9153e-05 eta: 6:32:35 time: 1.7003 data_time: 0.0144 memory: 11675 loss: 0.2028 2025/03/23 18:40:35 - mmengine - INFO - Iter(train) [ 3040/19176] lr: 1.9147e-05 eta: 6:32:29 time: 1.6316 data_time: 0.0141 memory: 11453 loss: 0.1988 2025/03/23 18:40:51 - mmengine - INFO - Iter(train) [ 3050/19176] lr: 1.9140e-05 eta: 6:32:21 time: 1.5863 data_time: 0.0143 memory: 11657 loss: 0.2126 2025/03/23 18:41:06 - mmengine - INFO - Iter(train) [ 3060/19176] lr: 1.9133e-05 eta: 6:32:10 time: 1.5310 data_time: 0.0145 memory: 11249 loss: 0.2261 2025/03/23 18:41:21 - mmengine - INFO - Iter(train) [ 3070/19176] lr: 1.9126e-05 eta: 6:31:55 time: 1.4352 data_time: 0.0142 memory: 11066 loss: 0.2764 2025/03/23 18:41:34 - mmengine - INFO - Iter(train) [ 3080/19176] lr: 1.9119e-05 eta: 6:31:34 time: 1.3484 data_time: 0.0139 memory: 10867 loss: 0.4495 2025/03/23 18:41:46 - mmengine - INFO - Iter(train) [ 3090/19176] lr: 1.9112e-05 eta: 6:31:02 time: 1.1212 data_time: 0.0127 memory: 10478 loss: 0.2956 2025/03/23 18:41:54 - mmengine - INFO - Iter(train) [ 3100/19176] lr: 1.9105e-05 eta: 6:30:14 time: 0.8127 data_time: 0.0117 memory: 10000 loss: 0.2395 2025/03/23 18:42:15 - mmengine - INFO - Iter(train) [ 3110/19176] lr: 1.9098e-05 eta: 6:30:33 time: 2.1033 data_time: 0.0141 memory: 15904 loss: 0.2426 2025/03/23 18:42:33 - mmengine - INFO - Iter(train) [ 3120/19176] lr: 1.9091e-05 eta: 6:30:35 time: 1.7823 data_time: 0.0145 memory: 12353 loss: 0.2082 2025/03/23 18:42:50 - mmengine - INFO - Iter(train) [ 3130/19176] lr: 1.9084e-05 eta: 6:30:33 time: 1.7053 data_time: 0.0147 memory: 11726 loss: 0.2119 2025/03/23 18:43:06 - mmengine - INFO - Iter(train) [ 3140/19176] lr: 1.9077e-05 eta: 6:30:26 time: 1.6230 data_time: 0.0147 memory: 11475 loss: 0.2406 2025/03/23 18:43:21 - mmengine - INFO - Iter(train) [ 3150/19176] lr: 1.9070e-05 eta: 6:30:17 time: 1.5628 data_time: 0.0144 memory: 11296 loss: 0.2299 2025/03/23 18:43:37 - mmengine - INFO - Iter(train) [ 3160/19176] lr: 1.9063e-05 eta: 6:30:05 time: 1.5082 data_time: 0.0142 memory: 11276 loss: 0.2565 2025/03/23 18:43:50 - mmengine - INFO - Iter(train) [ 3170/19176] lr: 1.9056e-05 eta: 6:29:46 time: 1.3803 data_time: 0.0133 memory: 11004 loss: 0.2507 2025/03/23 18:44:02 - mmengine - INFO - Iter(train) [ 3180/19176] lr: 1.9048e-05 eta: 6:29:16 time: 1.1573 data_time: 0.0134 memory: 10622 loss: 0.2528 2025/03/23 18:44:11 - mmengine - INFO - Iter(train) [ 3190/19176] lr: 1.9041e-05 eta: 6:28:36 time: 0.9476 data_time: 0.0129 memory: 10005 loss: 0.2780 2025/03/23 18:44:18 - mmengine - INFO - Iter(train) [ 3200/19176] lr: 1.9034e-05 eta: 6:27:41 time: 0.6503 data_time: 0.0110 memory: 9547 loss: 0.2422 2025/03/23 18:44:37 - mmengine - INFO - Iter(train) [ 3210/19176] lr: 1.9027e-05 eta: 6:27:49 time: 1.9200 data_time: 0.0137 memory: 13636 loss: 0.1971 2025/03/23 18:44:54 - mmengine - INFO - Iter(train) [ 3220/19176] lr: 1.9019e-05 eta: 6:27:49 time: 1.7379 data_time: 0.0143 memory: 11998 loss: 0.2047 2025/03/23 18:45:11 - mmengine - INFO - Iter(train) [ 3230/19176] lr: 1.9012e-05 eta: 6:27:45 time: 1.6843 data_time: 0.0145 memory: 11668 loss: 0.2331 2025/03/23 18:45:28 - mmengine - INFO - Iter(train) [ 3240/19176] lr: 1.9005e-05 eta: 6:27:40 time: 1.6370 data_time: 0.0142 memory: 11532 loss: 0.2081 2025/03/23 18:45:43 - mmengine - INFO - Iter(train) [ 3250/19176] lr: 1.8997e-05 eta: 6:27:30 time: 1.5599 data_time: 0.0142 memory: 11307 loss: 0.2279 2025/03/23 18:45:58 - mmengine - INFO - Iter(train) [ 3260/19176] lr: 1.8990e-05 eta: 6:27:18 time: 1.5173 data_time: 0.0139 memory: 11179 loss: 0.2240 2025/03/23 18:46:12 - mmengine - INFO - Iter(train) [ 3270/19176] lr: 1.8983e-05 eta: 6:27:01 time: 1.4004 data_time: 0.0139 memory: 11021 loss: 0.2637 2025/03/23 18:46:25 - mmengine - INFO - Iter(train) [ 3280/19176] lr: 1.8975e-05 eta: 6:26:37 time: 1.2827 data_time: 0.0138 memory: 10794 loss: 0.2393 2025/03/23 18:46:36 - mmengine - INFO - Iter(train) [ 3290/19176] lr: 1.8968e-05 eta: 6:26:03 time: 1.0587 data_time: 0.0127 memory: 10229 loss: 0.2558 2025/03/23 18:46:43 - mmengine - INFO - Iter(train) [ 3300/19176] lr: 1.8960e-05 eta: 6:25:14 time: 0.7321 data_time: 0.0118 memory: 9857 loss: 0.2445 2025/03/23 18:47:05 - mmengine - INFO - Iter(train) [ 3310/19176] lr: 1.8953e-05 eta: 6:25:36 time: 2.2205 data_time: 0.0135 memory: 16729 loss: 0.2349 2025/03/23 18:47:23 - mmengine - INFO - Iter(train) [ 3320/19176] lr: 1.8945e-05 eta: 6:25:37 time: 1.7775 data_time: 0.0146 memory: 12040 loss: 0.2074 2025/03/23 18:47:40 - mmengine - INFO - Iter(train) [ 3330/19176] lr: 1.8938e-05 eta: 6:25:32 time: 1.6724 data_time: 0.0147 memory: 11708 loss: 0.1806 2025/03/23 18:47:56 - mmengine - INFO - Iter(train) [ 3340/19176] lr: 1.8930e-05 eta: 6:25:26 time: 1.6333 data_time: 0.0151 memory: 11655 loss: 0.1858 2025/03/23 18:48:12 - mmengine - INFO - Iter(train) [ 3350/19176] lr: 1.8922e-05 eta: 6:25:16 time: 1.5606 data_time: 0.0148 memory: 11337 loss: 0.2038 2025/03/23 18:48:27 - mmengine - INFO - Iter(train) [ 3360/19176] lr: 1.8915e-05 eta: 6:25:04 time: 1.5223 data_time: 0.0146 memory: 11195 loss: 0.2229 2025/03/23 18:48:42 - mmengine - INFO - Iter(train) [ 3370/19176] lr: 1.8907e-05 eta: 6:24:49 time: 1.4524 data_time: 0.0144 memory: 11079 loss: 0.2705 2025/03/23 18:48:55 - mmengine - INFO - Iter(train) [ 3380/19176] lr: 1.8899e-05 eta: 6:24:29 time: 1.3439 data_time: 0.0142 memory: 10902 loss: 0.5938 2025/03/23 18:49:06 - mmengine - INFO - Iter(train) [ 3390/19176] lr: 1.8892e-05 eta: 6:23:58 time: 1.1105 data_time: 0.0124 memory: 10321 loss: 0.1958 2025/03/23 18:49:14 - mmengine - INFO - Iter(train) [ 3400/19176] lr: 1.8884e-05 eta: 6:23:11 time: 0.7608 data_time: 0.0114 memory: 10044 loss: 0.2518 2025/03/23 18:49:34 - mmengine - INFO - Iter(train) [ 3410/19176] lr: 1.8876e-05 eta: 6:23:24 time: 2.0476 data_time: 0.0138 memory: 13429 loss: 0.2010 2025/03/23 18:49:52 - mmengine - INFO - Iter(train) [ 3420/19176] lr: 1.8868e-05 eta: 6:23:24 time: 1.7724 data_time: 0.0144 memory: 12232 loss: 0.1922 2025/03/23 18:50:09 - mmengine - INFO - Iter(train) [ 3430/19176] lr: 1.8861e-05 eta: 6:23:20 time: 1.6954 data_time: 0.0141 memory: 11705 loss: 0.1988 2025/03/23 18:50:25 - mmengine - INFO - Iter(train) [ 3440/19176] lr: 1.8853e-05 eta: 6:23:14 time: 1.6523 data_time: 0.0139 memory: 11539 loss: 0.1763 2025/03/23 18:50:41 - mmengine - INFO - Iter(train) [ 3450/19176] lr: 1.8845e-05 eta: 6:23:06 time: 1.6070 data_time: 0.0143 memory: 11445 loss: 0.2144 2025/03/23 18:50:57 - mmengine - INFO - Iter(train) [ 3460/19176] lr: 1.8837e-05 eta: 6:22:56 time: 1.5572 data_time: 0.0140 memory: 11297 loss: 0.1966 2025/03/23 18:51:12 - mmengine - INFO - Iter(train) [ 3470/19176] lr: 1.8829e-05 eta: 6:22:43 time: 1.4920 data_time: 0.0145 memory: 11198 loss: 0.3164 2025/03/23 18:51:25 - mmengine - INFO - Iter(train) [ 3480/19176] lr: 1.8821e-05 eta: 6:22:22 time: 1.3296 data_time: 0.0143 memory: 10903 loss: 0.2340 2025/03/23 18:51:36 - mmengine - INFO - Iter(train) [ 3490/19176] lr: 1.8813e-05 eta: 6:21:52 time: 1.1182 data_time: 0.0123 memory: 10429 loss: 0.2194 2025/03/23 18:51:45 - mmengine - INFO - Iter(train) [ 3500/19176] lr: 1.8805e-05 eta: 6:21:12 time: 0.8970 data_time: 0.0122 memory: 10101 loss: 0.2260 2025/03/23 18:52:04 - mmengine - INFO - Iter(train) [ 3510/19176] lr: 1.8797e-05 eta: 6:21:18 time: 1.9090 data_time: 0.0139 memory: 15007 loss: 0.2422 2025/03/23 18:52:22 - mmengine - INFO - Iter(train) [ 3520/19176] lr: 1.8789e-05 eta: 6:21:16 time: 1.7451 data_time: 0.0162 memory: 11919 loss: 0.2407 2025/03/23 18:52:39 - mmengine - INFO - Iter(train) [ 3530/19176] lr: 1.8781e-05 eta: 6:21:12 time: 1.7130 data_time: 0.0154 memory: 11666 loss: 0.1873 2025/03/23 18:52:56 - mmengine - INFO - Iter(train) [ 3540/19176] lr: 1.8773e-05 eta: 6:21:06 time: 1.6563 data_time: 0.0158 memory: 11705 loss: 0.2285 2025/03/23 18:53:11 - mmengine - INFO - Iter(train) [ 3550/19176] lr: 1.8765e-05 eta: 6:20:57 time: 1.5778 data_time: 0.0144 memory: 11354 loss: 0.1853 2025/03/23 18:53:27 - mmengine - INFO - Iter(train) [ 3560/19176] lr: 1.8757e-05 eta: 6:20:44 time: 1.5126 data_time: 0.0147 memory: 11229 loss: 0.2498 2025/03/23 18:53:41 - mmengine - INFO - Iter(train) [ 3570/19176] lr: 1.8749e-05 eta: 6:20:30 time: 1.4754 data_time: 0.0142 memory: 11129 loss: 0.2363 2025/03/23 18:53:55 - mmengine - INFO - Iter(train) [ 3580/19176] lr: 1.8740e-05 eta: 6:20:10 time: 1.3393 data_time: 0.0139 memory: 10923 loss: 0.2390 2025/03/23 18:54:06 - mmengine - INFO - Iter(train) [ 3590/19176] lr: 1.8732e-05 eta: 6:19:41 time: 1.1361 data_time: 0.0129 memory: 10467 loss: 0.2351 2025/03/23 18:54:15 - mmengine - INFO - Iter(train) [ 3600/19176] lr: 1.8724e-05 eta: 6:19:00 time: 0.8450 data_time: 0.0119 memory: 10051 loss: 0.3155 2025/03/23 18:54:35 - mmengine - INFO - Iter(train) [ 3610/19176] lr: 1.8716e-05 eta: 6:19:09 time: 2.0105 data_time: 0.0137 memory: 15665 loss: 0.2011 2025/03/23 18:54:52 - mmengine - INFO - Iter(train) [ 3620/19176] lr: 1.8707e-05 eta: 6:19:08 time: 1.7621 data_time: 0.0144 memory: 11993 loss: 0.1849 2025/03/23 18:55:09 - mmengine - INFO - Iter(train) [ 3630/19176] lr: 1.8699e-05 eta: 6:19:03 time: 1.6980 data_time: 0.0146 memory: 11720 loss: 0.1867 2025/03/23 18:55:26 - mmengine - INFO - Iter(train) [ 3640/19176] lr: 1.8691e-05 eta: 6:18:56 time: 1.6337 data_time: 0.0145 memory: 11600 loss: 0.2095 2025/03/23 18:55:41 - mmengine - INFO - Iter(train) [ 3650/19176] lr: 1.8682e-05 eta: 6:18:46 time: 1.5768 data_time: 0.0145 memory: 11347 loss: 0.2224 2025/03/23 18:55:56 - mmengine - INFO - Iter(train) [ 3660/19176] lr: 1.8674e-05 eta: 6:18:33 time: 1.5146 data_time: 0.0147 memory: 11183 loss: 0.2195 2025/03/23 18:56:11 - mmengine - INFO - Iter(train) [ 3670/19176] lr: 1.8665e-05 eta: 6:18:16 time: 1.4082 data_time: 0.0140 memory: 10965 loss: 0.2376 2025/03/23 18:56:22 - mmengine - INFO - Iter(train) [ 3680/19176] lr: 1.8657e-05 eta: 6:17:48 time: 1.1312 data_time: 0.0123 memory: 10549 loss: 0.2467 2025/03/23 18:56:31 - mmengine - INFO - Iter(train) [ 3690/19176] lr: 1.8649e-05 eta: 6:17:11 time: 0.9476 data_time: 0.0119 memory: 10030 loss: 0.2264 2025/03/23 18:56:39 - mmengine - INFO - Iter(train) [ 3700/19176] lr: 1.8640e-05 eta: 6:16:27 time: 0.7381 data_time: 0.0111 memory: 9552 loss: 0.2934 2025/03/23 18:56:59 - mmengine - INFO - Iter(train) [ 3710/19176] lr: 1.8632e-05 eta: 6:16:36 time: 2.0374 data_time: 0.0139 memory: 16526 loss: 0.1696 2025/03/23 18:57:16 - mmengine - INFO - Iter(train) [ 3720/19176] lr: 1.8623e-05 eta: 6:16:33 time: 1.7363 data_time: 0.0145 memory: 11960 loss: 0.1940 2025/03/23 18:57:33 - mmengine - INFO - Iter(train) [ 3730/19176] lr: 1.8614e-05 eta: 6:16:27 time: 1.6678 data_time: 0.0141 memory: 11537 loss: 0.2109 2025/03/23 18:57:49 - mmengine - INFO - Iter(train) [ 3740/19176] lr: 1.8606e-05 eta: 6:16:19 time: 1.6221 data_time: 0.0146 memory: 11422 loss: 0.2100 2025/03/23 18:58:05 - mmengine - INFO - Iter(train) [ 3750/19176] lr: 1.8597e-05 eta: 6:16:08 time: 1.5541 data_time: 0.0144 memory: 11316 loss: 0.2685 2025/03/23 18:58:20 - mmengine - INFO - Iter(train) [ 3760/19176] lr: 1.8589e-05 eta: 6:15:55 time: 1.5039 data_time: 0.0144 memory: 11192 loss: 0.2295 2025/03/23 18:58:34 - mmengine - INFO - Iter(train) [ 3770/19176] lr: 1.8580e-05 eta: 6:15:38 time: 1.4153 data_time: 0.0137 memory: 11028 loss: 0.2381 2025/03/23 18:58:47 - mmengine - INFO - Iter(train) [ 3780/19176] lr: 1.8571e-05 eta: 6:15:16 time: 1.2607 data_time: 0.0133 memory: 10765 loss: 0.2553 2025/03/23 18:58:57 - mmengine - INFO - Iter(train) [ 3790/19176] lr: 1.8563e-05 eta: 6:14:44 time: 1.0480 data_time: 0.0121 memory: 10200 loss: 0.2096 2025/03/23 18:59:05 - mmengine - INFO - Iter(train) [ 3800/19176] lr: 1.8554e-05 eta: 6:14:02 time: 0.7792 data_time: 0.0117 memory: 9840 loss: 0.2712 2025/03/23 18:59:25 - mmengine - INFO - Iter(train) [ 3810/19176] lr: 1.8545e-05 eta: 6:14:09 time: 1.9965 data_time: 0.0135 memory: 16168 loss: 0.2111 2025/03/23 18:59:42 - mmengine - INFO - Iter(train) [ 3820/19176] lr: 1.8536e-05 eta: 6:14:06 time: 1.7477 data_time: 0.0142 memory: 12056 loss: 0.1979 2025/03/23 18:59:59 - mmengine - INFO - Iter(train) [ 3830/19176] lr: 1.8527e-05 eta: 6:14:00 time: 1.6852 data_time: 0.0140 memory: 11722 loss: 0.2209 2025/03/23 19:00:16 - mmengine - INFO - Iter(train) [ 3840/19176] lr: 1.8519e-05 eta: 6:13:52 time: 1.6288 data_time: 0.0144 memory: 11424 loss: 0.2215 2025/03/23 19:00:31 - mmengine - INFO - Iter(train) [ 3850/19176] lr: 1.8510e-05 eta: 6:13:42 time: 1.5816 data_time: 0.0141 memory: 11314 loss: 0.2400 2025/03/23 19:00:47 - mmengine - INFO - Iter(train) [ 3860/19176] lr: 1.8501e-05 eta: 6:13:30 time: 1.5201 data_time: 0.0140 memory: 11237 loss: 0.2183 2025/03/23 19:01:02 - mmengine - INFO - Iter(train) [ 3870/19176] lr: 1.8492e-05 eta: 6:13:17 time: 1.5102 data_time: 0.0139 memory: 11275 loss: 0.2285 2025/03/23 19:01:16 - mmengine - INFO - Iter(train) [ 3880/19176] lr: 1.8483e-05 eta: 6:13:01 time: 1.4138 data_time: 0.0137 memory: 11031 loss: 0.2800 2025/03/23 19:01:27 - mmengine - INFO - Iter(train) [ 3890/19176] lr: 1.8474e-05 eta: 6:12:34 time: 1.1654 data_time: 0.0125 memory: 10519 loss: 0.2345 2025/03/23 19:01:36 - mmengine - INFO - Iter(train) [ 3900/19176] lr: 1.8465e-05 eta: 6:11:55 time: 0.8349 data_time: 0.0116 memory: 10096 loss: 0.2654 2025/03/23 19:01:55 - mmengine - INFO - Iter(train) [ 3910/19176] lr: 1.8456e-05 eta: 6:11:59 time: 1.9257 data_time: 0.0140 memory: 14154 loss: 0.2014 2025/03/23 19:02:12 - mmengine - INFO - Iter(train) [ 3920/19176] lr: 1.8447e-05 eta: 6:11:55 time: 1.7371 data_time: 0.0146 memory: 11970 loss: 0.2160 2025/03/23 19:02:29 - mmengine - INFO - Iter(train) [ 3930/19176] lr: 1.8438e-05 eta: 6:11:48 time: 1.6795 data_time: 0.0149 memory: 12105 loss: 0.1949 2025/03/23 19:02:45 - mmengine - INFO - Iter(train) [ 3940/19176] lr: 1.8429e-05 eta: 6:11:39 time: 1.5973 data_time: 0.0145 memory: 11409 loss: 0.1966 2025/03/23 19:03:01 - mmengine - INFO - Iter(train) [ 3950/19176] lr: 1.8420e-05 eta: 6:11:27 time: 1.5382 data_time: 0.0144 memory: 11272 loss: 0.2376 2025/03/23 19:03:15 - mmengine - INFO - Iter(train) [ 3960/19176] lr: 1.8411e-05 eta: 6:11:13 time: 1.4693 data_time: 0.0140 memory: 11111 loss: 0.2258 2025/03/23 19:03:29 - mmengine - INFO - Iter(train) [ 3970/19176] lr: 1.8402e-05 eta: 6:10:54 time: 1.3466 data_time: 0.0144 memory: 11002 loss: 0.2999 2025/03/23 19:03:40 - mmengine - INFO - Iter(train) [ 3980/19176] lr: 1.8392e-05 eta: 6:10:26 time: 1.1137 data_time: 0.0125 memory: 10365 loss: 0.2058 2025/03/23 19:03:50 - mmengine - INFO - Iter(train) [ 3990/19176] lr: 1.8383e-05 eta: 6:09:52 time: 0.9724 data_time: 0.0118 memory: 10162 loss: 0.2545 2025/03/23 19:03:57 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 19:03:57 - mmengine - INFO - Iter(train) [ 4000/19176] lr: 1.8374e-05 eta: 6:09:09 time: 0.7018 data_time: 0.0118 memory: 9494 loss: 0.2615 2025/03/23 19:03:57 - mmengine - INFO - Saving checkpoint at 4000 iterations 2025/03/23 19:04:18 - mmengine - INFO - Iter(train) [ 4010/19176] lr: 1.8365e-05 eta: 6:09:22 time: 2.1874 data_time: 0.0956 memory: 18264 loss: 0.2049 2025/03/23 19:04:36 - mmengine - INFO - Iter(train) [ 4020/19176] lr: 1.8355e-05 eta: 6:09:18 time: 1.7346 data_time: 0.0145 memory: 11979 loss: 0.2357 2025/03/23 19:04:52 - mmengine - INFO - Iter(train) [ 4030/19176] lr: 1.8346e-05 eta: 6:09:10 time: 1.6384 data_time: 0.0145 memory: 11548 loss: 0.2013 2025/03/23 19:05:08 - mmengine - INFO - Iter(train) [ 4040/19176] lr: 1.8337e-05 eta: 6:09:00 time: 1.5884 data_time: 0.0147 memory: 11359 loss: 0.2298 2025/03/23 19:05:23 - mmengine - INFO - Iter(train) [ 4050/19176] lr: 1.8328e-05 eta: 6:08:47 time: 1.5028 data_time: 0.0142 memory: 11204 loss: 0.2718 2025/03/23 19:05:38 - mmengine - INFO - Iter(train) [ 4060/19176] lr: 1.8318e-05 eta: 6:08:32 time: 1.4618 data_time: 0.0150 memory: 11107 loss: 0.2038 2025/03/23 19:05:52 - mmengine - INFO - Iter(train) [ 4070/19176] lr: 1.8309e-05 eta: 6:08:15 time: 1.3967 data_time: 0.0141 memory: 10915 loss: 0.2699 2025/03/23 19:06:04 - mmengine - INFO - Iter(train) [ 4080/19176] lr: 1.8299e-05 eta: 6:07:53 time: 1.2589 data_time: 0.0136 memory: 10709 loss: 0.2382 2025/03/23 19:06:15 - mmengine - INFO - Iter(train) [ 4090/19176] lr: 1.8290e-05 eta: 6:07:24 time: 1.0913 data_time: 0.0125 memory: 10342 loss: 0.1782 2025/03/23 19:06:24 - mmengine - INFO - Iter(train) [ 4100/19176] lr: 1.8280e-05 eta: 6:06:49 time: 0.8868 data_time: 0.0117 memory: 10024 loss: 0.2483 2025/03/23 19:06:44 - mmengine - INFO - Iter(train) [ 4110/19176] lr: 1.8271e-05 eta: 6:06:53 time: 1.9751 data_time: 0.0151 memory: 13459 loss: 0.1946 2025/03/23 19:07:01 - mmengine - INFO - Iter(train) [ 4120/19176] lr: 1.8261e-05 eta: 6:06:49 time: 1.7526 data_time: 0.0160 memory: 12115 loss: 0.1906 2025/03/23 19:07:18 - mmengine - INFO - Iter(train) [ 4130/19176] lr: 1.8252e-05 eta: 6:06:41 time: 1.6582 data_time: 0.0142 memory: 11595 loss: 0.1934 2025/03/23 19:07:34 - mmengine - INFO - Iter(train) [ 4140/19176] lr: 1.8242e-05 eta: 6:06:33 time: 1.6242 data_time: 0.0139 memory: 11407 loss: 0.2020 2025/03/23 19:07:50 - mmengine - INFO - Iter(train) [ 4150/19176] lr: 1.8233e-05 eta: 6:06:22 time: 1.5619 data_time: 0.0136 memory: 11291 loss: 0.2066 2025/03/23 19:08:05 - mmengine - INFO - Iter(train) [ 4160/19176] lr: 1.8223e-05 eta: 6:06:08 time: 1.4972 data_time: 0.0135 memory: 11147 loss: 0.2428 2025/03/23 19:08:19 - mmengine - INFO - Iter(train) [ 4170/19176] lr: 1.8214e-05 eta: 6:05:53 time: 1.4369 data_time: 0.0134 memory: 11032 loss: 0.2423 2025/03/23 19:08:32 - mmengine - INFO - Iter(train) [ 4180/19176] lr: 1.8204e-05 eta: 6:05:33 time: 1.3106 data_time: 0.0134 memory: 10876 loss: 0.2855 2025/03/23 19:08:43 - mmengine - INFO - Iter(train) [ 4190/19176] lr: 1.8194e-05 eta: 6:05:06 time: 1.1140 data_time: 0.0125 memory: 10338 loss: 0.2242 2025/03/23 19:08:52 - mmengine - INFO - Iter(train) [ 4200/19176] lr: 1.8185e-05 eta: 6:04:28 time: 0.8217 data_time: 0.0114 memory: 9933 loss: 0.2379 2025/03/23 19:09:10 - mmengine - INFO - Iter(train) [ 4210/19176] lr: 1.8175e-05 eta: 6:04:26 time: 1.8192 data_time: 0.0139 memory: 12495 loss: 0.2098 2025/03/23 19:09:27 - mmengine - INFO - Iter(train) [ 4220/19176] lr: 1.8165e-05 eta: 6:04:21 time: 1.7233 data_time: 0.0141 memory: 11826 loss: 0.2231 2025/03/23 19:09:44 - mmengine - INFO - Iter(train) [ 4230/19176] lr: 1.8155e-05 eta: 6:04:14 time: 1.6760 data_time: 0.0139 memory: 11621 loss: 0.2278 2025/03/23 19:10:00 - mmengine - INFO - Iter(train) [ 4240/19176] lr: 1.8146e-05 eta: 6:04:05 time: 1.6251 data_time: 0.0136 memory: 11429 loss: 0.1923 2025/03/23 19:10:16 - mmengine - INFO - Iter(train) [ 4250/19176] lr: 1.8136e-05 eta: 6:03:54 time: 1.5613 data_time: 0.0138 memory: 11369 loss: 0.2313 2025/03/23 19:10:30 - mmengine - INFO - Iter(train) [ 4260/19176] lr: 1.8126e-05 eta: 6:03:39 time: 1.4631 data_time: 0.0137 memory: 11133 loss: 0.1962 2025/03/23 19:10:44 - mmengine - INFO - Iter(train) [ 4270/19176] lr: 1.8116e-05 eta: 6:03:22 time: 1.3972 data_time: 0.0154 memory: 10987 loss: 0.2778 2025/03/23 19:10:56 - mmengine - INFO - Iter(train) [ 4280/19176] lr: 1.8106e-05 eta: 6:02:58 time: 1.1810 data_time: 0.0138 memory: 10520 loss: 0.2358 2025/03/23 19:11:07 - mmengine - INFO - Iter(train) [ 4290/19176] lr: 1.8096e-05 eta: 6:02:30 time: 1.0721 data_time: 0.0136 memory: 10339 loss: 0.2419 2025/03/23 19:11:15 - mmengine - INFO - Iter(train) [ 4300/19176] lr: 1.8086e-05 eta: 6:01:51 time: 0.7725 data_time: 0.0118 memory: 9943 loss: 0.2989 2025/03/23 19:11:34 - mmengine - INFO - Iter(train) [ 4310/19176] lr: 1.8076e-05 eta: 6:01:55 time: 1.9908 data_time: 0.0141 memory: 14462 loss: 0.1931 2025/03/23 19:11:52 - mmengine - INFO - Iter(train) [ 4320/19176] lr: 1.8066e-05 eta: 6:01:50 time: 1.7417 data_time: 0.0145 memory: 12105 loss: 0.2429 2025/03/23 19:12:09 - mmengine - INFO - Iter(train) [ 4330/19176] lr: 1.8056e-05 eta: 6:01:43 time: 1.6826 data_time: 0.0148 memory: 11740 loss: 0.2100 2025/03/23 19:12:25 - mmengine - INFO - Iter(train) [ 4340/19176] lr: 1.8046e-05 eta: 6:01:34 time: 1.6276 data_time: 0.0143 memory: 11488 loss: 0.2016 2025/03/23 19:12:41 - mmengine - INFO - Iter(train) [ 4350/19176] lr: 1.8036e-05 eta: 6:01:25 time: 1.6109 data_time: 0.0146 memory: 11399 loss: 0.2035 2025/03/23 19:12:57 - mmengine - INFO - Iter(train) [ 4360/19176] lr: 1.8026e-05 eta: 6:01:13 time: 1.5543 data_time: 0.0144 memory: 11278 loss: 0.2178 2025/03/23 19:13:11 - mmengine - INFO - Iter(train) [ 4370/19176] lr: 1.8016e-05 eta: 6:00:58 time: 1.4604 data_time: 0.0141 memory: 11120 loss: 0.2605 2025/03/23 19:13:24 - mmengine - INFO - Iter(train) [ 4380/19176] lr: 1.8006e-05 eta: 6:00:38 time: 1.2942 data_time: 0.0136 memory: 10901 loss: 0.2154 2025/03/23 19:13:34 - mmengine - INFO - Iter(train) [ 4390/19176] lr: 1.7996e-05 eta: 6:00:08 time: 1.0036 data_time: 0.0121 memory: 10257 loss: 0.3731 2025/03/23 19:13:41 - mmengine - INFO - Iter(train) [ 4400/19176] lr: 1.7986e-05 eta: 5:59:27 time: 0.6683 data_time: 0.0110 memory: 9620 loss: 0.2331 2025/03/23 19:14:02 - mmengine - INFO - Iter(train) [ 4410/19176] lr: 1.7976e-05 eta: 5:59:35 time: 2.1362 data_time: 0.0138 memory: 15963 loss: 0.2232 2025/03/23 19:14:20 - mmengine - INFO - Iter(train) [ 4420/19176] lr: 1.7966e-05 eta: 5:59:31 time: 1.7800 data_time: 0.0144 memory: 12286 loss: 0.2950 2025/03/23 19:14:37 - mmengine - INFO - Iter(train) [ 4430/19176] lr: 1.7955e-05 eta: 5:59:23 time: 1.6722 data_time: 0.0144 memory: 11631 loss: 0.1996 2025/03/23 19:14:53 - mmengine - INFO - Iter(train) [ 4440/19176] lr: 1.7945e-05 eta: 5:59:13 time: 1.6126 data_time: 0.0142 memory: 11395 loss: 0.2153 2025/03/23 19:15:08 - mmengine - INFO - Iter(train) [ 4450/19176] lr: 1.7935e-05 eta: 5:59:02 time: 1.5600 data_time: 0.0144 memory: 11292 loss: 0.2079 2025/03/23 19:15:23 - mmengine - INFO - Iter(train) [ 4460/19176] lr: 1.7924e-05 eta: 5:58:48 time: 1.4903 data_time: 0.0149 memory: 11136 loss: 0.1930 2025/03/23 19:15:37 - mmengine - INFO - Iter(train) [ 4470/19176] lr: 1.7914e-05 eta: 5:58:30 time: 1.3537 data_time: 0.0139 memory: 10942 loss: 0.2587 2025/03/23 19:15:48 - mmengine - INFO - Iter(train) [ 4480/19176] lr: 1.7904e-05 eta: 5:58:03 time: 1.0821 data_time: 0.0124 memory: 10329 loss: 0.2247 2025/03/23 19:15:57 - mmengine - INFO - Iter(train) [ 4490/19176] lr: 1.7893e-05 eta: 5:57:31 time: 0.9321 data_time: 0.0126 memory: 9965 loss: 0.2231 2025/03/23 19:16:05 - mmengine - INFO - Iter(train) [ 4500/19176] lr: 1.7883e-05 eta: 5:56:53 time: 0.7461 data_time: 0.0111 memory: 9548 loss: 0.3271 2025/03/23 19:16:23 - mmengine - INFO - Iter(train) [ 4510/19176] lr: 1.7873e-05 eta: 5:56:52 time: 1.8582 data_time: 0.0146 memory: 13802 loss: 0.2134 2025/03/23 19:16:40 - mmengine - INFO - Iter(train) [ 4520/19176] lr: 1.7862e-05 eta: 5:56:44 time: 1.6855 data_time: 0.0151 memory: 11684 loss: 0.1984 2025/03/23 19:16:56 - mmengine - INFO - Iter(train) [ 4530/19176] lr: 1.7852e-05 eta: 5:56:35 time: 1.6123 data_time: 0.0144 memory: 11470 loss: 0.1871 2025/03/23 19:17:12 - mmengine - INFO - Iter(train) [ 4540/19176] lr: 1.7841e-05 eta: 5:56:23 time: 1.5690 data_time: 0.0143 memory: 11310 loss: 0.1942 2025/03/23 19:17:27 - mmengine - INFO - Iter(train) [ 4550/19176] lr: 1.7831e-05 eta: 5:56:11 time: 1.5249 data_time: 0.0143 memory: 11194 loss: 0.2090 2025/03/23 19:17:42 - mmengine - INFO - Iter(train) [ 4560/19176] lr: 1.7820e-05 eta: 5:55:57 time: 1.4832 data_time: 0.0143 memory: 11089 loss: 0.2113 2025/03/23 19:17:56 - mmengine - INFO - Iter(train) [ 4570/19176] lr: 1.7810e-05 eta: 5:55:40 time: 1.3980 data_time: 0.0141 memory: 10982 loss: 0.2497 2025/03/23 19:18:09 - mmengine - INFO - Iter(train) [ 4580/19176] lr: 1.7799e-05 eta: 5:55:20 time: 1.2763 data_time: 0.0135 memory: 10707 loss: 0.3205 2025/03/23 19:18:19 - mmengine - INFO - Iter(train) [ 4590/19176] lr: 1.7789e-05 eta: 5:54:52 time: 1.0558 data_time: 0.0117 memory: 10267 loss: 0.2071 2025/03/23 19:18:26 - mmengine - INFO - Iter(train) [ 4600/19176] lr: 1.7778e-05 eta: 5:54:15 time: 0.7273 data_time: 0.0116 memory: 9901 loss: 0.2930 2025/03/23 19:18:46 - mmengine - INFO - Iter(train) [ 4610/19176] lr: 1.7767e-05 eta: 5:54:17 time: 1.9957 data_time: 0.0137 memory: 15316 loss: 0.2042 2025/03/23 19:19:04 - mmengine - INFO - Iter(train) [ 4620/19176] lr: 1.7757e-05 eta: 5:54:11 time: 1.7475 data_time: 0.0143 memory: 12033 loss: 0.1657 2025/03/23 19:19:20 - mmengine - INFO - Iter(train) [ 4630/19176] lr: 1.7746e-05 eta: 5:54:03 time: 1.6527 data_time: 0.0143 memory: 11664 loss: 0.2690 2025/03/23 19:19:37 - mmengine - INFO - Iter(train) [ 4640/19176] lr: 1.7735e-05 eta: 5:53:53 time: 1.6107 data_time: 0.0143 memory: 11394 loss: 0.2220 2025/03/23 19:19:52 - mmengine - INFO - Iter(train) [ 4650/19176] lr: 1.7725e-05 eta: 5:53:42 time: 1.5678 data_time: 0.0136 memory: 11284 loss: 0.2325 2025/03/23 19:20:07 - mmengine - INFO - Iter(train) [ 4660/19176] lr: 1.7714e-05 eta: 5:53:29 time: 1.5097 data_time: 0.0147 memory: 11262 loss: 0.2135 2025/03/23 19:20:21 - mmengine - INFO - Iter(train) [ 4670/19176] lr: 1.7703e-05 eta: 5:53:13 time: 1.4204 data_time: 0.0142 memory: 11026 loss: 0.3062 2025/03/23 19:20:35 - mmengine - INFO - Iter(train) [ 4680/19176] lr: 1.7692e-05 eta: 5:52:54 time: 1.3194 data_time: 0.0130 memory: 10820 loss: 0.2463 2025/03/23 19:20:46 - mmengine - INFO - Iter(train) [ 4690/19176] lr: 1.7682e-05 eta: 5:52:30 time: 1.1511 data_time: 0.0130 memory: 10538 loss: 0.3547 2025/03/23 19:20:55 - mmengine - INFO - Iter(train) [ 4700/19176] lr: 1.7671e-05 eta: 5:51:56 time: 0.8579 data_time: 0.0115 memory: 10099 loss: 0.2625 2025/03/23 19:21:14 - mmengine - INFO - Iter(train) [ 4710/19176] lr: 1.7660e-05 eta: 5:51:55 time: 1.8938 data_time: 0.0139 memory: 13246 loss: 0.1985 2025/03/23 19:21:31 - mmengine - INFO - Iter(train) [ 4720/19176] lr: 1.7649e-05 eta: 5:51:48 time: 1.6880 data_time: 0.0144 memory: 11706 loss: 0.1858 2025/03/23 19:21:47 - mmengine - INFO - Iter(train) [ 4730/19176] lr: 1.7638e-05 eta: 5:51:38 time: 1.6314 data_time: 0.0145 memory: 11473 loss: 0.1925 2025/03/23 19:22:03 - mmengine - INFO - Iter(train) [ 4740/19176] lr: 1.7627e-05 eta: 5:51:27 time: 1.5798 data_time: 0.0151 memory: 11358 loss: 0.2217 2025/03/23 19:22:18 - mmengine - INFO - Iter(train) [ 4750/19176] lr: 1.7616e-05 eta: 5:51:15 time: 1.5368 data_time: 0.0150 memory: 11246 loss: 0.2008 2025/03/23 19:22:33 - mmengine - INFO - Iter(train) [ 4760/19176] lr: 1.7605e-05 eta: 5:51:00 time: 1.4495 data_time: 0.0147 memory: 11040 loss: 0.2601 2025/03/23 19:22:46 - mmengine - INFO - Iter(train) [ 4770/19176] lr: 1.7594e-05 eta: 5:50:43 time: 1.3738 data_time: 0.0136 memory: 10903 loss: 0.2510 2025/03/23 19:22:58 - mmengine - INFO - Iter(train) [ 4780/19176] lr: 1.7583e-05 eta: 5:50:20 time: 1.1957 data_time: 0.0131 memory: 10686 loss: 0.2218 2025/03/23 19:23:08 - mmengine - INFO - Iter(train) [ 4790/19176] lr: 1.7572e-05 eta: 5:49:51 time: 0.9724 data_time: 0.0126 memory: 10120 loss: 0.2411 2025/03/23 19:23:11 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 19:23:11 - mmengine - WARNING - Reach the end of the dataloader, it will be restarted and continue to iterate. It is recommended to use `mmengine.dataset.InfiniteSampler` to enable the dataloader to iterate infinitely. 2025/03/23 19:23:26 - mmengine - INFO - Iter(train) [ 4800/19176] lr: 1.7561e-05 eta: 5:49:47 time: 1.8156 data_time: 0.2623 memory: 19198 loss: 0.2167 2025/03/23 19:23:43 - mmengine - INFO - Iter(train) [ 4810/19176] lr: 1.7550e-05 eta: 5:49:41 time: 1.7318 data_time: 0.0150 memory: 11944 loss: 0.1838 2025/03/23 19:24:00 - mmengine - INFO - Iter(train) [ 4820/19176] lr: 1.7539e-05 eta: 5:49:32 time: 1.6727 data_time: 0.0154 memory: 11579 loss: 0.1779 2025/03/23 19:24:16 - mmengine - INFO - Iter(train) [ 4830/19176] lr: 1.7528e-05 eta: 5:49:22 time: 1.6211 data_time: 0.0151 memory: 11447 loss: 0.1810 2025/03/23 19:24:32 - mmengine - INFO - Iter(train) [ 4840/19176] lr: 1.7517e-05 eta: 5:49:11 time: 1.5721 data_time: 0.0149 memory: 11314 loss: 0.1752 2025/03/23 19:24:47 - mmengine - INFO - Iter(train) [ 4850/19176] lr: 1.7506e-05 eta: 5:48:58 time: 1.5256 data_time: 0.0159 memory: 11185 loss: 0.2007 2025/03/23 19:25:02 - mmengine - INFO - Iter(train) [ 4860/19176] lr: 1.7495e-05 eta: 5:48:43 time: 1.4361 data_time: 0.0149 memory: 11066 loss: 0.2157 2025/03/23 19:25:15 - mmengine - INFO - Iter(train) [ 4870/19176] lr: 1.7483e-05 eta: 5:48:23 time: 1.2805 data_time: 0.0140 memory: 10810 loss: 0.2200 2025/03/23 19:25:25 - mmengine - INFO - Iter(train) [ 4880/19176] lr: 1.7472e-05 eta: 5:47:57 time: 1.0666 data_time: 0.0123 memory: 10291 loss: 0.2191 2025/03/23 19:25:34 - mmengine - INFO - Iter(train) [ 4890/19176] lr: 1.7461e-05 eta: 5:47:26 time: 0.8916 data_time: 0.0122 memory: 9952 loss: 0.2155 2025/03/23 19:25:49 - mmengine - INFO - Iter(train) [ 4900/19176] lr: 1.7450e-05 eta: 5:47:11 time: 1.4473 data_time: 0.0133 memory: 13422 loss: 0.2026 2025/03/23 19:26:06 - mmengine - INFO - Iter(train) [ 4910/19176] lr: 1.7438e-05 eta: 5:47:06 time: 1.7848 data_time: 0.0146 memory: 12157 loss: 0.1745 2025/03/23 19:26:24 - mmengine - INFO - Iter(train) [ 4920/19176] lr: 1.7427e-05 eta: 5:46:59 time: 1.7228 data_time: 0.0149 memory: 11852 loss: 0.1621 2025/03/23 19:26:40 - mmengine - INFO - Iter(train) [ 4930/19176] lr: 1.7416e-05 eta: 5:46:49 time: 1.6487 data_time: 0.0155 memory: 11582 loss: 0.1975 2025/03/23 19:26:56 - mmengine - INFO - Iter(train) [ 4940/19176] lr: 1.7405e-05 eta: 5:46:39 time: 1.6031 data_time: 0.0152 memory: 11386 loss: 0.2175 2025/03/23 19:27:12 - mmengine - INFO - Iter(train) [ 4950/19176] lr: 1.7393e-05 eta: 5:46:27 time: 1.5539 data_time: 0.0151 memory: 11292 loss: 0.2022 2025/03/23 19:27:26 - mmengine - INFO - Iter(train) [ 4960/19176] lr: 1.7382e-05 eta: 5:46:12 time: 1.4516 data_time: 0.0144 memory: 11136 loss: 0.2118 2025/03/23 19:27:39 - mmengine - INFO - Iter(train) [ 4970/19176] lr: 1.7370e-05 eta: 5:45:52 time: 1.2631 data_time: 0.0133 memory: 10707 loss: 0.2878 2025/03/23 19:27:50 - mmengine - INFO - Iter(train) [ 4980/19176] lr: 1.7359e-05 eta: 5:45:27 time: 1.1153 data_time: 0.0124 memory: 10377 loss: 0.2091 2025/03/23 19:28:00 - mmengine - INFO - Iter(train) [ 4990/19176] lr: 1.7348e-05 eta: 5:44:58 time: 0.9531 data_time: 0.0118 memory: 10090 loss: 0.2201 2025/03/23 19:28:13 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 19:28:13 - mmengine - INFO - Iter(train) [ 5000/19176] lr: 1.7336e-05 eta: 5:44:41 time: 1.3755 data_time: 0.0116 memory: 14467 loss: 0.1865 2025/03/23 19:28:13 - mmengine - INFO - Saving checkpoint at 5000 iterations 2025/03/23 19:28:32 - mmengine - INFO - Iter(train) [ 5010/19176] lr: 1.7325e-05 eta: 5:44:38 time: 1.8368 data_time: 0.0938 memory: 12065 loss: 0.1905 2025/03/23 19:28:48 - mmengine - INFO - Iter(train) [ 5020/19176] lr: 1.7313e-05 eta: 5:44:29 time: 1.6721 data_time: 0.0138 memory: 11653 loss: 0.1752 2025/03/23 19:29:05 - mmengine - INFO - Iter(train) [ 5030/19176] lr: 1.7302e-05 eta: 5:44:19 time: 1.6150 data_time: 0.0139 memory: 11457 loss: 0.2068 2025/03/23 19:29:21 - mmengine - INFO - Iter(train) [ 5040/19176] lr: 1.7290e-05 eta: 5:44:08 time: 1.5963 data_time: 0.0138 memory: 11367 loss: 0.2008 2025/03/23 19:29:36 - mmengine - INFO - Iter(train) [ 5050/19176] lr: 1.7278e-05 eta: 5:43:55 time: 1.5127 data_time: 0.0135 memory: 11230 loss: 0.1831 2025/03/23 19:29:50 - mmengine - INFO - Iter(train) [ 5060/19176] lr: 1.7267e-05 eta: 5:43:41 time: 1.4781 data_time: 0.0141 memory: 11146 loss: 0.2065 2025/03/23 19:30:04 - mmengine - INFO - Iter(train) [ 5070/19176] lr: 1.7255e-05 eta: 5:43:24 time: 1.4019 data_time: 0.0136 memory: 11011 loss: 0.1954 2025/03/23 19:30:16 - mmengine - INFO - Iter(train) [ 5080/19176] lr: 1.7244e-05 eta: 5:43:02 time: 1.1966 data_time: 0.0127 memory: 10597 loss: 0.2020 2025/03/23 19:30:26 - mmengine - INFO - Iter(train) [ 5090/19176] lr: 1.7232e-05 eta: 5:42:35 time: 1.0046 data_time: 0.0117 memory: 10230 loss: 0.2015 2025/03/23 19:30:41 - mmengine - INFO - Iter(train) [ 5100/19176] lr: 1.7220e-05 eta: 5:42:21 time: 1.4640 data_time: 0.0121 memory: 13641 loss: 0.2125 2025/03/23 19:30:58 - mmengine - INFO - Iter(train) [ 5110/19176] lr: 1.7209e-05 eta: 5:42:14 time: 1.7285 data_time: 0.0140 memory: 11860 loss: 0.1807 2025/03/23 19:31:15 - mmengine - INFO - Iter(train) [ 5120/19176] lr: 1.7197e-05 eta: 5:42:04 time: 1.6554 data_time: 0.0143 memory: 11640 loss: 0.1735 2025/03/23 19:31:31 - mmengine - INFO - Iter(train) [ 5130/19176] lr: 1.7185e-05 eta: 5:41:54 time: 1.6092 data_time: 0.0150 memory: 11493 loss: 0.1812 2025/03/23 19:31:46 - mmengine - INFO - Iter(train) [ 5140/19176] lr: 1.7173e-05 eta: 5:41:41 time: 1.5390 data_time: 0.0148 memory: 11271 loss: 0.1952 2025/03/23 19:32:01 - mmengine - INFO - Iter(train) [ 5150/19176] lr: 1.7162e-05 eta: 5:41:27 time: 1.4858 data_time: 0.0150 memory: 11169 loss: 0.1870 2025/03/23 19:32:15 - mmengine - INFO - Iter(train) [ 5160/19176] lr: 1.7150e-05 eta: 5:41:11 time: 1.3995 data_time: 0.0148 memory: 10978 loss: 0.1812 2025/03/23 19:32:28 - mmengine - INFO - Iter(train) [ 5170/19176] lr: 1.7138e-05 eta: 5:40:50 time: 1.2289 data_time: 0.0137 memory: 10686 loss: 0.2556 2025/03/23 19:32:38 - mmengine - INFO - Iter(train) [ 5180/19176] lr: 1.7126e-05 eta: 5:40:25 time: 1.0603 data_time: 0.0128 memory: 10210 loss: 0.2093 2025/03/23 19:32:47 - mmengine - INFO - Iter(train) [ 5190/19176] lr: 1.7114e-05 eta: 5:39:54 time: 0.8582 data_time: 0.0124 memory: 9952 loss: 0.2184 2025/03/23 19:33:00 - mmengine - INFO - Iter(train) [ 5200/19176] lr: 1.7102e-05 eta: 5:39:37 time: 1.3686 data_time: 0.0123 memory: 15756 loss: 0.2223 2025/03/23 19:33:18 - mmengine - INFO - Iter(train) [ 5210/19176] lr: 1.7090e-05 eta: 5:39:30 time: 1.7442 data_time: 0.0139 memory: 11944 loss: 0.1822 2025/03/23 19:33:35 - mmengine - INFO - Iter(train) [ 5220/19176] lr: 1.7079e-05 eta: 5:39:22 time: 1.6938 data_time: 0.0141 memory: 11679 loss: 0.1926 2025/03/23 19:33:51 - mmengine - INFO - Iter(train) [ 5230/19176] lr: 1.7067e-05 eta: 5:39:12 time: 1.6440 data_time: 0.0143 memory: 11493 loss: 0.1676 2025/03/23 19:34:07 - mmengine - INFO - Iter(train) [ 5240/19176] lr: 1.7055e-05 eta: 5:39:00 time: 1.5638 data_time: 0.0140 memory: 11332 loss: 0.1725 2025/03/23 19:34:22 - mmengine - INFO - Iter(train) [ 5250/19176] lr: 1.7043e-05 eta: 5:38:48 time: 1.5387 data_time: 0.0144 memory: 11208 loss: 0.2011 2025/03/23 19:34:37 - mmengine - INFO - Iter(train) [ 5260/19176] lr: 1.7031e-05 eta: 5:38:33 time: 1.4652 data_time: 0.0143 memory: 11040 loss: 0.1768 2025/03/23 19:34:51 - mmengine - INFO - Iter(train) [ 5270/19176] lr: 1.7019e-05 eta: 5:38:18 time: 1.4290 data_time: 0.0138 memory: 11027 loss: 0.1981 2025/03/23 19:35:03 - mmengine - INFO - Iter(train) [ 5280/19176] lr: 1.7007e-05 eta: 5:37:56 time: 1.1944 data_time: 0.0124 memory: 10883 loss: 0.2040 2025/03/23 19:35:13 - mmengine - INFO - Iter(train) [ 5290/19176] lr: 1.6995e-05 eta: 5:37:29 time: 0.9874 data_time: 0.0124 memory: 10095 loss: 0.1914 2025/03/23 19:35:28 - mmengine - INFO - Iter(train) [ 5300/19176] lr: 1.6982e-05 eta: 5:37:15 time: 1.4632 data_time: 0.0135 memory: 13361 loss: 0.1786 2025/03/23 19:35:45 - mmengine - INFO - Iter(train) [ 5310/19176] lr: 1.6970e-05 eta: 5:37:09 time: 1.7788 data_time: 0.0148 memory: 12079 loss: 0.1646 2025/03/23 19:36:02 - mmengine - INFO - Iter(train) [ 5320/19176] lr: 1.6958e-05 eta: 5:37:00 time: 1.6903 data_time: 0.0156 memory: 11693 loss: 0.1732 2025/03/23 19:36:19 - mmengine - INFO - Iter(train) [ 5330/19176] lr: 1.6946e-05 eta: 5:36:50 time: 1.6168 data_time: 0.0145 memory: 11533 loss: 0.2255 2025/03/23 19:36:34 - mmengine - INFO - Iter(train) [ 5340/19176] lr: 1.6934e-05 eta: 5:36:38 time: 1.5614 data_time: 0.0148 memory: 11291 loss: 0.1824 2025/03/23 19:36:49 - mmengine - INFO - Iter(train) [ 5350/19176] lr: 1.6922e-05 eta: 5:36:24 time: 1.4841 data_time: 0.0148 memory: 11134 loss: 0.1819 2025/03/23 19:37:03 - mmengine - INFO - Iter(train) [ 5360/19176] lr: 1.6910e-05 eta: 5:36:08 time: 1.4292 data_time: 0.0147 memory: 10981 loss: 0.1900 2025/03/23 19:37:16 - mmengine - INFO - Iter(train) [ 5370/19176] lr: 1.6897e-05 eta: 5:35:50 time: 1.3214 data_time: 0.0143 memory: 10830 loss: 0.2175 2025/03/23 19:37:28 - mmengine - INFO - Iter(train) [ 5380/19176] lr: 1.6885e-05 eta: 5:35:27 time: 1.1123 data_time: 0.0123 memory: 10351 loss: 0.1772 2025/03/23 19:37:37 - mmengine - INFO - Iter(train) [ 5390/19176] lr: 1.6873e-05 eta: 5:35:00 time: 0.9697 data_time: 0.0120 memory: 10111 loss: 0.2070 2025/03/23 19:37:51 - mmengine - INFO - Iter(train) [ 5400/19176] lr: 1.6861e-05 eta: 5:34:44 time: 1.4118 data_time: 0.0118 memory: 13555 loss: 0.2064 2025/03/23 19:38:09 - mmengine - INFO - Iter(train) [ 5410/19176] lr: 1.6848e-05 eta: 5:34:37 time: 1.7559 data_time: 0.0144 memory: 12042 loss: 0.1669 2025/03/23 19:38:26 - mmengine - INFO - Iter(train) [ 5420/19176] lr: 1.6836e-05 eta: 5:34:28 time: 1.6932 data_time: 0.0144 memory: 11659 loss: 0.1950 2025/03/23 19:38:42 - mmengine - INFO - Iter(train) [ 5430/19176] lr: 1.6824e-05 eta: 5:34:18 time: 1.6434 data_time: 0.0140 memory: 11499 loss: 0.1859 2025/03/23 19:38:58 - mmengine - INFO - Iter(train) [ 5440/19176] lr: 1.6811e-05 eta: 5:34:07 time: 1.5990 data_time: 0.0136 memory: 11386 loss: 0.1961 2025/03/23 19:39:14 - mmengine - INFO - Iter(train) [ 5450/19176] lr: 1.6799e-05 eta: 5:33:54 time: 1.5240 data_time: 0.0139 memory: 11277 loss: 0.1831 2025/03/23 19:39:28 - mmengine - INFO - Iter(train) [ 5460/19176] lr: 1.6786e-05 eta: 5:33:39 time: 1.4285 data_time: 0.0141 memory: 11082 loss: 0.1943 2025/03/23 19:39:41 - mmengine - INFO - Iter(train) [ 5470/19176] lr: 1.6774e-05 eta: 5:33:21 time: 1.3354 data_time: 0.0133 memory: 10874 loss: 0.1796 2025/03/23 19:39:53 - mmengine - INFO - Iter(train) [ 5480/19176] lr: 1.6762e-05 eta: 5:33:00 time: 1.1732 data_time: 0.0124 memory: 10565 loss: 0.1902 2025/03/23 19:40:03 - mmengine - INFO - Iter(train) [ 5490/19176] lr: 1.6749e-05 eta: 5:32:33 time: 0.9646 data_time: 0.0122 memory: 10137 loss: 0.2529 2025/03/23 19:40:18 - mmengine - INFO - Iter(train) [ 5500/19176] lr: 1.6737e-05 eta: 5:32:19 time: 1.5082 data_time: 0.0123 memory: 15096 loss: 0.2014 2025/03/23 19:40:35 - mmengine - INFO - Iter(train) [ 5510/19176] lr: 1.6724e-05 eta: 5:32:12 time: 1.7574 data_time: 0.0135 memory: 12181 loss: 0.1615 2025/03/23 19:40:52 - mmengine - INFO - Iter(train) [ 5520/19176] lr: 1.6712e-05 eta: 5:32:03 time: 1.6754 data_time: 0.0144 memory: 11675 loss: 0.1900 2025/03/23 19:41:09 - mmengine - INFO - Iter(train) [ 5530/19176] lr: 1.6699e-05 eta: 5:31:53 time: 1.6509 data_time: 0.0151 memory: 11691 loss: 0.1712 2025/03/23 19:41:24 - mmengine - INFO - Iter(train) [ 5540/19176] lr: 1.6687e-05 eta: 5:31:41 time: 1.5558 data_time: 0.0146 memory: 11359 loss: 0.1924 2025/03/23 19:41:39 - mmengine - INFO - Iter(train) [ 5550/19176] lr: 1.6674e-05 eta: 5:31:28 time: 1.5144 data_time: 0.0152 memory: 11192 loss: 0.2121 2025/03/23 19:41:54 - mmengine - INFO - Iter(train) [ 5560/19176] lr: 1.6661e-05 eta: 5:31:13 time: 1.4798 data_time: 0.0148 memory: 11111 loss: 0.1861 2025/03/23 19:42:08 - mmengine - INFO - Iter(train) [ 5570/19176] lr: 1.6649e-05 eta: 5:30:57 time: 1.3942 data_time: 0.0148 memory: 10974 loss: 0.1905 2025/03/23 19:42:19 - mmengine - INFO - Iter(train) [ 5580/19176] lr: 1.6636e-05 eta: 5:30:35 time: 1.1516 data_time: 0.0132 memory: 10565 loss: 0.1906 2025/03/23 19:42:29 - mmengine - INFO - Iter(train) [ 5590/19176] lr: 1.6624e-05 eta: 5:30:08 time: 0.9522 data_time: 0.0124 memory: 10119 loss: 0.2138 2025/03/23 19:42:45 - mmengine - INFO - Iter(train) [ 5600/19176] lr: 1.6611e-05 eta: 5:29:58 time: 1.6257 data_time: 0.0124 memory: 18041 loss: 0.1990 2025/03/23 19:43:03 - mmengine - INFO - Iter(train) [ 5610/19176] lr: 1.6598e-05 eta: 5:29:51 time: 1.7898 data_time: 0.0147 memory: 12360 loss: 0.2258 2025/03/23 19:43:20 - mmengine - INFO - Iter(train) [ 5620/19176] lr: 1.6586e-05 eta: 5:29:42 time: 1.6842 data_time: 0.0146 memory: 11697 loss: 0.2015 2025/03/23 19:43:36 - mmengine - INFO - Iter(train) [ 5630/19176] lr: 1.6573e-05 eta: 5:29:32 time: 1.6390 data_time: 0.0148 memory: 11467 loss: 0.1831 2025/03/23 19:43:52 - mmengine - INFO - Iter(train) [ 5640/19176] lr: 1.6560e-05 eta: 5:29:21 time: 1.6038 data_time: 0.0150 memory: 11371 loss: 0.1721 2025/03/23 19:44:08 - mmengine - INFO - Iter(train) [ 5650/19176] lr: 1.6547e-05 eta: 5:29:08 time: 1.5454 data_time: 0.0147 memory: 11329 loss: 0.2177 2025/03/23 19:44:23 - mmengine - INFO - Iter(train) [ 5660/19176] lr: 1.6535e-05 eta: 5:28:54 time: 1.4612 data_time: 0.0154 memory: 11179 loss: 0.2139 2025/03/23 19:44:35 - mmengine - INFO - Iter(train) [ 5670/19176] lr: 1.6522e-05 eta: 5:28:34 time: 1.2356 data_time: 0.0136 memory: 10742 loss: 0.2354 2025/03/23 19:44:45 - mmengine - INFO - Iter(train) [ 5680/19176] lr: 1.6509e-05 eta: 5:28:09 time: 1.0308 data_time: 0.0130 memory: 10217 loss: 0.2126 2025/03/23 19:44:53 - mmengine - INFO - Iter(train) [ 5690/19176] lr: 1.6496e-05 eta: 5:27:38 time: 0.7919 data_time: 0.0121 memory: 9572 loss: 0.1901 2025/03/23 19:45:07 - mmengine - INFO - Iter(train) [ 5700/19176] lr: 1.6483e-05 eta: 5:27:22 time: 1.3931 data_time: 0.0134 memory: 13228 loss: 0.1721 2025/03/23 19:45:25 - mmengine - INFO - Iter(train) [ 5710/19176] lr: 1.6470e-05 eta: 5:27:15 time: 1.7662 data_time: 0.0154 memory: 12171 loss: 0.1747 2025/03/23 19:45:42 - mmengine - INFO - Iter(train) [ 5720/19176] lr: 1.6458e-05 eta: 5:27:06 time: 1.7021 data_time: 0.0151 memory: 11756 loss: 0.1914 2025/03/23 19:45:58 - mmengine - INFO - Iter(train) [ 5730/19176] lr: 1.6445e-05 eta: 5:26:56 time: 1.6317 data_time: 0.0143 memory: 11463 loss: 0.1825 2025/03/23 19:46:14 - mmengine - INFO - Iter(train) [ 5740/19176] lr: 1.6432e-05 eta: 5:26:43 time: 1.5623 data_time: 0.0145 memory: 11281 loss: 0.1997 2025/03/23 19:46:29 - mmengine - INFO - Iter(train) [ 5750/19176] lr: 1.6419e-05 eta: 5:26:31 time: 1.5398 data_time: 0.0143 memory: 11456 loss: 0.1848 2025/03/23 19:46:44 - mmengine - INFO - Iter(train) [ 5760/19176] lr: 1.6406e-05 eta: 5:26:16 time: 1.4586 data_time: 0.0143 memory: 11086 loss: 0.2088 2025/03/23 19:46:57 - mmengine - INFO - Iter(train) [ 5770/19176] lr: 1.6393e-05 eta: 5:25:59 time: 1.3640 data_time: 0.0139 memory: 10930 loss: 0.2159 2025/03/23 19:47:08 - mmengine - INFO - Iter(train) [ 5780/19176] lr: 1.6380e-05 eta: 5:25:37 time: 1.1066 data_time: 0.0129 memory: 10429 loss: 0.2801 2025/03/23 19:47:17 - mmengine - INFO - Iter(train) [ 5790/19176] lr: 1.6367e-05 eta: 5:25:08 time: 0.8590 data_time: 0.0117 memory: 9895 loss: 0.1839 2025/03/23 19:47:31 - mmengine - INFO - Iter(train) [ 5800/19176] lr: 1.6354e-05 eta: 5:24:51 time: 1.3590 data_time: 0.0127 memory: 13344 loss: 0.1739 2025/03/23 19:47:48 - mmengine - INFO - Iter(train) [ 5810/19176] lr: 1.6341e-05 eta: 5:24:44 time: 1.7813 data_time: 0.0143 memory: 12054 loss: 0.1732 2025/03/23 19:48:05 - mmengine - INFO - Iter(train) [ 5820/19176] lr: 1.6328e-05 eta: 5:24:35 time: 1.6883 data_time: 0.0146 memory: 11708 loss: 0.1854 2025/03/23 19:48:22 - mmengine - INFO - Iter(train) [ 5830/19176] lr: 1.6315e-05 eta: 5:24:24 time: 1.6348 data_time: 0.0145 memory: 11450 loss: 0.1642 2025/03/23 19:48:38 - mmengine - INFO - Iter(train) [ 5840/19176] lr: 1.6301e-05 eta: 5:24:13 time: 1.5953 data_time: 0.0143 memory: 11389 loss: 0.1798 2025/03/23 19:48:53 - mmengine - INFO - Iter(train) [ 5850/19176] lr: 1.6288e-05 eta: 5:24:00 time: 1.5465 data_time: 0.0147 memory: 11259 loss: 0.1814 2025/03/23 19:49:08 - mmengine - INFO - Iter(train) [ 5860/19176] lr: 1.6275e-05 eta: 5:23:46 time: 1.4807 data_time: 0.0145 memory: 11067 loss: 0.1999 2025/03/23 19:49:21 - mmengine - INFO - Iter(train) [ 5870/19176] lr: 1.6262e-05 eta: 5:23:28 time: 1.2812 data_time: 0.0142 memory: 10815 loss: 0.4131 2025/03/23 19:49:32 - mmengine - INFO - Iter(train) [ 5880/19176] lr: 1.6249e-05 eta: 5:23:05 time: 1.0977 data_time: 0.0124 memory: 10358 loss: 0.2096 2025/03/23 19:49:41 - mmengine - INFO - Iter(train) [ 5890/19176] lr: 1.6236e-05 eta: 5:22:39 time: 0.9442 data_time: 0.0121 memory: 9992 loss: 0.2467 2025/03/23 19:49:57 - mmengine - INFO - Iter(train) [ 5900/19176] lr: 1.6222e-05 eta: 5:22:27 time: 1.5883 data_time: 0.0126 memory: 15218 loss: 0.2323 2025/03/23 19:50:15 - mmengine - INFO - Iter(train) [ 5910/19176] lr: 1.6209e-05 eta: 5:22:20 time: 1.7877 data_time: 0.0142 memory: 12291 loss: 0.1697 2025/03/23 19:50:32 - mmengine - INFO - Iter(train) [ 5920/19176] lr: 1.6196e-05 eta: 5:22:12 time: 1.7350 data_time: 0.0145 memory: 12047 loss: 0.1868 2025/03/23 19:50:49 - mmengine - INFO - Iter(train) [ 5930/19176] lr: 1.6183e-05 eta: 5:22:01 time: 1.6452 data_time: 0.0143 memory: 11472 loss: 0.1845 2025/03/23 19:51:04 - mmengine - INFO - Iter(train) [ 5940/19176] lr: 1.6169e-05 eta: 5:21:49 time: 1.5889 data_time: 0.0142 memory: 11351 loss: 0.2111 2025/03/23 19:51:20 - mmengine - INFO - Iter(train) [ 5950/19176] lr: 1.6156e-05 eta: 5:21:36 time: 1.5309 data_time: 0.0143 memory: 11219 loss: 0.2211 2025/03/23 19:51:34 - mmengine - INFO - Iter(train) [ 5960/19176] lr: 1.6143e-05 eta: 5:21:21 time: 1.4421 data_time: 0.0141 memory: 11052 loss: 0.1939 2025/03/23 19:51:48 - mmengine - INFO - Iter(train) [ 5970/19176] lr: 1.6129e-05 eta: 5:21:04 time: 1.3328 data_time: 0.0137 memory: 10848 loss: 0.2585 2025/03/23 19:51:59 - mmengine - INFO - Iter(train) [ 5980/19176] lr: 1.6116e-05 eta: 5:20:43 time: 1.1441 data_time: 0.0127 memory: 10470 loss: 0.1704 2025/03/23 19:52:08 - mmengine - INFO - Iter(train) [ 5990/19176] lr: 1.6103e-05 eta: 5:20:16 time: 0.9272 data_time: 0.0120 memory: 10047 loss: 0.1946 2025/03/23 19:52:24 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 19:52:24 - mmengine - INFO - Iter(train) [ 6000/19176] lr: 1.6089e-05 eta: 5:20:05 time: 1.6061 data_time: 0.0124 memory: 18682 loss: 0.1844 2025/03/23 19:52:24 - mmengine - INFO - Saving checkpoint at 6000 iterations 2025/03/23 19:52:43 - mmengine - INFO - Iter(train) [ 6010/19176] lr: 1.6076e-05 eta: 5:19:59 time: 1.8350 data_time: 0.0949 memory: 12206 loss: 0.1829 2025/03/23 19:52:59 - mmengine - INFO - Iter(train) [ 6020/19176] lr: 1.6063e-05 eta: 5:19:49 time: 1.6643 data_time: 0.0147 memory: 11600 loss: 0.1844 2025/03/23 19:53:16 - mmengine - INFO - Iter(train) [ 6030/19176] lr: 1.6049e-05 eta: 5:19:38 time: 1.6307 data_time: 0.0146 memory: 11555 loss: 0.1728 2025/03/23 19:53:31 - mmengine - INFO - Iter(train) [ 6040/19176] lr: 1.6036e-05 eta: 5:19:26 time: 1.5753 data_time: 0.0144 memory: 11317 loss: 0.2215 2025/03/23 19:53:46 - mmengine - INFO - Iter(train) [ 6050/19176] lr: 1.6022e-05 eta: 5:19:12 time: 1.5109 data_time: 0.0144 memory: 11255 loss: 0.2070 2025/03/23 19:54:01 - mmengine - INFO - Iter(train) [ 6060/19176] lr: 1.6009e-05 eta: 5:18:58 time: 1.4607 data_time: 0.0146 memory: 11076 loss: 0.1874 2025/03/23 19:54:15 - mmengine - INFO - Iter(train) [ 6070/19176] lr: 1.5995e-05 eta: 5:18:41 time: 1.3658 data_time: 0.0145 memory: 10866 loss: 0.2463 2025/03/23 19:54:27 - mmengine - INFO - Iter(train) [ 6080/19176] lr: 1.5982e-05 eta: 5:18:21 time: 1.1974 data_time: 0.0130 memory: 10597 loss: 0.1656 2025/03/23 19:54:36 - mmengine - INFO - Iter(train) [ 6090/19176] lr: 1.5968e-05 eta: 5:17:55 time: 0.9361 data_time: 0.0125 memory: 10036 loss: 0.2143 2025/03/23 19:54:52 - mmengine - INFO - Iter(train) [ 6100/19176] lr: 1.5955e-05 eta: 5:17:43 time: 1.5708 data_time: 0.0129 memory: 14863 loss: 0.1976 2025/03/23 19:55:09 - mmengine - INFO - Iter(train) [ 6110/19176] lr: 1.5941e-05 eta: 5:17:35 time: 1.7546 data_time: 0.0149 memory: 12042 loss: 0.1579 2025/03/23 19:55:26 - mmengine - INFO - Iter(train) [ 6120/19176] lr: 1.5927e-05 eta: 5:17:25 time: 1.7027 data_time: 0.0147 memory: 11710 loss: 0.2303 2025/03/23 19:55:43 - mmengine - INFO - Iter(train) [ 6130/19176] lr: 1.5914e-05 eta: 5:17:15 time: 1.6452 data_time: 0.0144 memory: 11490 loss: 0.1755 2025/03/23 19:55:59 - mmengine - INFO - Iter(train) [ 6140/19176] lr: 1.5900e-05 eta: 5:17:03 time: 1.5989 data_time: 0.0146 memory: 11820 loss: 0.1840 2025/03/23 19:56:14 - mmengine - INFO - Iter(train) [ 6150/19176] lr: 1.5886e-05 eta: 5:16:49 time: 1.4997 data_time: 0.0152 memory: 11143 loss: 0.1595 2025/03/23 19:56:28 - mmengine - INFO - Iter(train) [ 6160/19176] lr: 1.5873e-05 eta: 5:16:34 time: 1.4419 data_time: 0.0144 memory: 11089 loss: 0.2016 2025/03/23 19:56:42 - mmengine - INFO - Iter(train) [ 6170/19176] lr: 1.5859e-05 eta: 5:16:17 time: 1.3354 data_time: 0.0140 memory: 10825 loss: 0.2015 2025/03/23 19:56:53 - mmengine - INFO - Iter(train) [ 6180/19176] lr: 1.5845e-05 eta: 5:15:56 time: 1.1461 data_time: 0.0133 memory: 10552 loss: 0.2303 2025/03/23 19:57:03 - mmengine - INFO - Iter(train) [ 6190/19176] lr: 1.5832e-05 eta: 5:15:32 time: 1.0109 data_time: 0.0121 memory: 10208 loss: 0.2120 2025/03/23 19:57:17 - mmengine - INFO - Iter(train) [ 6200/19176] lr: 1.5818e-05 eta: 5:15:16 time: 1.4054 data_time: 0.0127 memory: 13877 loss: 0.2080 2025/03/23 19:57:35 - mmengine - INFO - Iter(train) [ 6210/19176] lr: 1.5804e-05 eta: 5:15:08 time: 1.7538 data_time: 0.0143 memory: 12040 loss: 0.2034 2025/03/23 19:57:52 - mmengine - INFO - Iter(train) [ 6220/19176] lr: 1.5790e-05 eta: 5:14:58 time: 1.7027 data_time: 0.0144 memory: 11740 loss: 0.1934 2025/03/23 19:58:08 - mmengine - INFO - Iter(train) [ 6230/19176] lr: 1.5777e-05 eta: 5:14:47 time: 1.6265 data_time: 0.0143 memory: 11501 loss: 0.1789 2025/03/23 19:58:23 - mmengine - INFO - Iter(train) [ 6240/19176] lr: 1.5763e-05 eta: 5:14:35 time: 1.5464 data_time: 0.0140 memory: 11316 loss: 0.2273 2025/03/23 19:58:39 - mmengine - INFO - Iter(train) [ 6250/19176] lr: 1.5749e-05 eta: 5:14:21 time: 1.5230 data_time: 0.0141 memory: 11211 loss: 0.2081 2025/03/23 19:58:53 - mmengine - INFO - Iter(train) [ 6260/19176] lr: 1.5735e-05 eta: 5:14:07 time: 1.4723 data_time: 0.0141 memory: 11155 loss: 0.2445 2025/03/23 19:59:07 - mmengine - INFO - Iter(train) [ 6270/19176] lr: 1.5721e-05 eta: 5:13:50 time: 1.3470 data_time: 0.0136 memory: 10889 loss: 0.2383 2025/03/23 19:59:19 - mmengine - INFO - Iter(train) [ 6280/19176] lr: 1.5708e-05 eta: 5:13:30 time: 1.1899 data_time: 0.0130 memory: 10596 loss: 0.2224 2025/03/23 19:59:29 - mmengine - INFO - Iter(train) [ 6290/19176] lr: 1.5694e-05 eta: 5:13:06 time: 0.9831 data_time: 0.0125 memory: 10270 loss: 0.2418 2025/03/23 19:59:43 - mmengine - INFO - Iter(train) [ 6300/19176] lr: 1.5680e-05 eta: 5:12:51 time: 1.4602 data_time: 0.0131 memory: 13857 loss: 0.2033 2025/03/23 20:00:01 - mmengine - INFO - Iter(train) [ 6310/19176] lr: 1.5666e-05 eta: 5:12:43 time: 1.7592 data_time: 0.0147 memory: 11956 loss: 0.2100 2025/03/23 20:00:18 - mmengine - INFO - Iter(train) [ 6320/19176] lr: 1.5652e-05 eta: 5:12:33 time: 1.7093 data_time: 0.0150 memory: 11719 loss: 0.1871 2025/03/23 20:00:34 - mmengine - INFO - Iter(train) [ 6330/19176] lr: 1.5638e-05 eta: 5:12:22 time: 1.6386 data_time: 0.0146 memory: 11452 loss: 0.1991 2025/03/23 20:00:50 - mmengine - INFO - Iter(train) [ 6340/19176] lr: 1.5624e-05 eta: 5:12:10 time: 1.5801 data_time: 0.0144 memory: 11348 loss: 0.2029 2025/03/23 20:01:05 - mmengine - INFO - Iter(train) [ 6350/19176] lr: 1.5610e-05 eta: 5:11:56 time: 1.4854 data_time: 0.0138 memory: 11163 loss: 0.1866 2025/03/23 20:01:19 - mmengine - INFO - Iter(train) [ 6360/19176] lr: 1.5596e-05 eta: 5:11:41 time: 1.4327 data_time: 0.0145 memory: 11046 loss: 0.2129 2025/03/23 20:01:32 - mmengine - INFO - Iter(train) [ 6370/19176] lr: 1.5582e-05 eta: 5:11:22 time: 1.2574 data_time: 0.0135 memory: 10700 loss: 0.1954 2025/03/23 20:01:43 - mmengine - INFO - Iter(train) [ 6380/19176] lr: 1.5568e-05 eta: 5:11:00 time: 1.0774 data_time: 0.0123 memory: 10330 loss: 0.1910 2025/03/23 20:01:52 - mmengine - INFO - Iter(train) [ 6390/19176] lr: 1.5554e-05 eta: 5:10:34 time: 0.9035 data_time: 0.0117 memory: 10065 loss: 0.2086 2025/03/23 20:02:07 - mmengine - INFO - Iter(train) [ 6400/19176] lr: 1.5540e-05 eta: 5:10:21 time: 1.5257 data_time: 0.0123 memory: 18264 loss: 0.1755 2025/03/23 20:02:25 - mmengine - INFO - Iter(train) [ 6410/19176] lr: 1.5526e-05 eta: 5:10:13 time: 1.7768 data_time: 0.0141 memory: 12056 loss: 0.1808 2025/03/23 20:02:42 - mmengine - INFO - Iter(train) [ 6420/19176] lr: 1.5512e-05 eta: 5:10:03 time: 1.6862 data_time: 0.0147 memory: 11740 loss: 0.1831 2025/03/23 20:02:58 - mmengine - INFO - Iter(train) [ 6430/19176] lr: 1.5498e-05 eta: 5:09:52 time: 1.6288 data_time: 0.0144 memory: 11537 loss: 0.1993 2025/03/23 20:03:14 - mmengine - INFO - Iter(train) [ 6440/19176] lr: 1.5484e-05 eta: 5:09:40 time: 1.5986 data_time: 0.0145 memory: 11371 loss: 0.1955 2025/03/23 20:03:29 - mmengine - INFO - Iter(train) [ 6450/19176] lr: 1.5469e-05 eta: 5:09:26 time: 1.4871 data_time: 0.0140 memory: 11197 loss: 0.1970 2025/03/23 20:03:43 - mmengine - INFO - Iter(train) [ 6460/19176] lr: 1.5455e-05 eta: 5:09:10 time: 1.3992 data_time: 0.0138 memory: 11006 loss: 0.2271 2025/03/23 20:03:56 - mmengine - INFO - Iter(train) [ 6470/19176] lr: 1.5441e-05 eta: 5:08:53 time: 1.3391 data_time: 0.0145 memory: 10820 loss: 0.2233 2025/03/23 20:04:07 - mmengine - INFO - Iter(train) [ 6480/19176] lr: 1.5427e-05 eta: 5:08:32 time: 1.1374 data_time: 0.0129 memory: 10435 loss: 0.2310 2025/03/23 20:04:17 - mmengine - INFO - Iter(train) [ 6490/19176] lr: 1.5413e-05 eta: 5:08:08 time: 0.9593 data_time: 0.0123 memory: 10072 loss: 0.2055 2025/03/23 20:04:32 - mmengine - INFO - Iter(train) [ 6500/19176] lr: 1.5399e-05 eta: 5:07:55 time: 1.5345 data_time: 0.0135 memory: 14641 loss: 0.2199 2025/03/23 20:04:50 - mmengine - INFO - Iter(train) [ 6510/19176] lr: 1.5384e-05 eta: 5:07:47 time: 1.8098 data_time: 0.0146 memory: 12312 loss: 0.2499 2025/03/23 20:05:07 - mmengine - INFO - Iter(train) [ 6520/19176] lr: 1.5370e-05 eta: 5:07:37 time: 1.6895 data_time: 0.0149 memory: 11695 loss: 0.1592 2025/03/23 20:05:24 - mmengine - INFO - Iter(train) [ 6530/19176] lr: 1.5356e-05 eta: 5:07:26 time: 1.6321 data_time: 0.0149 memory: 11506 loss: 0.2418 2025/03/23 20:05:40 - mmengine - INFO - Iter(train) [ 6540/19176] lr: 1.5342e-05 eta: 5:07:14 time: 1.5810 data_time: 0.0147 memory: 11376 loss: 0.1896 2025/03/23 20:05:55 - mmengine - INFO - Iter(train) [ 6550/19176] lr: 1.5327e-05 eta: 5:07:00 time: 1.5119 data_time: 0.0144 memory: 11182 loss: 0.1984 2025/03/23 20:06:09 - mmengine - INFO - Iter(train) [ 6560/19176] lr: 1.5313e-05 eta: 5:06:45 time: 1.4510 data_time: 0.0145 memory: 11064 loss: 0.2644 2025/03/23 20:06:22 - mmengine - INFO - Iter(train) [ 6570/19176] lr: 1.5299e-05 eta: 5:06:27 time: 1.2891 data_time: 0.0139 memory: 10925 loss: 0.1982 2025/03/23 20:06:33 - mmengine - INFO - Iter(train) [ 6580/19176] lr: 1.5284e-05 eta: 5:06:06 time: 1.0863 data_time: 0.0123 memory: 10393 loss: 0.2332 2025/03/23 20:06:41 - mmengine - INFO - Iter(train) [ 6590/19176] lr: 1.5270e-05 eta: 5:05:39 time: 0.8453 data_time: 0.0116 memory: 9946 loss: 0.2089 2025/03/23 20:06:56 - mmengine - INFO - Iter(train) [ 6600/19176] lr: 1.5256e-05 eta: 5:05:25 time: 1.4461 data_time: 0.0127 memory: 13791 loss: 0.2230 2025/03/23 20:07:14 - mmengine - INFO - Iter(train) [ 6610/19176] lr: 1.5241e-05 eta: 5:05:16 time: 1.7948 data_time: 0.0146 memory: 12272 loss: 0.1955 2025/03/23 20:07:31 - mmengine - INFO - Iter(train) [ 6620/19176] lr: 1.5227e-05 eta: 5:05:07 time: 1.7093 data_time: 0.0146 memory: 11783 loss: 0.1626 2025/03/23 20:07:47 - mmengine - INFO - Iter(train) [ 6630/19176] lr: 1.5212e-05 eta: 5:04:56 time: 1.6467 data_time: 0.0151 memory: 11495 loss: 0.1835 2025/03/23 20:08:03 - mmengine - INFO - Iter(train) [ 6640/19176] lr: 1.5198e-05 eta: 5:04:43 time: 1.5768 data_time: 0.0146 memory: 11345 loss: 0.1727 2025/03/23 20:08:18 - mmengine - INFO - Iter(train) [ 6650/19176] lr: 1.5184e-05 eta: 5:04:30 time: 1.5353 data_time: 0.0146 memory: 11234 loss: 0.2128 2025/03/23 20:08:33 - mmengine - INFO - Iter(train) [ 6660/19176] lr: 1.5169e-05 eta: 5:04:16 time: 1.4878 data_time: 0.0150 memory: 11250 loss: 0.1788 2025/03/23 20:08:47 - mmengine - INFO - Iter(train) [ 6670/19176] lr: 1.5155e-05 eta: 5:04:00 time: 1.3705 data_time: 0.0144 memory: 10891 loss: 0.2198 2025/03/23 20:09:00 - mmengine - INFO - Iter(train) [ 6680/19176] lr: 1.5140e-05 eta: 5:03:41 time: 1.2478 data_time: 0.0144 memory: 10703 loss: 0.2281 2025/03/23 20:09:10 - mmengine - INFO - Iter(train) [ 6690/19176] lr: 1.5126e-05 eta: 5:03:20 time: 1.0820 data_time: 0.0129 memory: 10313 loss: 0.2036 2025/03/23 20:09:27 - mmengine - INFO - Iter(train) [ 6700/19176] lr: 1.5111e-05 eta: 5:03:08 time: 1.6345 data_time: 0.0134 memory: 15361 loss: 0.2676 2025/03/23 20:09:44 - mmengine - INFO - Iter(train) [ 6710/19176] lr: 1.5097e-05 eta: 5:03:00 time: 1.7725 data_time: 0.0146 memory: 12326 loss: 0.1728 2025/03/23 20:10:01 - mmengine - INFO - Iter(train) [ 6720/19176] lr: 1.5082e-05 eta: 5:02:49 time: 1.6583 data_time: 0.0148 memory: 11557 loss: 0.1900 2025/03/23 20:10:17 - mmengine - INFO - Iter(train) [ 6730/19176] lr: 1.5068e-05 eta: 5:02:37 time: 1.6285 data_time: 0.0146 memory: 11432 loss: 0.1770 2025/03/23 20:10:33 - mmengine - INFO - Iter(train) [ 6740/19176] lr: 1.5053e-05 eta: 5:02:25 time: 1.5672 data_time: 0.0146 memory: 11321 loss: 0.2041 2025/03/23 20:10:48 - mmengine - INFO - Iter(train) [ 6750/19176] lr: 1.5038e-05 eta: 5:02:12 time: 1.5365 data_time: 0.0145 memory: 11325 loss: 0.1801 2025/03/23 20:11:03 - mmengine - INFO - Iter(train) [ 6760/19176] lr: 1.5024e-05 eta: 5:01:57 time: 1.4604 data_time: 0.0144 memory: 11111 loss: 0.1850 2025/03/23 20:11:16 - mmengine - INFO - Iter(train) [ 6770/19176] lr: 1.5009e-05 eta: 5:01:40 time: 1.3515 data_time: 0.0138 memory: 10924 loss: 0.2038 2025/03/23 20:11:27 - mmengine - INFO - Iter(train) [ 6780/19176] lr: 1.4995e-05 eta: 5:01:19 time: 1.1057 data_time: 0.0129 memory: 10538 loss: 0.2263 2025/03/23 20:11:37 - mmengine - INFO - Iter(train) [ 6790/19176] lr: 1.4980e-05 eta: 5:00:56 time: 0.9468 data_time: 0.0124 memory: 10030 loss: 0.2094 2025/03/23 20:11:52 - mmengine - INFO - Iter(train) [ 6800/19176] lr: 1.4965e-05 eta: 5:00:41 time: 1.4636 data_time: 0.0123 memory: 15316 loss: 0.2480 2025/03/23 20:12:09 - mmengine - INFO - Iter(train) [ 6810/19176] lr: 1.4951e-05 eta: 5:00:32 time: 1.7732 data_time: 0.0145 memory: 12244 loss: 0.1550 2025/03/23 20:12:26 - mmengine - INFO - Iter(train) [ 6820/19176] lr: 1.4936e-05 eta: 5:00:22 time: 1.6790 data_time: 0.0147 memory: 11688 loss: 0.1713 2025/03/23 20:12:43 - mmengine - INFO - Iter(train) [ 6830/19176] lr: 1.4921e-05 eta: 5:00:10 time: 1.6401 data_time: 0.0147 memory: 11480 loss: 0.1697 2025/03/23 20:12:59 - mmengine - INFO - Iter(train) [ 6840/19176] lr: 1.4907e-05 eta: 4:59:58 time: 1.6027 data_time: 0.0151 memory: 11381 loss: 0.1873 2025/03/23 20:13:14 - mmengine - INFO - Iter(train) [ 6850/19176] lr: 1.4892e-05 eta: 4:59:46 time: 1.5618 data_time: 0.0150 memory: 11357 loss: 0.1931 2025/03/23 20:13:29 - mmengine - INFO - Iter(train) [ 6860/19176] lr: 1.4877e-05 eta: 4:59:32 time: 1.5283 data_time: 0.0146 memory: 11214 loss: 0.2447 2025/03/23 20:13:44 - mmengine - INFO - Iter(train) [ 6870/19176] lr: 1.4862e-05 eta: 4:59:18 time: 1.4651 data_time: 0.0149 memory: 11130 loss: 0.1984 2025/03/23 20:13:57 - mmengine - INFO - Iter(train) [ 6880/19176] lr: 1.4848e-05 eta: 4:59:01 time: 1.3381 data_time: 0.0147 memory: 10903 loss: 0.2432 2025/03/23 20:14:07 - mmengine - INFO - Iter(train) [ 6890/19176] lr: 1.4833e-05 eta: 4:58:38 time: 0.9869 data_time: 0.0127 memory: 10238 loss: 0.1930 2025/03/23 20:14:23 - mmengine - INFO - Iter(train) [ 6900/19176] lr: 1.4818e-05 eta: 4:58:26 time: 1.5894 data_time: 0.0133 memory: 14533 loss: 0.2203 2025/03/23 20:14:42 - mmengine - INFO - Iter(train) [ 6910/19176] lr: 1.4803e-05 eta: 4:58:18 time: 1.8331 data_time: 0.0151 memory: 12534 loss: 0.1895 2025/03/23 20:14:59 - mmengine - INFO - Iter(train) [ 6920/19176] lr: 1.4788e-05 eta: 4:58:08 time: 1.7186 data_time: 0.0150 memory: 11872 loss: 0.2294 2025/03/23 20:15:15 - mmengine - INFO - Iter(train) [ 6930/19176] lr: 1.4774e-05 eta: 4:57:57 time: 1.6722 data_time: 0.0149 memory: 11557 loss: 0.2160 2025/03/23 20:15:32 - mmengine - INFO - Iter(train) [ 6940/19176] lr: 1.4759e-05 eta: 4:57:46 time: 1.6407 data_time: 0.0144 memory: 11472 loss: 0.1816 2025/03/23 20:15:47 - mmengine - INFO - Iter(train) [ 6950/19176] lr: 1.4744e-05 eta: 4:57:33 time: 1.5597 data_time: 0.0143 memory: 11302 loss: 0.1892 2025/03/23 20:16:03 - mmengine - INFO - Iter(train) [ 6960/19176] lr: 1.4729e-05 eta: 4:57:19 time: 1.5288 data_time: 0.0146 memory: 11375 loss: 0.2280 2025/03/23 20:16:17 - mmengine - INFO - Iter(train) [ 6970/19176] lr: 1.4714e-05 eta: 4:57:04 time: 1.4471 data_time: 0.0145 memory: 11065 loss: 0.2283 2025/03/23 20:16:30 - mmengine - INFO - Iter(train) [ 6980/19176] lr: 1.4699e-05 eta: 4:56:47 time: 1.2774 data_time: 0.0135 memory: 10789 loss: 0.2319 2025/03/23 20:16:41 - mmengine - INFO - Iter(train) [ 6990/19176] lr: 1.4684e-05 eta: 4:56:25 time: 1.0813 data_time: 0.0124 memory: 10332 loss: 0.2042 2025/03/23 20:16:57 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 20:16:57 - mmengine - INFO - Iter(train) [ 7000/19176] lr: 1.4669e-05 eta: 4:56:14 time: 1.6382 data_time: 0.0131 memory: 16168 loss: 0.1966 2025/03/23 20:16:57 - mmengine - INFO - Saving checkpoint at 7000 iterations 2025/03/23 20:17:16 - mmengine - INFO - Iter(train) [ 7010/19176] lr: 1.4654e-05 eta: 4:56:06 time: 1.8517 data_time: 0.0910 memory: 12190 loss: 0.1818 2025/03/23 20:17:33 - mmengine - INFO - Iter(train) [ 7020/19176] lr: 1.4639e-05 eta: 4:55:55 time: 1.6782 data_time: 0.0144 memory: 11645 loss: 0.1564 2025/03/23 20:17:49 - mmengine - INFO - Iter(train) [ 7030/19176] lr: 1.4624e-05 eta: 4:55:44 time: 1.6437 data_time: 0.0143 memory: 11485 loss: 0.1854 2025/03/23 20:18:05 - mmengine - INFO - Iter(train) [ 7040/19176] lr: 1.4610e-05 eta: 4:55:32 time: 1.6095 data_time: 0.0151 memory: 11372 loss: 0.1796 2025/03/23 20:18:21 - mmengine - INFO - Iter(train) [ 7050/19176] lr: 1.4595e-05 eta: 4:55:19 time: 1.5521 data_time: 0.0154 memory: 11288 loss: 0.1646 2025/03/23 20:18:35 - mmengine - INFO - Iter(train) [ 7060/19176] lr: 1.4580e-05 eta: 4:55:05 time: 1.4841 data_time: 0.0152 memory: 11159 loss: 0.2182 2025/03/23 20:18:49 - mmengine - INFO - Iter(train) [ 7070/19176] lr: 1.4564e-05 eta: 4:54:49 time: 1.4046 data_time: 0.0148 memory: 10982 loss: 0.2043 2025/03/23 20:19:01 - mmengine - INFO - Iter(train) [ 7080/19176] lr: 1.4549e-05 eta: 4:54:29 time: 1.1486 data_time: 0.0128 memory: 10494 loss: 0.1805 2025/03/23 20:19:10 - mmengine - INFO - Iter(train) [ 7090/19176] lr: 1.4534e-05 eta: 4:54:06 time: 0.9527 data_time: 0.0122 memory: 10030 loss: 0.2239 2025/03/23 20:19:24 - mmengine - INFO - Iter(train) [ 7100/19176] lr: 1.4519e-05 eta: 4:53:50 time: 1.3804 data_time: 0.0119 memory: 14704 loss: 0.1694 2025/03/23 20:19:42 - mmengine - INFO - Iter(train) [ 7110/19176] lr: 1.4504e-05 eta: 4:53:40 time: 1.7479 data_time: 0.0145 memory: 11907 loss: 0.1606 2025/03/23 20:19:59 - mmengine - INFO - Iter(train) [ 7120/19176] lr: 1.4489e-05 eta: 4:53:29 time: 1.6949 data_time: 0.0146 memory: 11704 loss: 0.1976 2025/03/23 20:20:15 - mmengine - INFO - Iter(train) [ 7130/19176] lr: 1.4474e-05 eta: 4:53:18 time: 1.6375 data_time: 0.0146 memory: 11541 loss: 0.2038 2025/03/23 20:20:31 - mmengine - INFO - Iter(train) [ 7140/19176] lr: 1.4459e-05 eta: 4:53:05 time: 1.5593 data_time: 0.0146 memory: 11288 loss: 0.2004 2025/03/23 20:20:46 - mmengine - INFO - Iter(train) [ 7150/19176] lr: 1.4444e-05 eta: 4:52:51 time: 1.5240 data_time: 0.0147 memory: 11188 loss: 0.1867 2025/03/23 20:21:00 - mmengine - INFO - Iter(train) [ 7160/19176] lr: 1.4429e-05 eta: 4:52:36 time: 1.4438 data_time: 0.0138 memory: 11048 loss: 0.2322 2025/03/23 20:21:14 - mmengine - INFO - Iter(train) [ 7170/19176] lr: 1.4414e-05 eta: 4:52:20 time: 1.3788 data_time: 0.0141 memory: 10940 loss: 0.2395 2025/03/23 20:21:26 - mmengine - INFO - Iter(train) [ 7180/19176] lr: 1.4398e-05 eta: 4:52:01 time: 1.1660 data_time: 0.0123 memory: 10562 loss: 0.1998 2025/03/23 20:21:36 - mmengine - INFO - Iter(train) [ 7190/19176] lr: 1.4383e-05 eta: 4:51:39 time: 0.9996 data_time: 0.0123 memory: 10158 loss: 0.1882 2025/03/23 20:21:51 - mmengine - INFO - Iter(train) [ 7200/19176] lr: 1.4368e-05 eta: 4:51:25 time: 1.5002 data_time: 0.0129 memory: 15167 loss: 0.1982 2025/03/23 20:22:09 - mmengine - INFO - Iter(train) [ 7210/19176] lr: 1.4353e-05 eta: 4:51:16 time: 1.7904 data_time: 0.0146 memory: 12360 loss: 0.1666 2025/03/23 20:22:26 - mmengine - INFO - Iter(train) [ 7220/19176] lr: 1.4338e-05 eta: 4:51:05 time: 1.7194 data_time: 0.0144 memory: 11840 loss: 0.1829 2025/03/23 20:22:43 - mmengine - INFO - Iter(train) [ 7230/19176] lr: 1.4322e-05 eta: 4:50:54 time: 1.6693 data_time: 0.0147 memory: 11628 loss: 0.1786 2025/03/23 20:22:59 - mmengine - INFO - Iter(train) [ 7240/19176] lr: 1.4307e-05 eta: 4:50:42 time: 1.6116 data_time: 0.0148 memory: 11429 loss: 0.2001 2025/03/23 20:23:14 - mmengine - INFO - Iter(train) [ 7250/19176] lr: 1.4292e-05 eta: 4:50:29 time: 1.5413 data_time: 0.0142 memory: 11275 loss: 0.1853 2025/03/23 20:23:29 - mmengine - INFO - Iter(train) [ 7260/19176] lr: 1.4277e-05 eta: 4:50:14 time: 1.4497 data_time: 0.0142 memory: 11159 loss: 0.2218 2025/03/23 20:23:42 - mmengine - INFO - Iter(train) [ 7270/19176] lr: 1.4261e-05 eta: 4:49:57 time: 1.3461 data_time: 0.0145 memory: 10833 loss: 0.2210 2025/03/23 20:23:53 - mmengine - INFO - Iter(train) [ 7280/19176] lr: 1.4246e-05 eta: 4:49:37 time: 1.0943 data_time: 0.0128 memory: 10549 loss: 0.2008 2025/03/23 20:24:01 - mmengine - INFO - Iter(train) [ 7290/19176] lr: 1.4231e-05 eta: 4:49:12 time: 0.8336 data_time: 0.0121 memory: 9880 loss: 0.2032 2025/03/23 20:24:15 - mmengine - INFO - Iter(train) [ 7300/19176] lr: 1.4216e-05 eta: 4:48:56 time: 1.3656 data_time: 0.0128 memory: 12694 loss: 0.1878 2025/03/23 20:24:33 - mmengine - INFO - Iter(train) [ 7310/19176] lr: 1.4200e-05 eta: 4:48:46 time: 1.7809 data_time: 0.0146 memory: 12117 loss: 0.1876 2025/03/23 20:24:50 - mmengine - INFO - Iter(train) [ 7320/19176] lr: 1.4185e-05 eta: 4:48:36 time: 1.6920 data_time: 0.0145 memory: 11787 loss: 0.1842 2025/03/23 20:25:06 - mmengine - INFO - Iter(train) [ 7330/19176] lr: 1.4170e-05 eta: 4:48:24 time: 1.6240 data_time: 0.0144 memory: 11429 loss: 0.1817 2025/03/23 20:25:22 - mmengine - INFO - Iter(train) [ 7340/19176] lr: 1.4154e-05 eta: 4:48:11 time: 1.6007 data_time: 0.0153 memory: 11358 loss: 0.1793 2025/03/23 20:25:37 - mmengine - INFO - Iter(train) [ 7350/19176] lr: 1.4139e-05 eta: 4:47:57 time: 1.4790 data_time: 0.0146 memory: 11303 loss: 0.2182 2025/03/23 20:25:51 - mmengine - INFO - Iter(train) [ 7360/19176] lr: 1.4123e-05 eta: 4:47:42 time: 1.4130 data_time: 0.0144 memory: 10966 loss: 0.2278 2025/03/23 20:26:04 - mmengine - INFO - Iter(train) [ 7370/19176] lr: 1.4108e-05 eta: 4:47:24 time: 1.3094 data_time: 0.0155 memory: 10811 loss: 0.2632 2025/03/23 20:26:16 - mmengine - INFO - Iter(train) [ 7380/19176] lr: 1.4093e-05 eta: 4:47:05 time: 1.1549 data_time: 0.0154 memory: 10521 loss: 0.1940 2025/03/23 20:26:25 - mmengine - INFO - Iter(train) [ 7390/19176] lr: 1.4077e-05 eta: 4:46:43 time: 0.9683 data_time: 0.0141 memory: 10101 loss: 0.2040 2025/03/23 20:26:40 - mmengine - INFO - Iter(train) [ 7400/19176] lr: 1.4062e-05 eta: 4:46:28 time: 1.4843 data_time: 0.0140 memory: 15665 loss: 0.1982 2025/03/23 20:26:58 - mmengine - INFO - Iter(train) [ 7410/19176] lr: 1.4046e-05 eta: 4:46:19 time: 1.7921 data_time: 0.0161 memory: 12211 loss: 0.1741 2025/03/23 20:27:15 - mmengine - INFO - Iter(train) [ 7420/19176] lr: 1.4031e-05 eta: 4:46:08 time: 1.7153 data_time: 0.0167 memory: 11744 loss: 0.1586 2025/03/23 20:27:32 - mmengine - INFO - Iter(train) [ 7430/19176] lr: 1.4015e-05 eta: 4:45:57 time: 1.6504 data_time: 0.0164 memory: 11705 loss: 0.1812 2025/03/23 20:27:48 - mmengine - INFO - Iter(train) [ 7440/19176] lr: 1.4000e-05 eta: 4:45:45 time: 1.6040 data_time: 0.0167 memory: 11482 loss: 0.1645 2025/03/23 20:28:03 - mmengine - INFO - Iter(train) [ 7450/19176] lr: 1.3984e-05 eta: 4:45:31 time: 1.5291 data_time: 0.0160 memory: 11257 loss: 0.1787 2025/03/23 20:28:18 - mmengine - INFO - Iter(train) [ 7460/19176] lr: 1.3969e-05 eta: 4:45:16 time: 1.4667 data_time: 0.0149 memory: 11084 loss: 0.1715 2025/03/23 20:28:32 - mmengine - INFO - Iter(train) [ 7470/19176] lr: 1.3953e-05 eta: 4:45:01 time: 1.3972 data_time: 0.0148 memory: 10973 loss: 0.2018 2025/03/23 20:28:44 - mmengine - INFO - Iter(train) [ 7480/19176] lr: 1.3938e-05 eta: 4:44:43 time: 1.2374 data_time: 0.0143 memory: 10750 loss: 0.1982 2025/03/23 20:28:54 - mmengine - INFO - Iter(train) [ 7490/19176] lr: 1.3922e-05 eta: 4:44:21 time: 1.0342 data_time: 0.0129 memory: 10229 loss: 0.1846 2025/03/23 20:29:11 - mmengine - INFO - Iter(train) [ 7500/19176] lr: 1.3907e-05 eta: 4:44:11 time: 1.7115 data_time: 0.0136 memory: 16557 loss: 0.2214 2025/03/23 20:29:29 - mmengine - INFO - Iter(train) [ 7510/19176] lr: 1.3891e-05 eta: 4:44:01 time: 1.7580 data_time: 0.0148 memory: 12174 loss: 0.2225 2025/03/23 20:29:46 - mmengine - INFO - Iter(train) [ 7520/19176] lr: 1.3876e-05 eta: 4:43:49 time: 1.6666 data_time: 0.0146 memory: 11713 loss: 0.1833 2025/03/23 20:30:02 - mmengine - INFO - Iter(train) [ 7530/19176] lr: 1.3860e-05 eta: 4:43:37 time: 1.6060 data_time: 0.0143 memory: 11393 loss: 0.1724 2025/03/23 20:30:17 - mmengine - INFO - Iter(train) [ 7540/19176] lr: 1.3845e-05 eta: 4:43:24 time: 1.5457 data_time: 0.0147 memory: 11281 loss: 0.2482 2025/03/23 20:30:32 - mmengine - INFO - Iter(train) [ 7550/19176] lr: 1.3829e-05 eta: 4:43:09 time: 1.4305 data_time: 0.0150 memory: 11018 loss: 0.2128 2025/03/23 20:30:45 - mmengine - INFO - Iter(train) [ 7560/19176] lr: 1.3813e-05 eta: 4:42:52 time: 1.3289 data_time: 0.0145 memory: 10837 loss: 0.2514 2025/03/23 20:30:56 - mmengine - INFO - Iter(train) [ 7570/19176] lr: 1.3798e-05 eta: 4:42:32 time: 1.1430 data_time: 0.0129 memory: 10508 loss: 0.2103 2025/03/23 20:31:07 - mmengine - INFO - Iter(train) [ 7580/19176] lr: 1.3782e-05 eta: 4:42:11 time: 1.0275 data_time: 0.0128 memory: 10134 loss: 0.1962 2025/03/23 20:31:15 - mmengine - INFO - Iter(train) [ 7590/19176] lr: 1.3766e-05 eta: 4:41:47 time: 0.8558 data_time: 0.0127 memory: 9785 loss: 0.1930 2025/03/23 20:31:29 - mmengine - INFO - Iter(train) [ 7600/19176] lr: 1.3751e-05 eta: 4:41:32 time: 1.3939 data_time: 0.0132 memory: 12766 loss: 0.2121 2025/03/23 20:31:47 - mmengine - INFO - Iter(train) [ 7610/19176] lr: 1.3735e-05 eta: 4:41:22 time: 1.7589 data_time: 0.0151 memory: 12152 loss: 0.1902 2025/03/23 20:32:03 - mmengine - INFO - Iter(train) [ 7620/19176] lr: 1.3719e-05 eta: 4:41:10 time: 1.6682 data_time: 0.0155 memory: 11626 loss: 0.1612 2025/03/23 20:32:19 - mmengine - INFO - Iter(train) [ 7630/19176] lr: 1.3704e-05 eta: 4:40:58 time: 1.6196 data_time: 0.0154 memory: 11435 loss: 0.1836 2025/03/23 20:32:35 - mmengine - INFO - Iter(train) [ 7640/19176] lr: 1.3688e-05 eta: 4:40:45 time: 1.5628 data_time: 0.0151 memory: 11276 loss: 0.2219 2025/03/23 20:32:50 - mmengine - INFO - Iter(train) [ 7650/19176] lr: 1.3672e-05 eta: 4:40:31 time: 1.4925 data_time: 0.0153 memory: 11134 loss: 0.1992 2025/03/23 20:33:04 - mmengine - INFO - Iter(train) [ 7660/19176] lr: 1.3657e-05 eta: 4:40:16 time: 1.4314 data_time: 0.0147 memory: 11045 loss: 0.1783 2025/03/23 20:33:17 - mmengine - INFO - Iter(train) [ 7670/19176] lr: 1.3641e-05 eta: 4:39:58 time: 1.2187 data_time: 0.0138 memory: 10750 loss: 0.2229 2025/03/23 20:33:27 - mmengine - INFO - Iter(train) [ 7680/19176] lr: 1.3625e-05 eta: 4:39:37 time: 1.0235 data_time: 0.0127 memory: 10183 loss: 0.2304 2025/03/23 20:33:35 - mmengine - INFO - Iter(train) [ 7690/19176] lr: 1.3609e-05 eta: 4:39:13 time: 0.8720 data_time: 0.0117 memory: 9857 loss: 0.1887 2025/03/23 20:33:50 - mmengine - INFO - Iter(train) [ 7700/19176] lr: 1.3594e-05 eta: 4:38:59 time: 1.4517 data_time: 0.0124 memory: 13644 loss: 0.2025 2025/03/23 20:34:08 - mmengine - INFO - Iter(train) [ 7710/19176] lr: 1.3578e-05 eta: 4:38:48 time: 1.7547 data_time: 0.0146 memory: 11986 loss: 0.2043 2025/03/23 20:34:24 - mmengine - INFO - Iter(train) [ 7720/19176] lr: 1.3562e-05 eta: 4:38:37 time: 1.6876 data_time: 0.0150 memory: 11686 loss: 0.1656 2025/03/23 20:34:41 - mmengine - INFO - Iter(train) [ 7730/19176] lr: 1.3546e-05 eta: 4:38:25 time: 1.6254 data_time: 0.0146 memory: 11472 loss: 0.1763 2025/03/23 20:34:57 - mmengine - INFO - Iter(train) [ 7740/19176] lr: 1.3531e-05 eta: 4:38:12 time: 1.5824 data_time: 0.0149 memory: 11344 loss: 0.1923 2025/03/23 20:35:12 - mmengine - INFO - Iter(train) [ 7750/19176] lr: 1.3515e-05 eta: 4:37:59 time: 1.5342 data_time: 0.0145 memory: 11233 loss: 0.2038 2025/03/23 20:35:26 - mmengine - INFO - Iter(train) [ 7760/19176] lr: 1.3499e-05 eta: 4:37:44 time: 1.4326 data_time: 0.0144 memory: 11033 loss: 0.2699 2025/03/23 20:35:39 - mmengine - INFO - Iter(train) [ 7770/19176] lr: 1.3483e-05 eta: 4:37:27 time: 1.2866 data_time: 0.0142 memory: 10820 loss: 0.2369 2025/03/23 20:35:50 - mmengine - INFO - Iter(train) [ 7780/19176] lr: 1.3467e-05 eta: 4:37:07 time: 1.0981 data_time: 0.0127 memory: 10329 loss: 0.2382 2025/03/23 20:35:59 - mmengine - INFO - Iter(train) [ 7790/19176] lr: 1.3451e-05 eta: 4:36:44 time: 0.9047 data_time: 0.0120 memory: 9979 loss: 0.2294 2025/03/23 20:36:14 - mmengine - INFO - Iter(train) [ 7800/19176] lr: 1.3436e-05 eta: 4:36:30 time: 1.5219 data_time: 0.0126 memory: 14151 loss: 0.1998 2025/03/23 20:36:31 - mmengine - INFO - Iter(train) [ 7810/19176] lr: 1.3420e-05 eta: 4:36:20 time: 1.7207 data_time: 0.0144 memory: 11919 loss: 0.1755 2025/03/23 20:36:48 - mmengine - INFO - Iter(train) [ 7820/19176] lr: 1.3404e-05 eta: 4:36:08 time: 1.6621 data_time: 0.0144 memory: 11621 loss: 0.1664 2025/03/23 20:37:04 - mmengine - INFO - Iter(train) [ 7830/19176] lr: 1.3388e-05 eta: 4:35:56 time: 1.6095 data_time: 0.0146 memory: 11422 loss: 0.1708 2025/03/23 20:37:20 - mmengine - INFO - Iter(train) [ 7840/19176] lr: 1.3372e-05 eta: 4:35:42 time: 1.5418 data_time: 0.0146 memory: 11230 loss: 0.2265 2025/03/23 20:37:34 - mmengine - INFO - Iter(train) [ 7850/19176] lr: 1.3356e-05 eta: 4:35:28 time: 1.4784 data_time: 0.0148 memory: 11093 loss: 0.1721 2025/03/23 20:37:49 - mmengine - INFO - Iter(train) [ 7860/19176] lr: 1.3340e-05 eta: 4:35:13 time: 1.4437 data_time: 0.0148 memory: 11010 loss: 0.1805 2025/03/23 20:38:02 - mmengine - INFO - Iter(train) [ 7870/19176] lr: 1.3324e-05 eta: 4:34:56 time: 1.2939 data_time: 0.0141 memory: 10813 loss: 0.2174 2025/03/23 20:38:13 - mmengine - INFO - Iter(train) [ 7880/19176] lr: 1.3308e-05 eta: 4:34:36 time: 1.0767 data_time: 0.0123 memory: 10275 loss: 0.1853 2025/03/23 20:38:22 - mmengine - INFO - Iter(train) [ 7890/19176] lr: 1.3292e-05 eta: 4:34:13 time: 0.8999 data_time: 0.0122 memory: 9888 loss: 0.2002 2025/03/23 20:38:35 - mmengine - INFO - Iter(train) [ 7900/19176] lr: 1.3277e-05 eta: 4:33:58 time: 1.3918 data_time: 0.0126 memory: 13636 loss: 0.2361 2025/03/23 20:38:53 - mmengine - INFO - Iter(train) [ 7910/19176] lr: 1.3261e-05 eta: 4:33:47 time: 1.7469 data_time: 0.0147 memory: 11986 loss: 0.1943 2025/03/23 20:39:10 - mmengine - INFO - Iter(train) [ 7920/19176] lr: 1.3245e-05 eta: 4:33:36 time: 1.7042 data_time: 0.0145 memory: 11758 loss: 0.1884 2025/03/23 20:39:26 - mmengine - INFO - Iter(train) [ 7930/19176] lr: 1.3229e-05 eta: 4:33:24 time: 1.6285 data_time: 0.0145 memory: 11463 loss: 0.2322 2025/03/23 20:39:42 - mmengine - INFO - Iter(train) [ 7940/19176] lr: 1.3213e-05 eta: 4:33:11 time: 1.5824 data_time: 0.0146 memory: 11298 loss: 0.1893 2025/03/23 20:39:57 - mmengine - INFO - Iter(train) [ 7950/19176] lr: 1.3197e-05 eta: 4:32:58 time: 1.5333 data_time: 0.0146 memory: 11265 loss: 0.1666 2025/03/23 20:40:12 - mmengine - INFO - Iter(train) [ 7960/19176] lr: 1.3181e-05 eta: 4:32:43 time: 1.4671 data_time: 0.0145 memory: 11116 loss: 0.2158 2025/03/23 20:40:25 - mmengine - INFO - Iter(train) [ 7970/19176] lr: 1.3165e-05 eta: 4:32:27 time: 1.3361 data_time: 0.0141 memory: 10818 loss: 0.2161 2025/03/23 20:40:37 - mmengine - INFO - Iter(train) [ 7980/19176] lr: 1.3149e-05 eta: 4:32:09 time: 1.1806 data_time: 0.0130 memory: 10594 loss: 0.2374 2025/03/23 20:40:47 - mmengine - INFO - Iter(train) [ 7990/19176] lr: 1.3133e-05 eta: 4:31:47 time: 0.9415 data_time: 0.0120 memory: 10093 loss: 0.2329 2025/03/23 20:41:04 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 20:41:04 - mmengine - INFO - Iter(train) [ 8000/19176] lr: 1.3117e-05 eta: 4:31:35 time: 1.6961 data_time: 0.0127 memory: 18868 loss: 0.1823 2025/03/23 20:41:04 - mmengine - INFO - Saving checkpoint at 8000 iterations 2025/03/23 20:41:23 - mmengine - INFO - Iter(train) [ 8010/19176] lr: 1.3100e-05 eta: 4:31:27 time: 1.8999 data_time: 0.0916 memory: 12401 loss: 0.1761 2025/03/23 20:41:40 - mmengine - INFO - Iter(train) [ 8020/19176] lr: 1.3084e-05 eta: 4:31:16 time: 1.7082 data_time: 0.0144 memory: 11808 loss: 0.1866 2025/03/23 20:41:56 - mmengine - INFO - Iter(train) [ 8030/19176] lr: 1.3068e-05 eta: 4:31:04 time: 1.6303 data_time: 0.0142 memory: 11472 loss: 0.1771 2025/03/23 20:42:12 - mmengine - INFO - Iter(train) [ 8040/19176] lr: 1.3052e-05 eta: 4:30:51 time: 1.5658 data_time: 0.0147 memory: 11308 loss: 0.1899 2025/03/23 20:42:27 - mmengine - INFO - Iter(train) [ 8050/19176] lr: 1.3036e-05 eta: 4:30:37 time: 1.5020 data_time: 0.0143 memory: 11230 loss: 0.2016 2025/03/23 20:42:41 - mmengine - INFO - Iter(train) [ 8060/19176] lr: 1.3020e-05 eta: 4:30:21 time: 1.3873 data_time: 0.0142 memory: 10909 loss: 0.2028 2025/03/23 20:42:52 - mmengine - INFO - Iter(train) [ 8070/19176] lr: 1.3004e-05 eta: 4:30:02 time: 1.1324 data_time: 0.0124 memory: 10519 loss: 0.1703 2025/03/23 20:43:03 - mmengine - INFO - Iter(train) [ 8080/19176] lr: 1.2988e-05 eta: 4:29:42 time: 1.0624 data_time: 0.0121 memory: 10245 loss: 0.1957 2025/03/23 20:43:12 - mmengine - INFO - Iter(train) [ 8090/19176] lr: 1.2972e-05 eta: 4:29:20 time: 0.9607 data_time: 0.0120 memory: 10048 loss: 0.2013 2025/03/23 20:43:30 - mmengine - INFO - Iter(train) [ 8100/19176] lr: 1.2956e-05 eta: 4:29:10 time: 1.7749 data_time: 0.0127 memory: 16526 loss: 0.2539 2025/03/23 20:43:48 - mmengine - INFO - Iter(train) [ 8110/19176] lr: 1.2939e-05 eta: 4:29:01 time: 1.8563 data_time: 0.0147 memory: 12710 loss: 0.1743 2025/03/23 20:44:06 - mmengine - INFO - Iter(train) [ 8120/19176] lr: 1.2923e-05 eta: 4:28:50 time: 1.7101 data_time: 0.0145 memory: 11774 loss: 0.1900 2025/03/23 20:44:23 - mmengine - INFO - Iter(train) [ 8130/19176] lr: 1.2907e-05 eta: 4:28:39 time: 1.7020 data_time: 0.0146 memory: 11817 loss: 0.1800 2025/03/23 20:44:39 - mmengine - INFO - Iter(train) [ 8140/19176] lr: 1.2891e-05 eta: 4:28:26 time: 1.6130 data_time: 0.0147 memory: 11397 loss: 0.1769 2025/03/23 20:44:54 - mmengine - INFO - Iter(train) [ 8150/19176] lr: 1.2875e-05 eta: 4:28:13 time: 1.5519 data_time: 0.0144 memory: 11352 loss: 0.1876 2025/03/23 20:45:09 - mmengine - INFO - Iter(train) [ 8160/19176] lr: 1.2859e-05 eta: 4:27:58 time: 1.4824 data_time: 0.0147 memory: 11138 loss: 0.2084 2025/03/23 20:45:23 - mmengine - INFO - Iter(train) [ 8170/19176] lr: 1.2842e-05 eta: 4:27:42 time: 1.3468 data_time: 0.0136 memory: 10902 loss: 0.2092 2025/03/23 20:45:33 - mmengine - INFO - Iter(train) [ 8180/19176] lr: 1.2826e-05 eta: 4:27:23 time: 1.0956 data_time: 0.0127 memory: 10393 loss: 0.1757 2025/03/23 20:45:42 - mmengine - INFO - Iter(train) [ 8190/19176] lr: 1.2810e-05 eta: 4:27:00 time: 0.8568 data_time: 0.0114 memory: 9993 loss: 0.1897 2025/03/23 20:45:56 - mmengine - INFO - Iter(train) [ 8200/19176] lr: 1.2794e-05 eta: 4:26:45 time: 1.4391 data_time: 0.0120 memory: 13524 loss: 0.2121 2025/03/23 20:46:15 - mmengine - INFO - Iter(train) [ 8210/19176] lr: 1.2778e-05 eta: 4:26:36 time: 1.8795 data_time: 0.0145 memory: 12525 loss: 0.1769 2025/03/23 20:46:33 - mmengine - INFO - Iter(train) [ 8220/19176] lr: 1.2761e-05 eta: 4:26:26 time: 1.7530 data_time: 0.0147 memory: 12093 loss: 0.1753 2025/03/23 20:46:49 - mmengine - INFO - Iter(train) [ 8230/19176] lr: 1.2745e-05 eta: 4:26:14 time: 1.6690 data_time: 0.0144 memory: 11869 loss: 0.1778 2025/03/23 20:47:06 - mmengine - INFO - Iter(train) [ 8240/19176] lr: 1.2729e-05 eta: 4:26:02 time: 1.6235 data_time: 0.0146 memory: 11415 loss: 0.1863 2025/03/23 20:47:21 - mmengine - INFO - Iter(train) [ 8250/19176] lr: 1.2713e-05 eta: 4:25:48 time: 1.5474 data_time: 0.0145 memory: 11291 loss: 0.1994 2025/03/23 20:47:36 - mmengine - INFO - Iter(train) [ 8260/19176] lr: 1.2696e-05 eta: 4:25:34 time: 1.4740 data_time: 0.0147 memory: 11179 loss: 0.1968 2025/03/23 20:47:50 - mmengine - INFO - Iter(train) [ 8270/19176] lr: 1.2680e-05 eta: 4:25:18 time: 1.3976 data_time: 0.0150 memory: 10927 loss: 0.1923 2025/03/23 20:48:02 - mmengine - INFO - Iter(train) [ 8280/19176] lr: 1.2664e-05 eta: 4:25:01 time: 1.2461 data_time: 0.0142 memory: 10739 loss: 0.2278 2025/03/23 20:48:12 - mmengine - INFO - Iter(train) [ 8290/19176] lr: 1.2648e-05 eta: 4:24:40 time: 0.9606 data_time: 0.0123 memory: 10150 loss: 0.2155 2025/03/23 20:48:26 - mmengine - INFO - Iter(train) [ 8300/19176] lr: 1.2631e-05 eta: 4:24:24 time: 1.3985 data_time: 0.0129 memory: 12777 loss: 0.1939 2025/03/23 20:48:44 - mmengine - INFO - Iter(train) [ 8310/19176] lr: 1.2615e-05 eta: 4:24:14 time: 1.7697 data_time: 0.0146 memory: 12077 loss: 0.1698 2025/03/23 20:49:01 - mmengine - INFO - Iter(train) [ 8320/19176] lr: 1.2599e-05 eta: 4:24:02 time: 1.6990 data_time: 0.0148 memory: 11776 loss: 0.1783 2025/03/23 20:49:17 - mmengine - INFO - Iter(train) [ 8330/19176] lr: 1.2582e-05 eta: 4:23:50 time: 1.6257 data_time: 0.0149 memory: 11479 loss: 0.2115 2025/03/23 20:49:32 - mmengine - INFO - Iter(train) [ 8340/19176] lr: 1.2566e-05 eta: 4:23:37 time: 1.5504 data_time: 0.0145 memory: 11265 loss: 0.2160 2025/03/23 20:49:47 - mmengine - INFO - Iter(train) [ 8350/19176] lr: 1.2550e-05 eta: 4:23:23 time: 1.5085 data_time: 0.0146 memory: 11180 loss: 0.1943 2025/03/23 20:50:02 - mmengine - INFO - Iter(train) [ 8360/19176] lr: 1.2533e-05 eta: 4:23:07 time: 1.4082 data_time: 0.0145 memory: 11044 loss: 0.3496 2025/03/23 20:50:13 - mmengine - INFO - Iter(train) [ 8370/19176] lr: 1.2517e-05 eta: 4:22:49 time: 1.1436 data_time: 0.0130 memory: 10441 loss: 0.2100 2025/03/23 20:50:23 - mmengine - INFO - Iter(train) [ 8380/19176] lr: 1.2501e-05 eta: 4:22:28 time: 1.0225 data_time: 0.0124 memory: 10145 loss: 0.2215 2025/03/23 20:50:32 - mmengine - INFO - Iter(train) [ 8390/19176] lr: 1.2484e-05 eta: 4:22:06 time: 0.8749 data_time: 0.0119 memory: 9893 loss: 0.2074 2025/03/23 20:50:48 - mmengine - INFO - Iter(train) [ 8400/19176] lr: 1.2468e-05 eta: 4:21:53 time: 1.5813 data_time: 0.0123 memory: 18682 loss: 0.2029 2025/03/23 20:51:05 - mmengine - INFO - Iter(train) [ 8410/19176] lr: 1.2452e-05 eta: 4:21:43 time: 1.7584 data_time: 0.0144 memory: 12105 loss: 0.1826 2025/03/23 20:51:22 - mmengine - INFO - Iter(train) [ 8420/19176] lr: 1.2435e-05 eta: 4:21:31 time: 1.6800 data_time: 0.0148 memory: 11668 loss: 0.1621 2025/03/23 20:51:39 - mmengine - INFO - Iter(train) [ 8430/19176] lr: 1.2419e-05 eta: 4:21:19 time: 1.6502 data_time: 0.0147 memory: 11706 loss: 0.1504 2025/03/23 20:51:55 - mmengine - INFO - Iter(train) [ 8440/19176] lr: 1.2402e-05 eta: 4:21:06 time: 1.5896 data_time: 0.0142 memory: 11375 loss: 0.1939 2025/03/23 20:52:10 - mmengine - INFO - Iter(train) [ 8450/19176] lr: 1.2386e-05 eta: 4:20:52 time: 1.5361 data_time: 0.0147 memory: 11264 loss: 0.1833 2025/03/23 20:52:25 - mmengine - INFO - Iter(train) [ 8460/19176] lr: 1.2370e-05 eta: 4:20:38 time: 1.4844 data_time: 0.0143 memory: 11226 loss: 0.1979 2025/03/23 20:52:39 - mmengine - INFO - Iter(train) [ 8470/19176] lr: 1.2353e-05 eta: 4:20:22 time: 1.3909 data_time: 0.0139 memory: 10925 loss: 0.2103 2025/03/23 20:52:51 - mmengine - INFO - Iter(train) [ 8480/19176] lr: 1.2337e-05 eta: 4:20:05 time: 1.2015 data_time: 0.0133 memory: 10585 loss: 0.1983 2025/03/23 20:53:01 - mmengine - INFO - Iter(train) [ 8490/19176] lr: 1.2320e-05 eta: 4:19:45 time: 1.0526 data_time: 0.0126 memory: 10228 loss: 0.1930 2025/03/23 20:53:19 - mmengine - INFO - Iter(train) [ 8500/19176] lr: 1.2304e-05 eta: 4:19:34 time: 1.7513 data_time: 0.0137 memory: 18234 loss: 0.2250 2025/03/23 20:53:36 - mmengine - INFO - Iter(train) [ 8510/19176] lr: 1.2287e-05 eta: 4:19:23 time: 1.7582 data_time: 0.0145 memory: 12082 loss: 0.1645 2025/03/23 20:53:53 - mmengine - INFO - Iter(train) [ 8520/19176] lr: 1.2271e-05 eta: 4:19:11 time: 1.6824 data_time: 0.0146 memory: 11690 loss: 0.1863 2025/03/23 20:54:09 - mmengine - INFO - Iter(train) [ 8530/19176] lr: 1.2255e-05 eta: 4:18:59 time: 1.6050 data_time: 0.0144 memory: 11412 loss: 0.1837 2025/03/23 20:54:25 - mmengine - INFO - Iter(train) [ 8540/19176] lr: 1.2238e-05 eta: 4:18:45 time: 1.5779 data_time: 0.0143 memory: 11337 loss: 0.1632 2025/03/23 20:54:40 - mmengine - INFO - Iter(train) [ 8550/19176] lr: 1.2222e-05 eta: 4:18:31 time: 1.4799 data_time: 0.0144 memory: 11160 loss: 0.2015 2025/03/23 20:54:54 - mmengine - INFO - Iter(train) [ 8560/19176] lr: 1.2205e-05 eta: 4:18:16 time: 1.4448 data_time: 0.0143 memory: 11136 loss: 0.1975 2025/03/23 20:55:07 - mmengine - INFO - Iter(train) [ 8570/19176] lr: 1.2189e-05 eta: 4:18:00 time: 1.3059 data_time: 0.0139 memory: 10823 loss: 0.2102 2025/03/23 20:55:19 - mmengine - INFO - Iter(train) [ 8580/19176] lr: 1.2172e-05 eta: 4:17:41 time: 1.1359 data_time: 0.0125 memory: 10404 loss: 0.1812 2025/03/23 20:55:29 - mmengine - INFO - Iter(train) [ 8590/19176] lr: 1.2156e-05 eta: 4:17:21 time: 1.0228 data_time: 0.0125 memory: 10175 loss: 0.2133 2025/03/23 20:55:44 - mmengine - INFO - Iter(train) [ 8600/19176] lr: 1.2139e-05 eta: 4:17:07 time: 1.5027 data_time: 0.0156 memory: 13467 loss: 0.1757 2025/03/23 20:56:01 - mmengine - INFO - Iter(train) [ 8610/19176] lr: 1.2123e-05 eta: 4:16:56 time: 1.7498 data_time: 0.0146 memory: 12054 loss: 0.1677 2025/03/23 20:56:18 - mmengine - INFO - Iter(train) [ 8620/19176] lr: 1.2106e-05 eta: 4:16:44 time: 1.6992 data_time: 0.0147 memory: 11719 loss: 0.1739 2025/03/23 20:56:35 - mmengine - INFO - Iter(train) [ 8630/19176] lr: 1.2090e-05 eta: 4:16:32 time: 1.6356 data_time: 0.0147 memory: 11497 loss: 0.1797 2025/03/23 20:56:51 - mmengine - INFO - Iter(train) [ 8640/19176] lr: 1.2073e-05 eta: 4:16:19 time: 1.5904 data_time: 0.0147 memory: 11344 loss: 0.1954 2025/03/23 20:57:06 - mmengine - INFO - Iter(train) [ 8650/19176] lr: 1.2057e-05 eta: 4:16:05 time: 1.5022 data_time: 0.0142 memory: 11166 loss: 0.2014 2025/03/23 20:57:20 - mmengine - INFO - Iter(train) [ 8660/19176] lr: 1.2040e-05 eta: 4:15:51 time: 1.4702 data_time: 0.0147 memory: 11088 loss: 0.2537 2025/03/23 20:57:33 - mmengine - INFO - Iter(train) [ 8670/19176] lr: 1.2024e-05 eta: 4:15:34 time: 1.2941 data_time: 0.0137 memory: 10718 loss: 0.2736 2025/03/23 20:57:45 - mmengine - INFO - Iter(train) [ 8680/19176] lr: 1.2007e-05 eta: 4:15:15 time: 1.1231 data_time: 0.0127 memory: 10312 loss: 0.1913 2025/03/23 20:57:53 - mmengine - INFO - Iter(train) [ 8690/19176] lr: 1.1991e-05 eta: 4:14:54 time: 0.8866 data_time: 0.0120 memory: 10083 loss: 0.1964 2025/03/23 20:58:07 - mmengine - INFO - Iter(train) [ 8700/19176] lr: 1.1974e-05 eta: 4:14:37 time: 1.3209 data_time: 0.0126 memory: 13317 loss: 0.1912 2025/03/23 20:58:24 - mmengine - INFO - Iter(train) [ 8710/19176] lr: 1.1957e-05 eta: 4:14:26 time: 1.7483 data_time: 0.0146 memory: 11903 loss: 0.1741 2025/03/23 20:58:41 - mmengine - INFO - Iter(train) [ 8720/19176] lr: 1.1941e-05 eta: 4:14:15 time: 1.7072 data_time: 0.0147 memory: 12229 loss: 0.1646 2025/03/23 20:58:57 - mmengine - INFO - Iter(train) [ 8730/19176] lr: 1.1924e-05 eta: 4:14:02 time: 1.6276 data_time: 0.0143 memory: 11439 loss: 0.1565 2025/03/23 20:59:13 - mmengine - INFO - Iter(train) [ 8740/19176] lr: 1.1908e-05 eta: 4:13:49 time: 1.5828 data_time: 0.0147 memory: 11339 loss: 0.1751 2025/03/23 20:59:28 - mmengine - INFO - Iter(train) [ 8750/19176] lr: 1.1891e-05 eta: 4:13:35 time: 1.5222 data_time: 0.0145 memory: 11221 loss: 0.1792 2025/03/23 20:59:43 - mmengine - INFO - Iter(train) [ 8760/19176] lr: 1.1875e-05 eta: 4:13:21 time: 1.4637 data_time: 0.0143 memory: 11107 loss: 0.1742 2025/03/23 20:59:57 - mmengine - INFO - Iter(train) [ 8770/19176] lr: 1.1858e-05 eta: 4:13:05 time: 1.3675 data_time: 0.0144 memory: 10858 loss: 0.2484 2025/03/23 21:00:08 - mmengine - INFO - Iter(train) [ 8780/19176] lr: 1.1841e-05 eta: 4:12:46 time: 1.1230 data_time: 0.0128 memory: 10478 loss: 0.1935 2025/03/23 21:00:17 - mmengine - INFO - Iter(train) [ 8790/19176] lr: 1.1825e-05 eta: 4:12:26 time: 0.9384 data_time: 0.0125 memory: 10144 loss: 0.1904 2025/03/23 21:00:33 - mmengine - INFO - Iter(train) [ 8800/19176] lr: 1.1808e-05 eta: 4:12:12 time: 1.5692 data_time: 0.0129 memory: 15963 loss: 0.2011 2025/03/23 21:00:51 - mmengine - INFO - Iter(train) [ 8810/19176] lr: 1.1792e-05 eta: 4:12:02 time: 1.8388 data_time: 0.0146 memory: 12724 loss: 0.1845 2025/03/23 21:01:09 - mmengine - INFO - Iter(train) [ 8820/19176] lr: 1.1775e-05 eta: 4:11:51 time: 1.7306 data_time: 0.0163 memory: 11894 loss: 0.1778 2025/03/23 21:01:25 - mmengine - INFO - Iter(train) [ 8830/19176] lr: 1.1758e-05 eta: 4:11:38 time: 1.6138 data_time: 0.0146 memory: 11477 loss: 0.2050 2025/03/23 21:01:41 - mmengine - INFO - Iter(train) [ 8840/19176] lr: 1.1742e-05 eta: 4:11:25 time: 1.5812 data_time: 0.0144 memory: 11330 loss: 0.1745 2025/03/23 21:01:56 - mmengine - INFO - Iter(train) [ 8850/19176] lr: 1.1725e-05 eta: 4:11:11 time: 1.5126 data_time: 0.0146 memory: 11212 loss: 0.2196 2025/03/23 21:02:10 - mmengine - INFO - Iter(train) [ 8860/19176] lr: 1.1708e-05 eta: 4:10:56 time: 1.4549 data_time: 0.0146 memory: 11034 loss: 0.1982 2025/03/23 21:02:24 - mmengine - INFO - Iter(train) [ 8870/19176] lr: 1.1692e-05 eta: 4:10:41 time: 1.3695 data_time: 0.0138 memory: 10901 loss: 0.1899 2025/03/23 21:02:36 - mmengine - INFO - Iter(train) [ 8880/19176] lr: 1.1675e-05 eta: 4:10:23 time: 1.1856 data_time: 0.0133 memory: 10520 loss: 0.2318 2025/03/23 21:02:46 - mmengine - INFO - Iter(train) [ 8890/19176] lr: 1.1658e-05 eta: 4:10:03 time: 0.9688 data_time: 0.0126 memory: 10059 loss: 0.2121 2025/03/23 21:02:59 - mmengine - INFO - Iter(train) [ 8900/19176] lr: 1.1642e-05 eta: 4:09:47 time: 1.3742 data_time: 0.0122 memory: 14123 loss: 0.1884 2025/03/23 21:03:18 - mmengine - INFO - Iter(train) [ 8910/19176] lr: 1.1625e-05 eta: 4:09:37 time: 1.8257 data_time: 0.0146 memory: 12286 loss: 0.1728 2025/03/23 21:03:35 - mmengine - INFO - Iter(train) [ 8920/19176] lr: 1.1608e-05 eta: 4:09:25 time: 1.7277 data_time: 0.0150 memory: 11942 loss: 0.1765 2025/03/23 21:03:51 - mmengine - INFO - Iter(train) [ 8930/19176] lr: 1.1592e-05 eta: 4:09:13 time: 1.6380 data_time: 0.0147 memory: 11497 loss: 0.2106 2025/03/23 21:04:07 - mmengine - INFO - Iter(train) [ 8940/19176] lr: 1.1575e-05 eta: 4:09:00 time: 1.6059 data_time: 0.0143 memory: 11389 loss: 0.1845 2025/03/23 21:04:23 - mmengine - INFO - Iter(train) [ 8950/19176] lr: 1.1558e-05 eta: 4:08:46 time: 1.5522 data_time: 0.0144 memory: 11297 loss: 0.2086 2025/03/23 21:04:38 - mmengine - INFO - Iter(train) [ 8960/19176] lr: 1.1542e-05 eta: 4:08:32 time: 1.4860 data_time: 0.0142 memory: 11133 loss: 0.2422 2025/03/23 21:04:51 - mmengine - INFO - Iter(train) [ 8970/19176] lr: 1.1525e-05 eta: 4:08:16 time: 1.3238 data_time: 0.0140 memory: 10897 loss: 0.2532 2025/03/23 21:05:02 - mmengine - INFO - Iter(train) [ 8980/19176] lr: 1.1508e-05 eta: 4:07:57 time: 1.1038 data_time: 0.0124 memory: 10360 loss: 0.1934 2025/03/23 21:05:12 - mmengine - INFO - Iter(train) [ 8990/19176] lr: 1.1492e-05 eta: 4:07:37 time: 0.9574 data_time: 0.0121 memory: 10044 loss: 0.2160 2025/03/23 21:05:28 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 21:05:28 - mmengine - INFO - Iter(train) [ 9000/19176] lr: 1.1475e-05 eta: 4:07:24 time: 1.6402 data_time: 0.0127 memory: 16574 loss: 0.1701 2025/03/23 21:05:28 - mmengine - INFO - Saving checkpoint at 9000 iterations 2025/03/23 21:05:46 - mmengine - INFO - Iter(train) [ 9010/19176] lr: 1.1458e-05 eta: 4:07:14 time: 1.8475 data_time: 0.0911 memory: 12061 loss: 0.1837 2025/03/23 21:06:04 - mmengine - INFO - Iter(train) [ 9020/19176] lr: 1.1442e-05 eta: 4:07:02 time: 1.7035 data_time: 0.0145 memory: 11813 loss: 0.1707 2025/03/23 21:06:20 - mmengine - INFO - Iter(train) [ 9030/19176] lr: 1.1425e-05 eta: 4:06:50 time: 1.6448 data_time: 0.0143 memory: 11486 loss: 0.1583 2025/03/23 21:06:36 - mmengine - INFO - Iter(train) [ 9040/19176] lr: 1.1408e-05 eta: 4:06:36 time: 1.5727 data_time: 0.0148 memory: 11333 loss: 0.1860 2025/03/23 21:06:51 - mmengine - INFO - Iter(train) [ 9050/19176] lr: 1.1391e-05 eta: 4:06:23 time: 1.5352 data_time: 0.0150 memory: 11246 loss: 0.1849 2025/03/23 21:07:06 - mmengine - INFO - Iter(train) [ 9060/19176] lr: 1.1375e-05 eta: 4:06:08 time: 1.4921 data_time: 0.0149 memory: 11149 loss: 0.2018 2025/03/23 21:07:20 - mmengine - INFO - Iter(train) [ 9070/19176] lr: 1.1358e-05 eta: 4:05:53 time: 1.3849 data_time: 0.0144 memory: 11014 loss: 0.2706 2025/03/23 21:07:31 - mmengine - INFO - Iter(train) [ 9080/19176] lr: 1.1341e-05 eta: 4:05:34 time: 1.0745 data_time: 0.0126 memory: 10445 loss: 0.1838 2025/03/23 21:07:38 - mmengine - INFO - Iter(train) [ 9090/19176] lr: 1.1324e-05 eta: 4:05:12 time: 0.7863 data_time: 0.0116 memory: 9749 loss: 0.2086 2025/03/23 21:07:54 - mmengine - INFO - Iter(train) [ 9100/19176] lr: 1.1308e-05 eta: 4:04:59 time: 1.5836 data_time: 0.0132 memory: 16441 loss: 0.2066 2025/03/23 21:08:13 - mmengine - INFO - Iter(train) [ 9110/19176] lr: 1.1291e-05 eta: 4:04:48 time: 1.8338 data_time: 0.0145 memory: 12681 loss: 0.1888 2025/03/23 21:08:30 - mmengine - INFO - Iter(train) [ 9120/19176] lr: 1.1274e-05 eta: 4:04:37 time: 1.7172 data_time: 0.0145 memory: 12317 loss: 0.2109 2025/03/23 21:08:46 - mmengine - INFO - Iter(train) [ 9130/19176] lr: 1.1257e-05 eta: 4:04:24 time: 1.6568 data_time: 0.0145 memory: 11593 loss: 0.1747 2025/03/23 21:09:02 - mmengine - INFO - Iter(train) [ 9140/19176] lr: 1.1241e-05 eta: 4:04:11 time: 1.5897 data_time: 0.0147 memory: 11369 loss: 0.1963 2025/03/23 21:09:17 - mmengine - INFO - Iter(train) [ 9150/19176] lr: 1.1224e-05 eta: 4:03:57 time: 1.5001 data_time: 0.0146 memory: 11218 loss: 0.1606 2025/03/23 21:09:31 - mmengine - INFO - Iter(train) [ 9160/19176] lr: 1.1207e-05 eta: 4:03:42 time: 1.4156 data_time: 0.0144 memory: 11001 loss: 0.2730 2025/03/23 21:09:44 - mmengine - INFO - Iter(train) [ 9170/19176] lr: 1.1190e-05 eta: 4:03:25 time: 1.2934 data_time: 0.0141 memory: 10809 loss: 0.2510 2025/03/23 21:09:55 - mmengine - INFO - Iter(train) [ 9180/19176] lr: 1.1174e-05 eta: 4:03:07 time: 1.0674 data_time: 0.0123 memory: 10408 loss: 0.1780 2025/03/23 21:10:04 - mmengine - INFO - Iter(train) [ 9190/19176] lr: 1.1157e-05 eta: 4:02:46 time: 0.8859 data_time: 0.0119 memory: 9948 loss: 0.2249 2025/03/23 21:10:17 - mmengine - INFO - Iter(train) [ 9200/19176] lr: 1.1140e-05 eta: 4:02:30 time: 1.3637 data_time: 0.0118 memory: 13532 loss: 0.2156 2025/03/23 21:10:36 - mmengine - INFO - Iter(train) [ 9210/19176] lr: 1.1123e-05 eta: 4:02:20 time: 1.8540 data_time: 0.0144 memory: 12379 loss: 0.1684 2025/03/23 21:10:53 - mmengine - INFO - Iter(train) [ 9220/19176] lr: 1.1107e-05 eta: 4:02:08 time: 1.7171 data_time: 0.0144 memory: 11930 loss: 0.1641 2025/03/23 21:11:10 - mmengine - INFO - Iter(train) [ 9230/19176] lr: 1.1090e-05 eta: 4:01:56 time: 1.6736 data_time: 0.0141 memory: 11617 loss: 0.1681 2025/03/23 21:11:26 - mmengine - INFO - Iter(train) [ 9240/19176] lr: 1.1073e-05 eta: 4:01:43 time: 1.6343 data_time: 0.0143 memory: 11475 loss: 0.1910 2025/03/23 21:11:42 - mmengine - INFO - Iter(train) [ 9250/19176] lr: 1.1056e-05 eta: 4:01:29 time: 1.5278 data_time: 0.0139 memory: 11260 loss: 0.1964 2025/03/23 21:11:57 - mmengine - INFO - Iter(train) [ 9260/19176] lr: 1.1039e-05 eta: 4:01:15 time: 1.5047 data_time: 0.0154 memory: 11213 loss: 0.1962 2025/03/23 21:12:11 - mmengine - INFO - Iter(train) [ 9270/19176] lr: 1.1023e-05 eta: 4:01:00 time: 1.4230 data_time: 0.0146 memory: 11011 loss: 0.2042 2025/03/23 21:12:22 - mmengine - INFO - Iter(train) [ 9280/19176] lr: 1.1006e-05 eta: 4:00:42 time: 1.1540 data_time: 0.0133 memory: 10482 loss: 0.1794 2025/03/23 21:12:32 - mmengine - INFO - Iter(train) [ 9290/19176] lr: 1.0989e-05 eta: 4:00:22 time: 0.9452 data_time: 0.0125 memory: 10017 loss: 0.2670 2025/03/23 21:12:48 - mmengine - INFO - Iter(train) [ 9300/19176] lr: 1.0972e-05 eta: 4:00:09 time: 1.5797 data_time: 0.0130 memory: 16945 loss: 0.1877 2025/03/23 21:13:06 - mmengine - INFO - Iter(train) [ 9310/19176] lr: 1.0955e-05 eta: 3:59:58 time: 1.8056 data_time: 0.0150 memory: 12269 loss: 0.1803 2025/03/23 21:13:23 - mmengine - INFO - Iter(train) [ 9320/19176] lr: 1.0939e-05 eta: 3:59:46 time: 1.7077 data_time: 0.0148 memory: 11753 loss: 0.1482 2025/03/23 21:13:39 - mmengine - INFO - Iter(train) [ 9330/19176] lr: 1.0922e-05 eta: 3:59:33 time: 1.6463 data_time: 0.0145 memory: 11517 loss: 0.1634 2025/03/23 21:13:55 - mmengine - INFO - Iter(train) [ 9340/19176] lr: 1.0905e-05 eta: 3:59:20 time: 1.5816 data_time: 0.0146 memory: 11416 loss: 0.1766 2025/03/23 21:14:10 - mmengine - INFO - Iter(train) [ 9350/19176] lr: 1.0888e-05 eta: 3:59:06 time: 1.5177 data_time: 0.0146 memory: 11237 loss: 0.1808 2025/03/23 21:14:25 - mmengine - INFO - Iter(train) [ 9360/19176] lr: 1.0871e-05 eta: 3:58:51 time: 1.4352 data_time: 0.0150 memory: 11029 loss: 0.2368 2025/03/23 21:14:38 - mmengine - INFO - Iter(train) [ 9370/19176] lr: 1.0854e-05 eta: 3:58:36 time: 1.3879 data_time: 0.0137 memory: 10942 loss: 0.2185 2025/03/23 21:14:50 - mmengine - INFO - Iter(train) [ 9380/19176] lr: 1.0838e-05 eta: 3:58:18 time: 1.1948 data_time: 0.0124 memory: 10771 loss: 0.2082 2025/03/23 21:15:01 - mmengine - INFO - Iter(train) [ 9390/19176] lr: 1.0821e-05 eta: 3:57:59 time: 1.0105 data_time: 0.0124 memory: 10151 loss: 0.2203 2025/03/23 21:15:16 - mmengine - INFO - Iter(train) [ 9400/19176] lr: 1.0804e-05 eta: 3:57:46 time: 1.5875 data_time: 0.0130 memory: 14486 loss: 0.1759 2025/03/23 21:15:34 - mmengine - INFO - Iter(train) [ 9410/19176] lr: 1.0787e-05 eta: 3:57:35 time: 1.7864 data_time: 0.0147 memory: 12065 loss: 0.1910 2025/03/23 21:15:51 - mmengine - INFO - Iter(train) [ 9420/19176] lr: 1.0770e-05 eta: 3:57:23 time: 1.7007 data_time: 0.0143 memory: 11767 loss: 0.1558 2025/03/23 21:16:07 - mmengine - INFO - Iter(train) [ 9430/19176] lr: 1.0753e-05 eta: 3:57:10 time: 1.6228 data_time: 0.0140 memory: 11535 loss: 0.1947 2025/03/23 21:16:23 - mmengine - INFO - Iter(train) [ 9440/19176] lr: 1.0737e-05 eta: 3:56:56 time: 1.5598 data_time: 0.0140 memory: 11428 loss: 0.1765 2025/03/23 21:16:38 - mmengine - INFO - Iter(train) [ 9450/19176] lr: 1.0720e-05 eta: 3:56:42 time: 1.5207 data_time: 0.0143 memory: 11188 loss: 0.2168 2025/03/23 21:16:53 - mmengine - INFO - Iter(train) [ 9460/19176] lr: 1.0703e-05 eta: 3:56:27 time: 1.4578 data_time: 0.0143 memory: 11064 loss: 0.2009 2025/03/23 21:17:07 - mmengine - INFO - Iter(train) [ 9470/19176] lr: 1.0686e-05 eta: 3:56:12 time: 1.3859 data_time: 0.0141 memory: 10915 loss: 0.3763 2025/03/23 21:17:18 - mmengine - INFO - Iter(train) [ 9480/19176] lr: 1.0669e-05 eta: 3:55:54 time: 1.1509 data_time: 0.0135 memory: 10583 loss: 0.2234 2025/03/23 21:17:27 - mmengine - INFO - Iter(train) [ 9490/19176] lr: 1.0652e-05 eta: 3:55:34 time: 0.8648 data_time: 0.0116 memory: 9990 loss: 0.1928 2025/03/23 21:17:42 - mmengine - INFO - Iter(train) [ 9500/19176] lr: 1.0635e-05 eta: 3:55:19 time: 1.4688 data_time: 0.0125 memory: 15007 loss: 0.2056 2025/03/23 21:17:59 - mmengine - INFO - Iter(train) [ 9510/19176] lr: 1.0619e-05 eta: 3:55:08 time: 1.7548 data_time: 0.0143 memory: 12148 loss: 0.1707 2025/03/23 21:18:16 - mmengine - INFO - Iter(train) [ 9520/19176] lr: 1.0602e-05 eta: 3:54:55 time: 1.6569 data_time: 0.0145 memory: 11573 loss: 0.2119 2025/03/23 21:18:32 - mmengine - INFO - Iter(train) [ 9530/19176] lr: 1.0585e-05 eta: 3:54:42 time: 1.5968 data_time: 0.0146 memory: 11347 loss: 0.1835 2025/03/23 21:18:47 - mmengine - INFO - Iter(train) [ 9540/19176] lr: 1.0568e-05 eta: 3:54:28 time: 1.5499 data_time: 0.0145 memory: 11279 loss: 0.1989 2025/03/23 21:19:02 - mmengine - INFO - Iter(train) [ 9550/19176] lr: 1.0551e-05 eta: 3:54:14 time: 1.5147 data_time: 0.0142 memory: 11240 loss: 0.1829 2025/03/23 21:19:17 - mmengine - INFO - Iter(train) [ 9560/19176] lr: 1.0534e-05 eta: 3:53:59 time: 1.4515 data_time: 0.0143 memory: 11039 loss: 0.1876 2025/03/23 21:19:30 - mmengine - INFO - Iter(train) [ 9570/19176] lr: 1.0517e-05 eta: 3:53:43 time: 1.2895 data_time: 0.0141 memory: 10871 loss: 0.3302 2025/03/23 21:19:40 - mmengine - INFO - Iter(train) [ 9580/19176] lr: 1.0501e-05 eta: 3:53:24 time: 1.0219 data_time: 0.0125 memory: 10263 loss: 0.1999 2025/03/23 21:19:54 - mmengine - INFO - Iter(train) [ 9590/19176] lr: 1.0484e-05 eta: 3:53:09 time: 1.3614 data_time: 0.2610 memory: 19198 loss: 0.2275 2025/03/23 21:20:12 - mmengine - INFO - Iter(train) [ 9600/19176] lr: 1.0467e-05 eta: 3:52:58 time: 1.8464 data_time: 0.0148 memory: 12391 loss: 0.1628 2025/03/23 21:20:29 - mmengine - INFO - Iter(train) [ 9610/19176] lr: 1.0450e-05 eta: 3:52:46 time: 1.7320 data_time: 0.0154 memory: 11930 loss: 0.1521 2025/03/23 21:20:46 - mmengine - INFO - Iter(train) [ 9620/19176] lr: 1.0433e-05 eta: 3:52:33 time: 1.6711 data_time: 0.0147 memory: 11690 loss: 0.1626 2025/03/23 21:21:02 - mmengine - INFO - Iter(train) [ 9630/19176] lr: 1.0416e-05 eta: 3:52:20 time: 1.5966 data_time: 0.0148 memory: 11377 loss: 0.1775 2025/03/23 21:21:17 - mmengine - INFO - Iter(train) [ 9640/19176] lr: 1.0399e-05 eta: 3:52:06 time: 1.5318 data_time: 0.0145 memory: 11250 loss: 0.1661 2025/03/23 21:21:32 - mmengine - INFO - Iter(train) [ 9650/19176] lr: 1.0382e-05 eta: 3:51:52 time: 1.4838 data_time: 0.0145 memory: 11129 loss: 0.1698 2025/03/23 21:21:46 - mmengine - INFO - Iter(train) [ 9660/19176] lr: 1.0366e-05 eta: 3:51:36 time: 1.3739 data_time: 0.0142 memory: 10891 loss: 0.1651 2025/03/23 21:21:58 - mmengine - INFO - Iter(train) [ 9670/19176] lr: 1.0349e-05 eta: 3:51:19 time: 1.1614 data_time: 0.0132 memory: 10584 loss: 0.1755 2025/03/23 21:22:07 - mmengine - INFO - Iter(train) [ 9680/19176] lr: 1.0332e-05 eta: 3:50:59 time: 0.9601 data_time: 0.0123 memory: 10128 loss: 0.1769 2025/03/23 21:22:17 - mmengine - INFO - Iter(train) [ 9690/19176] lr: 1.0315e-05 eta: 3:50:40 time: 1.0092 data_time: 0.0117 memory: 13555 loss: 0.1724 2025/03/23 21:22:35 - mmengine - INFO - Iter(train) [ 9700/19176] lr: 1.0298e-05 eta: 3:50:29 time: 1.7902 data_time: 0.0148 memory: 12379 loss: 0.1516 2025/03/23 21:22:52 - mmengine - INFO - Iter(train) [ 9710/19176] lr: 1.0281e-05 eta: 3:50:17 time: 1.6962 data_time: 0.0150 memory: 11710 loss: 0.1792 2025/03/23 21:23:09 - mmengine - INFO - Iter(train) [ 9720/19176] lr: 1.0264e-05 eta: 3:50:04 time: 1.6653 data_time: 0.0149 memory: 11679 loss: 0.1886 2025/03/23 21:23:25 - mmengine - INFO - Iter(train) [ 9730/19176] lr: 1.0247e-05 eta: 3:49:51 time: 1.6091 data_time: 0.0147 memory: 11428 loss: 0.1764 2025/03/23 21:23:40 - mmengine - INFO - Iter(train) [ 9740/19176] lr: 1.0231e-05 eta: 3:49:37 time: 1.5198 data_time: 0.0147 memory: 11243 loss: 0.1688 2025/03/23 21:23:54 - mmengine - INFO - Iter(train) [ 9750/19176] lr: 1.0214e-05 eta: 3:49:22 time: 1.4270 data_time: 0.0146 memory: 11000 loss: 0.1659 2025/03/23 21:24:07 - mmengine - INFO - Iter(train) [ 9760/19176] lr: 1.0197e-05 eta: 3:49:06 time: 1.3111 data_time: 0.0136 memory: 10830 loss: 0.1927 2025/03/23 21:24:19 - mmengine - INFO - Iter(train) [ 9770/19176] lr: 1.0180e-05 eta: 3:48:48 time: 1.1570 data_time: 0.0127 memory: 10438 loss: 0.1832 2025/03/23 21:24:30 - mmengine - INFO - Iter(train) [ 9780/19176] lr: 1.0163e-05 eta: 3:48:30 time: 1.0602 data_time: 0.0130 memory: 10240 loss: 0.1723 2025/03/23 21:24:42 - mmengine - INFO - Iter(train) [ 9790/19176] lr: 1.0146e-05 eta: 3:48:14 time: 1.2669 data_time: 0.0120 memory: 18041 loss: 0.1687 2025/03/23 21:25:02 - mmengine - INFO - Iter(train) [ 9800/19176] lr: 1.0129e-05 eta: 3:48:04 time: 1.9784 data_time: 0.0148 memory: 14488 loss: 0.1701 2025/03/23 21:25:20 - mmengine - INFO - Iter(train) [ 9810/19176] lr: 1.0112e-05 eta: 3:47:52 time: 1.7777 data_time: 0.0154 memory: 12049 loss: 0.1527 2025/03/23 21:25:37 - mmengine - INFO - Iter(train) [ 9820/19176] lr: 1.0095e-05 eta: 3:47:40 time: 1.6831 data_time: 0.0148 memory: 11626 loss: 0.1510 2025/03/23 21:25:53 - mmengine - INFO - Iter(train) [ 9830/19176] lr: 1.0079e-05 eta: 3:47:27 time: 1.6416 data_time: 0.0149 memory: 11544 loss: 0.1408 2025/03/23 21:26:09 - mmengine - INFO - Iter(train) [ 9840/19176] lr: 1.0062e-05 eta: 3:47:14 time: 1.5792 data_time: 0.0146 memory: 11309 loss: 0.1543 2025/03/23 21:26:24 - mmengine - INFO - Iter(train) [ 9850/19176] lr: 1.0045e-05 eta: 3:47:00 time: 1.5559 data_time: 0.0146 memory: 11329 loss: 0.1660 2025/03/23 21:26:39 - mmengine - INFO - Iter(train) [ 9860/19176] lr: 1.0028e-05 eta: 3:46:45 time: 1.4642 data_time: 0.0147 memory: 11129 loss: 0.1616 2025/03/23 21:26:53 - mmengine - INFO - Iter(train) [ 9870/19176] lr: 1.0011e-05 eta: 3:46:30 time: 1.3551 data_time: 0.0142 memory: 10912 loss: 0.1748 2025/03/23 21:27:03 - mmengine - INFO - Iter(train) [ 9880/19176] lr: 9.9941e-06 eta: 3:46:12 time: 1.0830 data_time: 0.0126 memory: 10361 loss: 0.1664 2025/03/23 21:27:16 - mmengine - INFO - Iter(train) [ 9890/19176] lr: 9.9772e-06 eta: 3:45:55 time: 1.2470 data_time: 0.0117 memory: 18682 loss: 0.1683 2025/03/23 21:27:34 - mmengine - INFO - Iter(train) [ 9900/19176] lr: 9.9603e-06 eta: 3:45:44 time: 1.8438 data_time: 0.0147 memory: 12396 loss: 0.1681 2025/03/23 21:27:52 - mmengine - INFO - Iter(train) [ 9910/19176] lr: 9.9434e-06 eta: 3:45:32 time: 1.7226 data_time: 0.0149 memory: 11833 loss: 0.1523 2025/03/23 21:28:08 - mmengine - INFO - Iter(train) [ 9920/19176] lr: 9.9265e-06 eta: 3:45:19 time: 1.6748 data_time: 0.0149 memory: 11566 loss: 0.1676 2025/03/23 21:28:24 - mmengine - INFO - Iter(train) [ 9930/19176] lr: 9.9096e-06 eta: 3:45:06 time: 1.6052 data_time: 0.0146 memory: 11386 loss: 0.1550 2025/03/23 21:28:40 - mmengine - INFO - Iter(train) [ 9940/19176] lr: 9.8928e-06 eta: 3:44:52 time: 1.5546 data_time: 0.0145 memory: 11283 loss: 0.1596 2025/03/23 21:28:55 - mmengine - INFO - Iter(train) [ 9950/19176] lr: 9.8759e-06 eta: 3:44:38 time: 1.5034 data_time: 0.0146 memory: 11159 loss: 0.1678 2025/03/23 21:29:09 - mmengine - INFO - Iter(train) [ 9960/19176] lr: 9.8590e-06 eta: 3:44:23 time: 1.4162 data_time: 0.0143 memory: 10961 loss: 0.1935 2025/03/23 21:29:22 - mmengine - INFO - Iter(train) [ 9970/19176] lr: 9.8421e-06 eta: 3:44:07 time: 1.2860 data_time: 0.0136 memory: 10772 loss: 0.1731 2025/03/23 21:29:33 - mmengine - INFO - Iter(train) [ 9980/19176] lr: 9.8252e-06 eta: 3:43:48 time: 1.0619 data_time: 0.0126 memory: 10401 loss: 0.1594 2025/03/23 21:29:43 - mmengine - INFO - Iter(train) [ 9990/19176] lr: 9.8083e-06 eta: 3:43:30 time: 1.0745 data_time: 0.0114 memory: 16044 loss: 0.1676 2025/03/23 21:30:02 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 21:30:02 - mmengine - INFO - Iter(train) [10000/19176] lr: 9.7914e-06 eta: 3:43:19 time: 1.8419 data_time: 0.0144 memory: 12503 loss: 0.1540 2025/03/23 21:30:02 - mmengine - INFO - Saving checkpoint at 10000 iterations 2025/03/23 21:30:20 - mmengine - INFO - Iter(train) [10010/19176] lr: 9.7745e-06 eta: 3:43:08 time: 1.8045 data_time: 0.1194 memory: 11822 loss: 0.1612 2025/03/23 21:30:36 - mmengine - INFO - Iter(train) [10020/19176] lr: 9.7577e-06 eta: 3:42:55 time: 1.6529 data_time: 0.0149 memory: 11504 loss: 0.1718 2025/03/23 21:30:52 - mmengine - INFO - Iter(train) [10030/19176] lr: 9.7408e-06 eta: 3:42:41 time: 1.5730 data_time: 0.0141 memory: 11359 loss: 0.1741 2025/03/23 21:31:07 - mmengine - INFO - Iter(train) [10040/19176] lr: 9.7239e-06 eta: 3:42:27 time: 1.5181 data_time: 0.0148 memory: 11216 loss: 0.1788 2025/03/23 21:31:22 - mmengine - INFO - Iter(train) [10050/19176] lr: 9.7070e-06 eta: 3:42:13 time: 1.4502 data_time: 0.0145 memory: 11051 loss: 0.1643 2025/03/23 21:31:35 - mmengine - INFO - Iter(train) [10060/19176] lr: 9.6901e-06 eta: 3:41:57 time: 1.3478 data_time: 0.0146 memory: 10896 loss: 0.1566 2025/03/23 21:31:47 - mmengine - INFO - Iter(train) [10070/19176] lr: 9.6732e-06 eta: 3:41:39 time: 1.1387 data_time: 0.0123 memory: 10501 loss: 0.1795 2025/03/23 21:31:57 - mmengine - INFO - Iter(train) [10080/19176] lr: 9.6564e-06 eta: 3:41:21 time: 1.0182 data_time: 0.0123 memory: 10230 loss: 0.1555 2025/03/23 21:32:07 - mmengine - INFO - Iter(train) [10090/19176] lr: 9.6395e-06 eta: 3:41:02 time: 0.9868 data_time: 0.0111 memory: 14165 loss: 0.1834 2025/03/23 21:32:25 - mmengine - INFO - Iter(train) [10100/19176] lr: 9.6226e-06 eta: 3:40:50 time: 1.7909 data_time: 0.0147 memory: 12588 loss: 0.1599 2025/03/23 21:32:41 - mmengine - INFO - Iter(train) [10110/19176] lr: 9.6057e-06 eta: 3:40:37 time: 1.6461 data_time: 0.0143 memory: 11664 loss: 0.1723 2025/03/23 21:32:57 - mmengine - INFO - Iter(train) [10120/19176] lr: 9.5889e-06 eta: 3:40:24 time: 1.5780 data_time: 0.0147 memory: 11362 loss: 0.1954 2025/03/23 21:33:12 - mmengine - INFO - Iter(train) [10130/19176] lr: 9.5720e-06 eta: 3:40:10 time: 1.5137 data_time: 0.0145 memory: 11208 loss: 0.1737 2025/03/23 21:33:26 - mmengine - INFO - Iter(train) [10140/19176] lr: 9.5551e-06 eta: 3:39:55 time: 1.4452 data_time: 0.0150 memory: 11059 loss: 0.1527 2025/03/23 21:33:40 - mmengine - INFO - Iter(train) [10150/19176] lr: 9.5382e-06 eta: 3:39:39 time: 1.3586 data_time: 0.0137 memory: 10893 loss: 0.1607 2025/03/23 21:33:52 - mmengine - INFO - Iter(train) [10160/19176] lr: 9.5214e-06 eta: 3:39:23 time: 1.2372 data_time: 0.0135 memory: 10626 loss: 0.1950 2025/03/23 21:34:03 - mmengine - INFO - Iter(train) [10170/19176] lr: 9.5045e-06 eta: 3:39:05 time: 1.0632 data_time: 0.0121 memory: 10316 loss: 0.1727 2025/03/23 21:34:13 - mmengine - INFO - Iter(train) [10180/19176] lr: 9.4876e-06 eta: 3:38:46 time: 0.9621 data_time: 0.0120 memory: 10085 loss: 0.1710 2025/03/23 21:34:22 - mmengine - INFO - Iter(train) [10190/19176] lr: 9.4708e-06 eta: 3:38:26 time: 0.9108 data_time: 0.0109 memory: 14179 loss: 0.2030 2025/03/23 21:34:40 - mmengine - INFO - Iter(train) [10200/19176] lr: 9.4539e-06 eta: 3:38:15 time: 1.8230 data_time: 0.0143 memory: 12578 loss: 0.1640 2025/03/23 21:34:57 - mmengine - INFO - Iter(train) [10210/19176] lr: 9.4370e-06 eta: 3:38:03 time: 1.7228 data_time: 0.0141 memory: 11860 loss: 0.1600 2025/03/23 21:35:14 - mmengine - INFO - Iter(train) [10220/19176] lr: 9.4202e-06 eta: 3:37:50 time: 1.6593 data_time: 0.0139 memory: 11510 loss: 0.1813 2025/03/23 21:35:30 - mmengine - INFO - Iter(train) [10230/19176] lr: 9.4033e-06 eta: 3:37:36 time: 1.5982 data_time: 0.0135 memory: 11418 loss: 0.2003 2025/03/23 21:35:45 - mmengine - INFO - Iter(train) [10240/19176] lr: 9.3865e-06 eta: 3:37:23 time: 1.5508 data_time: 0.0143 memory: 11325 loss: 0.1910 2025/03/23 21:36:00 - mmengine - INFO - Iter(train) [10250/19176] lr: 9.3696e-06 eta: 3:37:08 time: 1.4817 data_time: 0.0144 memory: 11138 loss: 0.1615 2025/03/23 21:36:14 - mmengine - INFO - Iter(train) [10260/19176] lr: 9.3527e-06 eta: 3:36:53 time: 1.4173 data_time: 0.0140 memory: 11039 loss: 0.1868 2025/03/23 21:36:27 - mmengine - INFO - Iter(train) [10270/19176] lr: 9.3359e-06 eta: 3:36:37 time: 1.2476 data_time: 0.0135 memory: 10709 loss: 0.1874 2025/03/23 21:36:37 - mmengine - INFO - Iter(train) [10280/19176] lr: 9.3190e-06 eta: 3:36:19 time: 1.0686 data_time: 0.0122 memory: 10366 loss: 0.1605 2025/03/23 21:36:50 - mmengine - INFO - Iter(train) [10290/19176] lr: 9.3022e-06 eta: 3:36:03 time: 1.3028 data_time: 0.0118 memory: 18868 loss: 0.1848 2025/03/23 21:37:12 - mmengine - INFO - Iter(train) [10300/19176] lr: 9.2853e-06 eta: 3:35:54 time: 2.1339 data_time: 0.0141 memory: 13878 loss: 0.1669 2025/03/23 21:37:29 - mmengine - INFO - Iter(train) [10310/19176] lr: 9.2685e-06 eta: 3:35:42 time: 1.7444 data_time: 0.0142 memory: 12119 loss: 0.1572 2025/03/23 21:37:46 - mmengine - INFO - Iter(train) [10320/19176] lr: 9.2517e-06 eta: 3:35:29 time: 1.6576 data_time: 0.0143 memory: 11602 loss: 0.2014 2025/03/23 21:38:02 - mmengine - INFO - Iter(train) [10330/19176] lr: 9.2348e-06 eta: 3:35:16 time: 1.6290 data_time: 0.0141 memory: 11485 loss: 0.1560 2025/03/23 21:38:18 - mmengine - INFO - Iter(train) [10340/19176] lr: 9.2180e-06 eta: 3:35:02 time: 1.5701 data_time: 0.0141 memory: 11296 loss: 0.1533 2025/03/23 21:38:33 - mmengine - INFO - Iter(train) [10350/19176] lr: 9.2011e-06 eta: 3:34:48 time: 1.4957 data_time: 0.0143 memory: 11212 loss: 0.1791 2025/03/23 21:38:46 - mmengine - INFO - Iter(train) [10360/19176] lr: 9.1843e-06 eta: 3:34:32 time: 1.3431 data_time: 0.0135 memory: 10897 loss: 0.1871 2025/03/23 21:38:58 - mmengine - INFO - Iter(train) [10370/19176] lr: 9.1675e-06 eta: 3:34:15 time: 1.1400 data_time: 0.0124 memory: 10489 loss: 0.1723 2025/03/23 21:39:07 - mmengine - INFO - Iter(train) [10380/19176] lr: 9.1506e-06 eta: 3:33:56 time: 0.9452 data_time: 0.0121 memory: 9953 loss: 0.2166 2025/03/23 21:39:16 - mmengine - INFO - Iter(train) [10390/19176] lr: 9.1338e-06 eta: 3:33:37 time: 0.9428 data_time: 0.0109 memory: 14774 loss: 0.1803 2025/03/23 21:39:36 - mmengine - INFO - Iter(train) [10400/19176] lr: 9.1170e-06 eta: 3:33:27 time: 1.9409 data_time: 0.0140 memory: 13242 loss: 0.1789 2025/03/23 21:39:54 - mmengine - INFO - Iter(train) [10410/19176] lr: 9.1002e-06 eta: 3:33:15 time: 1.7786 data_time: 0.0143 memory: 12105 loss: 0.1476 2025/03/23 21:40:11 - mmengine - INFO - Iter(train) [10420/19176] lr: 9.0834e-06 eta: 3:33:02 time: 1.7034 data_time: 0.0141 memory: 11749 loss: 0.1464 2025/03/23 21:40:27 - mmengine - INFO - Iter(train) [10430/19176] lr: 9.0665e-06 eta: 3:32:49 time: 1.6464 data_time: 0.0142 memory: 11499 loss: 0.1539 2025/03/23 21:40:43 - mmengine - INFO - Iter(train) [10440/19176] lr: 9.0497e-06 eta: 3:32:36 time: 1.5854 data_time: 0.0136 memory: 11359 loss: 0.1770 2025/03/23 21:40:58 - mmengine - INFO - Iter(train) [10450/19176] lr: 9.0329e-06 eta: 3:32:22 time: 1.5227 data_time: 0.0141 memory: 11266 loss: 0.1775 2025/03/23 21:41:12 - mmengine - INFO - Iter(train) [10460/19176] lr: 9.0161e-06 eta: 3:32:07 time: 1.4128 data_time: 0.0141 memory: 11018 loss: 0.1788 2025/03/23 21:41:25 - mmengine - INFO - Iter(train) [10470/19176] lr: 8.9993e-06 eta: 3:31:51 time: 1.2903 data_time: 0.0135 memory: 10791 loss: 0.1626 2025/03/23 21:41:36 - mmengine - INFO - Iter(train) [10480/19176] lr: 8.9825e-06 eta: 3:31:33 time: 1.0582 data_time: 0.0121 memory: 10262 loss: 0.1832 2025/03/23 21:41:48 - mmengine - INFO - Iter(train) [10490/19176] lr: 8.9657e-06 eta: 3:31:16 time: 1.1717 data_time: 0.0113 memory: 16729 loss: 0.1786 2025/03/23 21:42:07 - mmengine - INFO - Iter(train) [10500/19176] lr: 8.9489e-06 eta: 3:31:05 time: 1.9725 data_time: 0.0144 memory: 14863 loss: 0.1636 2025/03/23 21:42:25 - mmengine - INFO - Iter(train) [10510/19176] lr: 8.9321e-06 eta: 3:30:53 time: 1.7429 data_time: 0.0140 memory: 11916 loss: 0.1546 2025/03/23 21:42:41 - mmengine - INFO - Iter(train) [10520/19176] lr: 8.9153e-06 eta: 3:30:40 time: 1.6695 data_time: 0.0142 memory: 11544 loss: 0.1468 2025/03/23 21:42:57 - mmengine - INFO - Iter(train) [10530/19176] lr: 8.8985e-06 eta: 3:30:27 time: 1.6054 data_time: 0.0142 memory: 11416 loss: 0.1496 2025/03/23 21:43:13 - mmengine - INFO - Iter(train) [10540/19176] lr: 8.8817e-06 eta: 3:30:13 time: 1.5388 data_time: 0.0143 memory: 11256 loss: 0.1667 2025/03/23 21:43:27 - mmengine - INFO - Iter(train) [10550/19176] lr: 8.8650e-06 eta: 3:29:58 time: 1.4513 data_time: 0.0136 memory: 11136 loss: 0.1771 2025/03/23 21:43:42 - mmengine - INFO - Iter(train) [10560/19176] lr: 8.8482e-06 eta: 3:29:43 time: 1.4141 data_time: 0.0143 memory: 10956 loss: 0.1560 2025/03/23 21:43:54 - mmengine - INFO - Iter(train) [10570/19176] lr: 8.8314e-06 eta: 3:29:27 time: 1.2616 data_time: 0.0138 memory: 10815 loss: 0.1936 2025/03/23 21:44:04 - mmengine - INFO - Iter(train) [10580/19176] lr: 8.8146e-06 eta: 3:29:09 time: 1.0292 data_time: 0.0123 memory: 10209 loss: 0.1783 2025/03/23 21:44:14 - mmengine - INFO - Iter(train) [10590/19176] lr: 8.7979e-06 eta: 3:28:50 time: 0.9665 data_time: 0.0107 memory: 16557 loss: 0.1608 2025/03/23 21:44:33 - mmengine - INFO - Iter(train) [10600/19176] lr: 8.7811e-06 eta: 3:28:39 time: 1.9174 data_time: 0.0144 memory: 13257 loss: 0.1545 2025/03/23 21:44:51 - mmengine - INFO - Iter(train) [10610/19176] lr: 8.7643e-06 eta: 3:28:27 time: 1.7379 data_time: 0.0144 memory: 11842 loss: 0.1575 2025/03/23 21:45:07 - mmengine - INFO - Iter(train) [10620/19176] lr: 8.7476e-06 eta: 3:28:14 time: 1.6853 data_time: 0.0143 memory: 11668 loss: 0.1682 2025/03/23 21:45:24 - mmengine - INFO - Iter(train) [10630/19176] lr: 8.7308e-06 eta: 3:28:01 time: 1.6358 data_time: 0.0142 memory: 11436 loss: 0.1980 2025/03/23 21:45:39 - mmengine - INFO - Iter(train) [10640/19176] lr: 8.7141e-06 eta: 3:27:47 time: 1.5604 data_time: 0.0144 memory: 11354 loss: 0.1631 2025/03/23 21:45:54 - mmengine - INFO - Iter(train) [10650/19176] lr: 8.6973e-06 eta: 3:27:33 time: 1.4668 data_time: 0.0141 memory: 11230 loss: 0.1757 2025/03/23 21:46:08 - mmengine - INFO - Iter(train) [10660/19176] lr: 8.6806e-06 eta: 3:27:17 time: 1.3377 data_time: 0.0138 memory: 10947 loss: 0.1506 2025/03/23 21:46:18 - mmengine - INFO - Iter(train) [10670/19176] lr: 8.6638e-06 eta: 3:26:59 time: 1.0672 data_time: 0.0120 memory: 10366 loss: 0.1617 2025/03/23 21:46:27 - mmengine - INFO - Iter(train) [10680/19176] lr: 8.6471e-06 eta: 3:26:40 time: 0.9193 data_time: 0.0121 memory: 9873 loss: 0.1743 2025/03/23 21:46:37 - mmengine - INFO - Iter(train) [10690/19176] lr: 8.6304e-06 eta: 3:26:22 time: 0.9286 data_time: 0.0108 memory: 14439 loss: 0.2566 2025/03/23 21:46:56 - mmengine - INFO - Iter(train) [10700/19176] lr: 8.6136e-06 eta: 3:26:11 time: 1.9357 data_time: 0.0140 memory: 13349 loss: 0.1600 2025/03/23 21:47:13 - mmengine - INFO - Iter(train) [10710/19176] lr: 8.5969e-06 eta: 3:25:58 time: 1.7064 data_time: 0.0142 memory: 11719 loss: 0.1631 2025/03/23 21:47:30 - mmengine - INFO - Iter(train) [10720/19176] lr: 8.5802e-06 eta: 3:25:45 time: 1.6597 data_time: 0.0142 memory: 11484 loss: 0.1864 2025/03/23 21:47:46 - mmengine - INFO - Iter(train) [10730/19176] lr: 8.5635e-06 eta: 3:25:32 time: 1.6099 data_time: 0.0144 memory: 11397 loss: 0.1661 2025/03/23 21:48:01 - mmengine - INFO - Iter(train) [10740/19176] lr: 8.5468e-06 eta: 3:25:18 time: 1.5389 data_time: 0.0145 memory: 11285 loss: 0.1657 2025/03/23 21:48:16 - mmengine - INFO - Iter(train) [10750/19176] lr: 8.5301e-06 eta: 3:25:03 time: 1.4687 data_time: 0.0140 memory: 11094 loss: 0.1660 2025/03/23 21:48:30 - mmengine - INFO - Iter(train) [10760/19176] lr: 8.5134e-06 eta: 3:24:48 time: 1.3888 data_time: 0.0137 memory: 10949 loss: 0.1706 2025/03/23 21:48:41 - mmengine - INFO - Iter(train) [10770/19176] lr: 8.4967e-06 eta: 3:24:31 time: 1.1666 data_time: 0.0125 memory: 10606 loss: 0.1660 2025/03/23 21:48:52 - mmengine - INFO - Iter(train) [10780/19176] lr: 8.4800e-06 eta: 3:24:13 time: 1.0198 data_time: 0.0122 memory: 10177 loss: 0.1971 2025/03/23 21:49:02 - mmengine - INFO - Iter(train) [10790/19176] lr: 8.4633e-06 eta: 3:23:55 time: 1.0650 data_time: 0.0114 memory: 13293 loss: 0.1751 2025/03/23 21:49:20 - mmengine - INFO - Iter(train) [10800/19176] lr: 8.4466e-06 eta: 3:23:43 time: 1.7982 data_time: 0.0147 memory: 12367 loss: 0.1720 2025/03/23 21:49:37 - mmengine - INFO - Iter(train) [10810/19176] lr: 8.4299e-06 eta: 3:23:31 time: 1.6911 data_time: 0.0146 memory: 11719 loss: 0.1681 2025/03/23 21:49:54 - mmengine - INFO - Iter(train) [10820/19176] lr: 8.4132e-06 eta: 3:23:18 time: 1.6553 data_time: 0.0146 memory: 11521 loss: 0.1694 2025/03/23 21:50:10 - mmengine - INFO - Iter(train) [10830/19176] lr: 8.3965e-06 eta: 3:23:04 time: 1.6066 data_time: 0.0146 memory: 11390 loss: 0.1526 2025/03/23 21:50:25 - mmengine - INFO - Iter(train) [10840/19176] lr: 8.3799e-06 eta: 3:22:50 time: 1.5582 data_time: 0.0146 memory: 11302 loss: 0.1721 2025/03/23 21:50:40 - mmengine - INFO - Iter(train) [10850/19176] lr: 8.3632e-06 eta: 3:22:36 time: 1.5041 data_time: 0.0142 memory: 11168 loss: 0.1783 2025/03/23 21:50:55 - mmengine - INFO - Iter(train) [10860/19176] lr: 8.3466e-06 eta: 3:22:21 time: 1.4511 data_time: 0.0145 memory: 11155 loss: 0.2023 2025/03/23 21:51:08 - mmengine - INFO - Iter(train) [10870/19176] lr: 8.3299e-06 eta: 3:22:06 time: 1.3588 data_time: 0.0143 memory: 10880 loss: 0.1877 2025/03/23 21:51:20 - mmengine - INFO - Iter(train) [10880/19176] lr: 8.3133e-06 eta: 3:21:49 time: 1.1377 data_time: 0.0129 memory: 10519 loss: 0.1726 2025/03/23 21:51:31 - mmengine - INFO - Iter(train) [10890/19176] lr: 8.2966e-06 eta: 3:21:31 time: 1.0639 data_time: 0.0115 memory: 13611 loss: 0.1540 2025/03/23 21:51:49 - mmengine - INFO - Iter(train) [10900/19176] lr: 8.2800e-06 eta: 3:21:19 time: 1.8066 data_time: 0.0145 memory: 12598 loss: 0.1715 2025/03/23 21:52:06 - mmengine - INFO - Iter(train) [10910/19176] lr: 8.2633e-06 eta: 3:21:07 time: 1.7114 data_time: 0.0149 memory: 11783 loss: 0.1399 2025/03/23 21:52:22 - mmengine - INFO - Iter(train) [10920/19176] lr: 8.2467e-06 eta: 3:20:53 time: 1.6326 data_time: 0.0152 memory: 11450 loss: 0.1661 2025/03/23 21:52:38 - mmengine - INFO - Iter(train) [10930/19176] lr: 8.2301e-06 eta: 3:20:40 time: 1.5813 data_time: 0.0149 memory: 11329 loss: 0.1682 2025/03/23 21:52:53 - mmengine - INFO - Iter(train) [10940/19176] lr: 8.2135e-06 eta: 3:20:26 time: 1.5412 data_time: 0.0150 memory: 11257 loss: 0.1875 2025/03/23 21:53:08 - mmengine - INFO - Iter(train) [10950/19176] lr: 8.1968e-06 eta: 3:20:11 time: 1.4539 data_time: 0.0144 memory: 11084 loss: 0.1573 2025/03/23 21:53:21 - mmengine - INFO - Iter(train) [10960/19176] lr: 8.1802e-06 eta: 3:19:56 time: 1.3664 data_time: 0.0143 memory: 10934 loss: 0.1825 2025/03/23 21:53:33 - mmengine - INFO - Iter(train) [10970/19176] lr: 8.1636e-06 eta: 3:19:39 time: 1.1654 data_time: 0.0129 memory: 10481 loss: 0.1490 2025/03/23 21:53:43 - mmengine - INFO - Iter(train) [10980/19176] lr: 8.1470e-06 eta: 3:19:21 time: 0.9671 data_time: 0.0120 memory: 10111 loss: 0.1823 2025/03/23 21:53:53 - mmengine - INFO - Iter(train) [10990/19176] lr: 8.1304e-06 eta: 3:19:02 time: 0.9823 data_time: 0.0109 memory: 14533 loss: 0.1868 2025/03/23 21:54:12 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 21:54:12 - mmengine - INFO - Iter(train) [11000/19176] lr: 8.1138e-06 eta: 3:18:51 time: 1.8983 data_time: 0.0145 memory: 12755 loss: 0.1706 2025/03/23 21:54:12 - mmengine - INFO - Saving checkpoint at 11000 iterations 2025/03/23 21:54:30 - mmengine - INFO - Iter(train) [11010/19176] lr: 8.0973e-06 eta: 3:18:39 time: 1.8253 data_time: 0.0911 memory: 11991 loss: 0.1475 2025/03/23 21:54:46 - mmengine - INFO - Iter(train) [11020/19176] lr: 8.0807e-06 eta: 3:18:26 time: 1.6652 data_time: 0.0147 memory: 11577 loss: 0.1520 2025/03/23 21:55:03 - mmengine - INFO - Iter(train) [11030/19176] lr: 8.0641e-06 eta: 3:18:13 time: 1.6207 data_time: 0.0143 memory: 11448 loss: 0.1361 2025/03/23 21:55:18 - mmengine - INFO - Iter(train) [11040/19176] lr: 8.0475e-06 eta: 3:17:59 time: 1.5744 data_time: 0.0144 memory: 11325 loss: 0.1654 2025/03/23 21:55:34 - mmengine - INFO - Iter(train) [11050/19176] lr: 8.0310e-06 eta: 3:17:45 time: 1.5215 data_time: 0.0141 memory: 11240 loss: 0.1617 2025/03/23 21:55:48 - mmengine - INFO - Iter(train) [11060/19176] lr: 8.0144e-06 eta: 3:17:30 time: 1.4530 data_time: 0.0139 memory: 11101 loss: 0.1663 2025/03/23 21:56:01 - mmengine - INFO - Iter(train) [11070/19176] lr: 7.9979e-06 eta: 3:17:14 time: 1.2895 data_time: 0.0136 memory: 10897 loss: 0.2068 2025/03/23 21:56:12 - mmengine - INFO - Iter(train) [11080/19176] lr: 7.9813e-06 eta: 3:16:57 time: 1.0843 data_time: 0.0122 memory: 10317 loss: 0.1579 2025/03/23 21:56:23 - mmengine - INFO - Iter(train) [11090/19176] lr: 7.9648e-06 eta: 3:16:40 time: 1.1320 data_time: 0.0119 memory: 15963 loss: 0.2233 2025/03/23 21:56:41 - mmengine - INFO - Iter(train) [11100/19176] lr: 7.9483e-06 eta: 3:16:28 time: 1.8049 data_time: 0.0141 memory: 12991 loss: 0.1679 2025/03/23 21:56:58 - mmengine - INFO - Iter(train) [11110/19176] lr: 7.9317e-06 eta: 3:16:15 time: 1.6742 data_time: 0.0145 memory: 11705 loss: 0.1508 2025/03/23 21:57:14 - mmengine - INFO - Iter(train) [11120/19176] lr: 7.9152e-06 eta: 3:16:01 time: 1.6196 data_time: 0.0146 memory: 11429 loss: 0.1714 2025/03/23 21:57:30 - mmengine - INFO - Iter(train) [11130/19176] lr: 7.8987e-06 eta: 3:15:47 time: 1.5351 data_time: 0.0143 memory: 11281 loss: 0.1689 2025/03/23 21:57:44 - mmengine - INFO - Iter(train) [11140/19176] lr: 7.8822e-06 eta: 3:15:33 time: 1.4838 data_time: 0.0141 memory: 11150 loss: 0.1829 2025/03/23 21:57:59 - mmengine - INFO - Iter(train) [11150/19176] lr: 7.8657e-06 eta: 3:15:18 time: 1.4117 data_time: 0.0141 memory: 10973 loss: 0.1807 2025/03/23 21:58:12 - mmengine - INFO - Iter(train) [11160/19176] lr: 7.8492e-06 eta: 3:15:03 time: 1.3366 data_time: 0.0142 memory: 10900 loss: 0.1550 2025/03/23 21:58:23 - mmengine - INFO - Iter(train) [11170/19176] lr: 7.8327e-06 eta: 3:14:45 time: 1.1121 data_time: 0.0126 memory: 10528 loss: 0.1976 2025/03/23 21:58:33 - mmengine - INFO - Iter(train) [11180/19176] lr: 7.8162e-06 eta: 3:14:27 time: 0.9658 data_time: 0.0124 memory: 10036 loss: 0.1746 2025/03/23 21:58:43 - mmengine - INFO - Iter(train) [11190/19176] lr: 7.7997e-06 eta: 3:14:09 time: 0.9831 data_time: 0.0111 memory: 15882 loss: 0.1664 2025/03/23 21:59:03 - mmengine - INFO - Iter(train) [11200/19176] lr: 7.7833e-06 eta: 3:13:59 time: 2.0058 data_time: 0.0145 memory: 13728 loss: 0.1694 2025/03/23 21:59:20 - mmengine - INFO - Iter(train) [11210/19176] lr: 7.7668e-06 eta: 3:13:46 time: 1.7510 data_time: 0.0142 memory: 12084 loss: 0.1640 2025/03/23 21:59:37 - mmengine - INFO - Iter(train) [11220/19176] lr: 7.7503e-06 eta: 3:13:33 time: 1.6990 data_time: 0.0145 memory: 12517 loss: 0.1585 2025/03/23 21:59:53 - mmengine - INFO - Iter(train) [11230/19176] lr: 7.7339e-06 eta: 3:13:20 time: 1.6426 data_time: 0.0143 memory: 11593 loss: 0.1498 2025/03/23 22:00:09 - mmengine - INFO - Iter(train) [11240/19176] lr: 7.7174e-06 eta: 3:13:06 time: 1.5975 data_time: 0.0145 memory: 11394 loss: 0.1515 2025/03/23 22:00:25 - mmengine - INFO - Iter(train) [11250/19176] lr: 7.7010e-06 eta: 3:12:52 time: 1.5156 data_time: 0.0139 memory: 11456 loss: 0.1447 2025/03/23 22:00:39 - mmengine - INFO - Iter(train) [11260/19176] lr: 7.6846e-06 eta: 3:12:37 time: 1.4038 data_time: 0.0140 memory: 11038 loss: 0.1520 2025/03/23 22:00:50 - mmengine - INFO - Iter(train) [11270/19176] lr: 7.6681e-06 eta: 3:12:20 time: 1.1397 data_time: 0.0126 memory: 10508 loss: 0.1798 2025/03/23 22:01:00 - mmengine - INFO - Iter(train) [11280/19176] lr: 7.6517e-06 eta: 3:12:02 time: 0.9896 data_time: 0.0123 memory: 10162 loss: 0.2064 2025/03/23 22:01:10 - mmengine - INFO - Iter(train) [11290/19176] lr: 7.6353e-06 eta: 3:11:45 time: 1.0084 data_time: 0.0112 memory: 14062 loss: 0.1826 2025/03/23 22:01:29 - mmengine - INFO - Iter(train) [11300/19176] lr: 7.6189e-06 eta: 3:11:33 time: 1.8746 data_time: 0.0143 memory: 12969 loss: 0.1762 2025/03/23 22:01:46 - mmengine - INFO - Iter(train) [11310/19176] lr: 7.6025e-06 eta: 3:11:20 time: 1.7395 data_time: 0.0147 memory: 12033 loss: 0.1499 2025/03/23 22:02:03 - mmengine - INFO - Iter(train) [11320/19176] lr: 7.5861e-06 eta: 3:11:07 time: 1.6556 data_time: 0.0145 memory: 11486 loss: 0.1738 2025/03/23 22:02:19 - mmengine - INFO - Iter(train) [11330/19176] lr: 7.5697e-06 eta: 3:10:53 time: 1.5920 data_time: 0.0143 memory: 11655 loss: 0.1529 2025/03/23 22:02:34 - mmengine - INFO - Iter(train) [11340/19176] lr: 7.5533e-06 eta: 3:10:39 time: 1.5290 data_time: 0.0168 memory: 11207 loss: 0.1444 2025/03/23 22:02:49 - mmengine - INFO - Iter(train) [11350/19176] lr: 7.5370e-06 eta: 3:10:25 time: 1.4757 data_time: 0.0140 memory: 11107 loss: 0.1511 2025/03/23 22:03:03 - mmengine - INFO - Iter(train) [11360/19176] lr: 7.5206e-06 eta: 3:10:10 time: 1.4380 data_time: 0.0140 memory: 11007 loss: 0.1660 2025/03/23 22:03:16 - mmengine - INFO - Iter(train) [11370/19176] lr: 7.5042e-06 eta: 3:09:55 time: 1.3341 data_time: 0.0136 memory: 10870 loss: 0.1915 2025/03/23 22:03:28 - mmengine - INFO - Iter(train) [11380/19176] lr: 7.4879e-06 eta: 3:09:38 time: 1.1116 data_time: 0.0126 memory: 10475 loss: 0.1550 2025/03/23 22:03:38 - mmengine - INFO - Iter(train) [11390/19176] lr: 7.4715e-06 eta: 3:09:20 time: 1.0359 data_time: 0.0112 memory: 14641 loss: 0.1689 2025/03/23 22:03:56 - mmengine - INFO - Iter(train) [11400/19176] lr: 7.4552e-06 eta: 3:09:08 time: 1.7857 data_time: 0.0145 memory: 12272 loss: 0.1701 2025/03/23 22:04:13 - mmengine - INFO - Iter(train) [11410/19176] lr: 7.4389e-06 eta: 3:08:55 time: 1.7032 data_time: 0.0141 memory: 12105 loss: 0.1618 2025/03/23 22:04:29 - mmengine - INFO - Iter(train) [11420/19176] lr: 7.4226e-06 eta: 3:08:41 time: 1.6527 data_time: 0.0143 memory: 11495 loss: 0.1523 2025/03/23 22:04:45 - mmengine - INFO - Iter(train) [11430/19176] lr: 7.4062e-06 eta: 3:08:28 time: 1.5851 data_time: 0.0144 memory: 11375 loss: 0.1686 2025/03/23 22:05:00 - mmengine - INFO - Iter(train) [11440/19176] lr: 7.3899e-06 eta: 3:08:13 time: 1.4898 data_time: 0.0144 memory: 11186 loss: 0.1880 2025/03/23 22:05:14 - mmengine - INFO - Iter(train) [11450/19176] lr: 7.3736e-06 eta: 3:07:58 time: 1.4150 data_time: 0.0140 memory: 11063 loss: 0.1929 2025/03/23 22:05:26 - mmengine - INFO - Iter(train) [11460/19176] lr: 7.3573e-06 eta: 3:07:42 time: 1.2000 data_time: 0.0134 memory: 10670 loss: 0.1458 2025/03/23 22:05:37 - mmengine - INFO - Iter(train) [11470/19176] lr: 7.3411e-06 eta: 3:07:25 time: 1.0589 data_time: 0.0121 memory: 10202 loss: 0.1451 2025/03/23 22:05:46 - mmengine - INFO - Iter(train) [11480/19176] lr: 7.3248e-06 eta: 3:07:07 time: 0.9146 data_time: 0.0118 memory: 10005 loss: 0.2152 2025/03/23 22:05:54 - mmengine - INFO - Iter(train) [11490/19176] lr: 7.3085e-06 eta: 3:06:48 time: 0.8544 data_time: 0.0103 memory: 13835 loss: 0.1497 2025/03/23 22:06:13 - mmengine - INFO - Iter(train) [11500/19176] lr: 7.2922e-06 eta: 3:06:36 time: 1.8768 data_time: 0.0143 memory: 13361 loss: 0.1492 2025/03/23 22:06:30 - mmengine - INFO - Iter(train) [11510/19176] lr: 7.2760e-06 eta: 3:06:23 time: 1.7116 data_time: 0.0142 memory: 11878 loss: 0.1458 2025/03/23 22:06:47 - mmengine - INFO - Iter(train) [11520/19176] lr: 7.2597e-06 eta: 3:06:10 time: 1.6611 data_time: 0.0143 memory: 11631 loss: 0.1585 2025/03/23 22:07:03 - mmengine - INFO - Iter(train) [11530/19176] lr: 7.2435e-06 eta: 3:05:56 time: 1.6091 data_time: 0.0144 memory: 11425 loss: 0.1449 2025/03/23 22:07:19 - mmengine - INFO - Iter(train) [11540/19176] lr: 7.2273e-06 eta: 3:05:42 time: 1.5557 data_time: 0.0141 memory: 11294 loss: 0.1714 2025/03/23 22:07:34 - mmengine - INFO - Iter(train) [11550/19176] lr: 7.2110e-06 eta: 3:05:28 time: 1.5107 data_time: 0.0142 memory: 11197 loss: 0.1569 2025/03/23 22:07:48 - mmengine - INFO - Iter(train) [11560/19176] lr: 7.1948e-06 eta: 3:05:13 time: 1.4208 data_time: 0.0143 memory: 11021 loss: 0.1797 2025/03/23 22:08:01 - mmengine - INFO - Iter(train) [11570/19176] lr: 7.1786e-06 eta: 3:04:57 time: 1.2615 data_time: 0.0142 memory: 10676 loss: 0.1744 2025/03/23 22:08:11 - mmengine - INFO - Iter(train) [11580/19176] lr: 7.1624e-06 eta: 3:04:40 time: 1.0244 data_time: 0.0120 memory: 10228 loss: 0.1618 2025/03/23 22:08:22 - mmengine - INFO - Iter(train) [11590/19176] lr: 7.1462e-06 eta: 3:04:23 time: 1.0797 data_time: 0.0117 memory: 14721 loss: 0.1718 2025/03/23 22:08:40 - mmengine - INFO - Iter(train) [11600/19176] lr: 7.1301e-06 eta: 3:04:11 time: 1.8463 data_time: 0.0144 memory: 12478 loss: 0.1871 2025/03/23 22:08:57 - mmengine - INFO - Iter(train) [11610/19176] lr: 7.1139e-06 eta: 3:03:58 time: 1.6936 data_time: 0.0144 memory: 11792 loss: 0.1468 2025/03/23 22:09:13 - mmengine - INFO - Iter(train) [11620/19176] lr: 7.0977e-06 eta: 3:03:44 time: 1.6069 data_time: 0.0142 memory: 11408 loss: 0.1538 2025/03/23 22:09:29 - mmengine - INFO - Iter(train) [11630/19176] lr: 7.0816e-06 eta: 3:03:30 time: 1.5827 data_time: 0.0140 memory: 11348 loss: 0.1539 2025/03/23 22:09:44 - mmengine - INFO - Iter(train) [11640/19176] lr: 7.0654e-06 eta: 3:03:16 time: 1.5477 data_time: 0.0144 memory: 11239 loss: 0.1622 2025/03/23 22:09:59 - mmengine - INFO - Iter(train) [11650/19176] lr: 7.0493e-06 eta: 3:03:02 time: 1.4922 data_time: 0.0141 memory: 11141 loss: 0.1740 2025/03/23 22:10:14 - mmengine - INFO - Iter(train) [11660/19176] lr: 7.0331e-06 eta: 3:02:47 time: 1.4316 data_time: 0.0141 memory: 11018 loss: 0.1709 2025/03/23 22:10:26 - mmengine - INFO - Iter(train) [11670/19176] lr: 7.0170e-06 eta: 3:02:31 time: 1.2144 data_time: 0.0132 memory: 10805 loss: 0.1804 2025/03/23 22:10:36 - mmengine - INFO - Iter(train) [11680/19176] lr: 7.0009e-06 eta: 3:02:14 time: 1.0509 data_time: 0.0119 memory: 10247 loss: 0.1463 2025/03/23 22:10:47 - mmengine - INFO - Iter(train) [11690/19176] lr: 6.9848e-06 eta: 3:01:57 time: 1.0500 data_time: 0.0117 memory: 16574 loss: 0.1635 2025/03/23 22:11:06 - mmengine - INFO - Iter(train) [11700/19176] lr: 6.9687e-06 eta: 3:01:45 time: 1.9166 data_time: 0.0145 memory: 12929 loss: 0.1599 2025/03/23 22:11:23 - mmengine - INFO - Iter(train) [11710/19176] lr: 6.9526e-06 eta: 3:01:32 time: 1.7358 data_time: 0.0141 memory: 11847 loss: 0.1692 2025/03/23 22:11:40 - mmengine - INFO - Iter(train) [11720/19176] lr: 6.9365e-06 eta: 3:01:19 time: 1.6681 data_time: 0.0141 memory: 11626 loss: 0.1483 2025/03/23 22:11:56 - mmengine - INFO - Iter(train) [11730/19176] lr: 6.9204e-06 eta: 3:01:05 time: 1.6030 data_time: 0.0139 memory: 11394 loss: 0.1548 2025/03/23 22:12:11 - mmengine - INFO - Iter(train) [11740/19176] lr: 6.9044e-06 eta: 3:00:51 time: 1.5292 data_time: 0.0139 memory: 11266 loss: 0.1776 2025/03/23 22:12:26 - mmengine - INFO - Iter(train) [11750/19176] lr: 6.8883e-06 eta: 3:00:37 time: 1.4779 data_time: 0.0138 memory: 11157 loss: 0.1622 2025/03/23 22:12:39 - mmengine - INFO - Iter(train) [11760/19176] lr: 6.8723e-06 eta: 3:00:21 time: 1.3309 data_time: 0.0133 memory: 10869 loss: 0.2287 2025/03/23 22:12:51 - mmengine - INFO - Iter(train) [11770/19176] lr: 6.8562e-06 eta: 3:00:05 time: 1.1715 data_time: 0.0127 memory: 10528 loss: 0.1470 2025/03/23 22:13:01 - mmengine - INFO - Iter(train) [11780/19176] lr: 6.8402e-06 eta: 2:59:47 time: 1.0018 data_time: 0.0124 memory: 10212 loss: 0.1641 2025/03/23 22:13:12 - mmengine - INFO - Iter(train) [11790/19176] lr: 6.8242e-06 eta: 2:59:31 time: 1.0990 data_time: 0.0119 memory: 16168 loss: 0.1807 2025/03/23 22:13:31 - mmengine - INFO - Iter(train) [11800/19176] lr: 6.8082e-06 eta: 2:59:19 time: 1.8720 data_time: 0.0144 memory: 12970 loss: 0.1445 2025/03/23 22:13:48 - mmengine - INFO - Iter(train) [11810/19176] lr: 6.7922e-06 eta: 2:59:05 time: 1.7083 data_time: 0.0144 memory: 11849 loss: 0.1672 2025/03/23 22:14:04 - mmengine - INFO - Iter(train) [11820/19176] lr: 6.7762e-06 eta: 2:58:52 time: 1.6428 data_time: 0.0143 memory: 11526 loss: 0.1697 2025/03/23 22:14:20 - mmengine - INFO - Iter(train) [11830/19176] lr: 6.7602e-06 eta: 2:58:38 time: 1.5842 data_time: 0.0142 memory: 11380 loss: 0.1587 2025/03/23 22:14:35 - mmengine - INFO - Iter(train) [11840/19176] lr: 6.7442e-06 eta: 2:58:24 time: 1.5188 data_time: 0.0142 memory: 11219 loss: 0.1478 2025/03/23 22:14:50 - mmengine - INFO - Iter(train) [11850/19176] lr: 6.7283e-06 eta: 2:58:09 time: 1.4117 data_time: 0.0137 memory: 10985 loss: 0.1541 2025/03/23 22:15:03 - mmengine - INFO - Iter(train) [11860/19176] lr: 6.7123e-06 eta: 2:57:54 time: 1.3251 data_time: 0.0138 memory: 10834 loss: 0.1968 2025/03/23 22:15:14 - mmengine - INFO - Iter(train) [11870/19176] lr: 6.6964e-06 eta: 2:57:37 time: 1.1649 data_time: 0.0125 memory: 10574 loss: 0.1826 2025/03/23 22:15:24 - mmengine - INFO - Iter(train) [11880/19176] lr: 6.6804e-06 eta: 2:57:20 time: 0.9850 data_time: 0.0120 memory: 10149 loss: 0.1799 2025/03/23 22:15:34 - mmengine - INFO - Iter(train) [11890/19176] lr: 6.6645e-06 eta: 2:57:02 time: 1.0039 data_time: 0.0117 memory: 13791 loss: 0.1829 2025/03/23 22:15:53 - mmengine - INFO - Iter(train) [11900/19176] lr: 6.6486e-06 eta: 2:56:50 time: 1.8712 data_time: 0.0143 memory: 12778 loss: 0.1700 2025/03/23 22:16:10 - mmengine - INFO - Iter(train) [11910/19176] lr: 6.6327e-06 eta: 2:56:37 time: 1.7253 data_time: 0.0144 memory: 11854 loss: 0.1466 2025/03/23 22:16:27 - mmengine - INFO - Iter(train) [11920/19176] lr: 6.6168e-06 eta: 2:56:24 time: 1.6930 data_time: 0.0141 memory: 11744 loss: 0.1576 2025/03/23 22:16:43 - mmengine - INFO - Iter(train) [11930/19176] lr: 6.6009e-06 eta: 2:56:11 time: 1.6163 data_time: 0.0141 memory: 11463 loss: 0.1776 2025/03/23 22:16:59 - mmengine - INFO - Iter(train) [11940/19176] lr: 6.5850e-06 eta: 2:55:57 time: 1.5686 data_time: 0.0142 memory: 11317 loss: 0.1575 2025/03/23 22:17:14 - mmengine - INFO - Iter(train) [11950/19176] lr: 6.5691e-06 eta: 2:55:43 time: 1.5240 data_time: 0.0142 memory: 11191 loss: 0.1586 2025/03/23 22:17:28 - mmengine - INFO - Iter(train) [11960/19176] lr: 6.5533e-06 eta: 2:55:28 time: 1.3990 data_time: 0.0142 memory: 11045 loss: 0.1698 2025/03/23 22:17:40 - mmengine - INFO - Iter(train) [11970/19176] lr: 6.5374e-06 eta: 2:55:11 time: 1.1662 data_time: 0.0130 memory: 10577 loss: 0.1727 2025/03/23 22:17:50 - mmengine - INFO - Iter(train) [11980/19176] lr: 6.5216e-06 eta: 2:54:54 time: 1.0398 data_time: 0.0125 memory: 10229 loss: 0.1859 2025/03/23 22:18:00 - mmengine - INFO - Iter(train) [11990/19176] lr: 6.5058e-06 eta: 2:54:37 time: 0.9858 data_time: 0.0114 memory: 12923 loss: 0.1925 2025/03/23 22:18:18 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 22:18:18 - mmengine - INFO - Iter(train) [12000/19176] lr: 6.4899e-06 eta: 2:54:24 time: 1.8232 data_time: 0.0144 memory: 12410 loss: 0.2301 2025/03/23 22:18:18 - mmengine - INFO - Saving checkpoint at 12000 iterations 2025/03/23 22:18:36 - mmengine - INFO - Iter(train) [12010/19176] lr: 6.4741e-06 eta: 2:54:12 time: 1.8066 data_time: 0.0909 memory: 11860 loss: 0.1815 2025/03/23 22:18:53 - mmengine - INFO - Iter(train) [12020/19176] lr: 6.4583e-06 eta: 2:53:58 time: 1.6603 data_time: 0.0142 memory: 11581 loss: 0.1738 2025/03/23 22:19:09 - mmengine - INFO - Iter(train) [12030/19176] lr: 6.4425e-06 eta: 2:53:45 time: 1.5967 data_time: 0.0142 memory: 11416 loss: 0.1771 2025/03/23 22:19:25 - mmengine - INFO - Iter(train) [12040/19176] lr: 6.4268e-06 eta: 2:53:31 time: 1.5804 data_time: 0.0143 memory: 11361 loss: 0.1627 2025/03/23 22:19:40 - mmengine - INFO - Iter(train) [12050/19176] lr: 6.4110e-06 eta: 2:53:17 time: 1.5145 data_time: 0.0141 memory: 11238 loss: 0.1555 2025/03/23 22:19:55 - mmengine - INFO - Iter(train) [12060/19176] lr: 6.3952e-06 eta: 2:53:02 time: 1.4884 data_time: 0.0141 memory: 11136 loss: 0.1463 2025/03/23 22:20:08 - mmengine - INFO - Iter(train) [12070/19176] lr: 6.3795e-06 eta: 2:52:47 time: 1.3599 data_time: 0.0139 memory: 10916 loss: 0.1953 2025/03/23 22:20:20 - mmengine - INFO - Iter(train) [12080/19176] lr: 6.3637e-06 eta: 2:52:30 time: 1.1369 data_time: 0.0128 memory: 10431 loss: 0.1511 2025/03/23 22:20:31 - mmengine - INFO - Iter(train) [12090/19176] lr: 6.3480e-06 eta: 2:52:14 time: 1.1487 data_time: 0.0123 memory: 13346 loss: 0.1729 2025/03/23 22:20:51 - mmengine - INFO - Iter(train) [12100/19176] lr: 6.3323e-06 eta: 2:52:03 time: 1.9819 data_time: 0.0145 memory: 13253 loss: 0.1543 2025/03/23 22:21:09 - mmengine - INFO - Iter(train) [12110/19176] lr: 6.3166e-06 eta: 2:51:50 time: 1.7881 data_time: 0.0142 memory: 12148 loss: 0.1591 2025/03/23 22:21:26 - mmengine - INFO - Iter(train) [12120/19176] lr: 6.3009e-06 eta: 2:51:37 time: 1.7085 data_time: 0.0148 memory: 11880 loss: 0.1469 2025/03/23 22:21:43 - mmengine - INFO - Iter(train) [12130/19176] lr: 6.2852e-06 eta: 2:51:23 time: 1.6391 data_time: 0.0147 memory: 11475 loss: 0.1648 2025/03/23 22:21:58 - mmengine - INFO - Iter(train) [12140/19176] lr: 6.2695e-06 eta: 2:51:09 time: 1.5705 data_time: 0.0147 memory: 11311 loss: 0.1676 2025/03/23 22:22:14 - mmengine - INFO - Iter(train) [12150/19176] lr: 6.2539e-06 eta: 2:50:55 time: 1.5309 data_time: 0.0145 memory: 11375 loss: 0.1861 2025/03/23 22:22:28 - mmengine - INFO - Iter(train) [12160/19176] lr: 6.2382e-06 eta: 2:50:41 time: 1.4779 data_time: 0.0138 memory: 11108 loss: 0.1621 2025/03/23 22:22:42 - mmengine - INFO - Iter(train) [12170/19176] lr: 6.2226e-06 eta: 2:50:26 time: 1.4031 data_time: 0.0140 memory: 11006 loss: 0.1710 2025/03/23 22:22:54 - mmengine - INFO - Iter(train) [12180/19176] lr: 6.2069e-06 eta: 2:50:09 time: 1.1635 data_time: 0.0128 memory: 10585 loss: 0.1659 2025/03/23 22:23:03 - mmengine - INFO - Iter(train) [12190/19176] lr: 6.1913e-06 eta: 2:49:52 time: 0.9282 data_time: 0.0113 memory: 13193 loss: 0.1848 2025/03/23 22:23:22 - mmengine - INFO - Iter(train) [12200/19176] lr: 6.1757e-06 eta: 2:49:39 time: 1.8753 data_time: 0.0142 memory: 12721 loss: 0.1509 2025/03/23 22:23:39 - mmengine - INFO - Iter(train) [12210/19176] lr: 6.1601e-06 eta: 2:49:26 time: 1.7257 data_time: 0.0143 memory: 11907 loss: 0.1589 2025/03/23 22:23:56 - mmengine - INFO - Iter(train) [12220/19176] lr: 6.1445e-06 eta: 2:49:13 time: 1.6732 data_time: 0.0145 memory: 11651 loss: 0.1531 2025/03/23 22:24:12 - mmengine - INFO - Iter(train) [12230/19176] lr: 6.1289e-06 eta: 2:48:59 time: 1.6172 data_time: 0.0149 memory: 11394 loss: 0.1592 2025/03/23 22:24:28 - mmengine - INFO - Iter(train) [12240/19176] lr: 6.1134e-06 eta: 2:48:45 time: 1.5623 data_time: 0.0145 memory: 11292 loss: 0.1574 2025/03/23 22:24:43 - mmengine - INFO - Iter(train) [12250/19176] lr: 6.0978e-06 eta: 2:48:31 time: 1.4792 data_time: 0.0141 memory: 11121 loss: 0.1669 2025/03/23 22:24:56 - mmengine - INFO - Iter(train) [12260/19176] lr: 6.0823e-06 eta: 2:48:16 time: 1.3716 data_time: 0.0143 memory: 10913 loss: 0.1493 2025/03/23 22:25:08 - mmengine - INFO - Iter(train) [12270/19176] lr: 6.0667e-06 eta: 2:47:59 time: 1.1225 data_time: 0.0122 memory: 10367 loss: 0.1452 2025/03/23 22:25:17 - mmengine - INFO - Iter(train) [12280/19176] lr: 6.0512e-06 eta: 2:47:42 time: 0.9950 data_time: 0.0120 memory: 10049 loss: 0.1901 2025/03/23 22:25:27 - mmengine - INFO - Iter(train) [12290/19176] lr: 6.0357e-06 eta: 2:47:25 time: 0.9925 data_time: 0.0116 memory: 12343 loss: 0.2382 2025/03/23 22:25:45 - mmengine - INFO - Iter(train) [12300/19176] lr: 6.0202e-06 eta: 2:47:12 time: 1.7634 data_time: 0.0142 memory: 12238 loss: 0.1578 2025/03/23 22:26:02 - mmengine - INFO - Iter(train) [12310/19176] lr: 6.0047e-06 eta: 2:46:59 time: 1.6846 data_time: 0.0143 memory: 11640 loss: 0.1413 2025/03/23 22:26:18 - mmengine - INFO - Iter(train) [12320/19176] lr: 5.9892e-06 eta: 2:46:45 time: 1.6176 data_time: 0.0145 memory: 11466 loss: 0.1522 2025/03/23 22:26:34 - mmengine - INFO - Iter(train) [12330/19176] lr: 5.9738e-06 eta: 2:46:31 time: 1.5620 data_time: 0.0147 memory: 11343 loss: 0.1575 2025/03/23 22:26:49 - mmengine - INFO - Iter(train) [12340/19176] lr: 5.9583e-06 eta: 2:46:16 time: 1.4849 data_time: 0.0143 memory: 11192 loss: 0.1617 2025/03/23 22:27:03 - mmengine - INFO - Iter(train) [12350/19176] lr: 5.9429e-06 eta: 2:46:02 time: 1.4146 data_time: 0.0137 memory: 10979 loss: 0.1456 2025/03/23 22:27:16 - mmengine - INFO - Iter(train) [12360/19176] lr: 5.9274e-06 eta: 2:45:46 time: 1.3704 data_time: 0.0138 memory: 10903 loss: 0.1833 2025/03/23 22:27:29 - mmengine - INFO - Iter(train) [12370/19176] lr: 5.9120e-06 eta: 2:45:30 time: 1.2123 data_time: 0.0130 memory: 10632 loss: 0.1541 2025/03/23 22:27:39 - mmengine - INFO - Iter(train) [12380/19176] lr: 5.8966e-06 eta: 2:45:14 time: 1.0491 data_time: 0.0125 memory: 10298 loss: 0.1644 2025/03/23 22:27:49 - mmengine - INFO - Iter(train) [12390/19176] lr: 5.8812e-06 eta: 2:44:56 time: 0.9774 data_time: 0.0113 memory: 13454 loss: 0.1853 2025/03/23 22:28:07 - mmengine - INFO - Iter(train) [12400/19176] lr: 5.8658e-06 eta: 2:44:44 time: 1.8370 data_time: 0.0144 memory: 12936 loss: 0.1537 2025/03/23 22:28:24 - mmengine - INFO - Iter(train) [12410/19176] lr: 5.8505e-06 eta: 2:44:31 time: 1.7217 data_time: 0.0143 memory: 11894 loss: 0.1397 2025/03/23 22:28:41 - mmengine - INFO - Iter(train) [12420/19176] lr: 5.8351e-06 eta: 2:44:17 time: 1.6607 data_time: 0.0143 memory: 11490 loss: 0.1591 2025/03/23 22:28:57 - mmengine - INFO - Iter(train) [12430/19176] lr: 5.8197e-06 eta: 2:44:04 time: 1.6263 data_time: 0.0146 memory: 11415 loss: 0.1383 2025/03/23 22:29:13 - mmengine - INFO - Iter(train) [12440/19176] lr: 5.8044e-06 eta: 2:43:50 time: 1.5677 data_time: 0.0143 memory: 11337 loss: 0.1703 2025/03/23 22:29:28 - mmengine - INFO - Iter(train) [12450/19176] lr: 5.7891e-06 eta: 2:43:35 time: 1.5209 data_time: 0.0144 memory: 11180 loss: 0.1542 2025/03/23 22:29:43 - mmengine - INFO - Iter(train) [12460/19176] lr: 5.7738e-06 eta: 2:43:21 time: 1.4807 data_time: 0.0140 memory: 11128 loss: 0.1782 2025/03/23 22:29:56 - mmengine - INFO - Iter(train) [12470/19176] lr: 5.7585e-06 eta: 2:43:06 time: 1.3494 data_time: 0.0142 memory: 10978 loss: 0.1891 2025/03/23 22:30:07 - mmengine - INFO - Iter(train) [12480/19176] lr: 5.7432e-06 eta: 2:42:49 time: 1.0734 data_time: 0.0124 memory: 10269 loss: 0.1611 2025/03/23 22:30:18 - mmengine - INFO - Iter(train) [12490/19176] lr: 5.7279e-06 eta: 2:42:33 time: 1.1234 data_time: 0.0120 memory: 13278 loss: 0.1700 2025/03/23 22:30:37 - mmengine - INFO - Iter(train) [12500/19176] lr: 5.7126e-06 eta: 2:42:20 time: 1.8209 data_time: 0.0143 memory: 12453 loss: 0.1511 2025/03/23 22:30:54 - mmengine - INFO - Iter(train) [12510/19176] lr: 5.6974e-06 eta: 2:42:07 time: 1.7279 data_time: 0.0143 memory: 11863 loss: 0.1707 2025/03/23 22:31:10 - mmengine - INFO - Iter(train) [12520/19176] lr: 5.6822e-06 eta: 2:41:53 time: 1.6595 data_time: 0.0144 memory: 11704 loss: 0.1513 2025/03/23 22:31:27 - mmengine - INFO - Iter(train) [12530/19176] lr: 5.6669e-06 eta: 2:41:39 time: 1.6077 data_time: 0.0142 memory: 11384 loss: 0.1492 2025/03/23 22:31:42 - mmengine - INFO - Iter(train) [12540/19176] lr: 5.6517e-06 eta: 2:41:25 time: 1.5410 data_time: 0.0144 memory: 11290 loss: 0.1654 2025/03/23 22:31:57 - mmengine - INFO - Iter(train) [12550/19176] lr: 5.6365e-06 eta: 2:41:11 time: 1.4745 data_time: 0.0141 memory: 11116 loss: 0.1598 2025/03/23 22:32:11 - mmengine - INFO - Iter(train) [12560/19176] lr: 5.6213e-06 eta: 2:40:56 time: 1.3958 data_time: 0.0138 memory: 10927 loss: 0.1744 2025/03/23 22:32:23 - mmengine - INFO - Iter(train) [12570/19176] lr: 5.6061e-06 eta: 2:40:40 time: 1.2497 data_time: 0.0133 memory: 10703 loss: 0.1868 2025/03/23 22:32:34 - mmengine - INFO - Iter(train) [12580/19176] lr: 5.5910e-06 eta: 2:40:24 time: 1.0875 data_time: 0.0123 memory: 10320 loss: 0.1755 2025/03/23 22:32:44 - mmengine - INFO - Iter(train) [12590/19176] lr: 5.5758e-06 eta: 2:40:06 time: 0.9752 data_time: 0.0113 memory: 13102 loss: 0.1743 2025/03/23 22:33:02 - mmengine - INFO - Iter(train) [12600/19176] lr: 5.5607e-06 eta: 2:39:54 time: 1.8149 data_time: 0.0143 memory: 12357 loss: 0.1483 2025/03/23 22:33:19 - mmengine - INFO - Iter(train) [12610/19176] lr: 5.5456e-06 eta: 2:39:41 time: 1.7285 data_time: 0.0140 memory: 11875 loss: 0.1556 2025/03/23 22:33:36 - mmengine - INFO - Iter(train) [12620/19176] lr: 5.5304e-06 eta: 2:39:27 time: 1.6669 data_time: 0.0144 memory: 11528 loss: 0.1710 2025/03/23 22:33:52 - mmengine - INFO - Iter(train) [12630/19176] lr: 5.5153e-06 eta: 2:39:13 time: 1.5850 data_time: 0.0143 memory: 11412 loss: 0.1607 2025/03/23 22:34:07 - mmengine - INFO - Iter(train) [12640/19176] lr: 5.5002e-06 eta: 2:38:59 time: 1.4915 data_time: 0.0143 memory: 11146 loss: 0.1705 2025/03/23 22:34:21 - mmengine - INFO - Iter(train) [12650/19176] lr: 5.4852e-06 eta: 2:38:44 time: 1.4168 data_time: 0.0142 memory: 11007 loss: 0.2048 2025/03/23 22:34:34 - mmengine - INFO - Iter(train) [12660/19176] lr: 5.4701e-06 eta: 2:38:29 time: 1.3127 data_time: 0.0135 memory: 10725 loss: 0.1742 2025/03/23 22:34:46 - mmengine - INFO - Iter(train) [12670/19176] lr: 5.4551e-06 eta: 2:38:13 time: 1.1886 data_time: 0.0128 memory: 10582 loss: 0.1505 2025/03/23 22:34:57 - mmengine - INFO - Iter(train) [12680/19176] lr: 5.4400e-06 eta: 2:37:56 time: 1.0706 data_time: 0.0127 memory: 10277 loss: 0.1559 2025/03/23 22:35:08 - mmengine - INFO - Iter(train) [12690/19176] lr: 5.4250e-06 eta: 2:37:40 time: 1.1001 data_time: 0.0117 memory: 13081 loss: 0.1409 2025/03/23 22:35:26 - mmengine - INFO - Iter(train) [12700/19176] lr: 5.4100e-06 eta: 2:37:27 time: 1.8495 data_time: 0.0143 memory: 12598 loss: 0.1760 2025/03/23 22:35:44 - mmengine - INFO - Iter(train) [12710/19176] lr: 5.3950e-06 eta: 2:37:14 time: 1.7502 data_time: 0.0143 memory: 11956 loss: 0.1391 2025/03/23 22:36:00 - mmengine - INFO - Iter(train) [12720/19176] lr: 5.3800e-06 eta: 2:37:00 time: 1.6777 data_time: 0.0143 memory: 11640 loss: 0.1623 2025/03/23 22:36:16 - mmengine - INFO - Iter(train) [12730/19176] lr: 5.3650e-06 eta: 2:36:47 time: 1.6160 data_time: 0.0143 memory: 11446 loss: 0.1502 2025/03/23 22:36:32 - mmengine - INFO - Iter(train) [12740/19176] lr: 5.3501e-06 eta: 2:36:33 time: 1.5694 data_time: 0.0141 memory: 11297 loss: 0.1611 2025/03/23 22:36:47 - mmengine - INFO - Iter(train) [12750/19176] lr: 5.3351e-06 eta: 2:36:18 time: 1.4973 data_time: 0.0139 memory: 11198 loss: 0.1699 2025/03/23 22:37:02 - mmengine - INFO - Iter(train) [12760/19176] lr: 5.3202e-06 eta: 2:36:03 time: 1.4480 data_time: 0.0144 memory: 11038 loss: 0.1378 2025/03/23 22:37:14 - mmengine - INFO - Iter(train) [12770/19176] lr: 5.3053e-06 eta: 2:35:48 time: 1.2453 data_time: 0.0133 memory: 10818 loss: 0.1898 2025/03/23 22:37:24 - mmengine - INFO - Iter(train) [12780/19176] lr: 5.2904e-06 eta: 2:35:31 time: 1.0079 data_time: 0.0125 memory: 10161 loss: 0.2033 2025/03/23 22:37:35 - mmengine - INFO - Iter(train) [12790/19176] lr: 5.2755e-06 eta: 2:35:14 time: 1.0438 data_time: 0.0115 memory: 14629 loss: 0.1823 2025/03/23 22:37:53 - mmengine - INFO - Iter(train) [12800/19176] lr: 5.2606e-06 eta: 2:35:02 time: 1.8390 data_time: 0.0150 memory: 12355 loss: 0.1433 2025/03/23 22:38:10 - mmengine - INFO - Iter(train) [12810/19176] lr: 5.2457e-06 eta: 2:34:48 time: 1.7220 data_time: 0.0146 memory: 11833 loss: 0.1559 2025/03/23 22:38:27 - mmengine - INFO - Iter(train) [12820/19176] lr: 5.2309e-06 eta: 2:34:35 time: 1.6614 data_time: 0.0147 memory: 11606 loss: 0.1599 2025/03/23 22:38:43 - mmengine - INFO - Iter(train) [12830/19176] lr: 5.2160e-06 eta: 2:34:21 time: 1.5928 data_time: 0.0146 memory: 11376 loss: 0.1709 2025/03/23 22:38:58 - mmengine - INFO - Iter(train) [12840/19176] lr: 5.2012e-06 eta: 2:34:07 time: 1.5347 data_time: 0.0142 memory: 11237 loss: 0.1819 2025/03/23 22:39:13 - mmengine - INFO - Iter(train) [12850/19176] lr: 5.1864e-06 eta: 2:33:52 time: 1.4895 data_time: 0.0141 memory: 11226 loss: 0.1817 2025/03/23 22:39:27 - mmengine - INFO - Iter(train) [12860/19176] lr: 5.1716e-06 eta: 2:33:37 time: 1.4321 data_time: 0.0141 memory: 10991 loss: 0.2030 2025/03/23 22:39:40 - mmengine - INFO - Iter(train) [12870/19176] lr: 5.1568e-06 eta: 2:33:22 time: 1.2294 data_time: 0.0131 memory: 10791 loss: 0.1948 2025/03/23 22:39:50 - mmengine - INFO - Iter(train) [12880/19176] lr: 5.1421e-06 eta: 2:33:05 time: 1.0392 data_time: 0.0126 memory: 10258 loss: 0.1773 2025/03/23 22:40:00 - mmengine - INFO - Iter(train) [12890/19176] lr: 5.1273e-06 eta: 2:32:48 time: 1.0438 data_time: 0.0115 memory: 15007 loss: 0.1770 2025/03/23 22:40:18 - mmengine - INFO - Iter(train) [12900/19176] lr: 5.1126e-06 eta: 2:32:35 time: 1.7963 data_time: 0.0147 memory: 12309 loss: 0.1477 2025/03/23 22:40:35 - mmengine - INFO - Iter(train) [12910/19176] lr: 5.0978e-06 eta: 2:32:22 time: 1.6986 data_time: 0.0144 memory: 12029 loss: 0.1495 2025/03/23 22:40:52 - mmengine - INFO - Iter(train) [12920/19176] lr: 5.0831e-06 eta: 2:32:08 time: 1.6346 data_time: 0.0144 memory: 11475 loss: 0.1408 2025/03/23 22:41:07 - mmengine - INFO - Iter(train) [12930/19176] lr: 5.0684e-06 eta: 2:31:54 time: 1.5713 data_time: 0.0141 memory: 11321 loss: 0.1574 2025/03/23 22:41:23 - mmengine - INFO - Iter(train) [12940/19176] lr: 5.0537e-06 eta: 2:31:40 time: 1.5404 data_time: 0.0143 memory: 11229 loss: 0.1740 2025/03/23 22:41:38 - mmengine - INFO - Iter(train) [12950/19176] lr: 5.0391e-06 eta: 2:31:26 time: 1.4964 data_time: 0.0139 memory: 11216 loss: 0.1639 2025/03/23 22:41:52 - mmengine - INFO - Iter(train) [12960/19176] lr: 5.0244e-06 eta: 2:31:11 time: 1.4376 data_time: 0.0143 memory: 11029 loss: 0.1520 2025/03/23 22:42:04 - mmengine - INFO - Iter(train) [12970/19176] lr: 5.0098e-06 eta: 2:30:55 time: 1.2279 data_time: 0.0128 memory: 10810 loss: 0.1514 2025/03/23 22:42:15 - mmengine - INFO - Iter(train) [12980/19176] lr: 4.9951e-06 eta: 2:30:39 time: 1.0299 data_time: 0.0124 memory: 10234 loss: 0.1715 2025/03/23 22:42:24 - mmengine - INFO - Iter(train) [12990/19176] lr: 4.9805e-06 eta: 2:30:21 time: 0.8982 data_time: 0.0112 memory: 12503 loss: 0.2171 2025/03/23 22:42:41 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 22:42:41 - mmengine - INFO - Iter(train) [13000/19176] lr: 4.9659e-06 eta: 2:30:08 time: 1.7574 data_time: 0.0149 memory: 12084 loss: 0.1641 2025/03/23 22:42:41 - mmengine - INFO - Saving checkpoint at 13000 iterations 2025/03/23 22:42:59 - mmengine - INFO - Iter(train) [13010/19176] lr: 4.9513e-06 eta: 2:29:55 time: 1.7760 data_time: 0.0914 memory: 11695 loss: 0.1422 2025/03/23 22:43:15 - mmengine - INFO - Iter(train) [13020/19176] lr: 4.9368e-06 eta: 2:29:41 time: 1.6355 data_time: 0.0147 memory: 11537 loss: 0.1552 2025/03/23 22:43:32 - mmengine - INFO - Iter(train) [13030/19176] lr: 4.9222e-06 eta: 2:29:27 time: 1.6179 data_time: 0.0147 memory: 11387 loss: 0.1499 2025/03/23 22:43:47 - mmengine - INFO - Iter(train) [13040/19176] lr: 4.9077e-06 eta: 2:29:13 time: 1.5521 data_time: 0.0146 memory: 11321 loss: 0.1601 2025/03/23 22:44:02 - mmengine - INFO - Iter(train) [13050/19176] lr: 4.8931e-06 eta: 2:28:59 time: 1.4823 data_time: 0.0143 memory: 11186 loss: 0.1688 2025/03/23 22:44:16 - mmengine - INFO - Iter(train) [13060/19176] lr: 4.8786e-06 eta: 2:28:44 time: 1.4039 data_time: 0.0140 memory: 10976 loss: 0.1961 2025/03/23 22:44:28 - mmengine - INFO - Iter(train) [13070/19176] lr: 4.8641e-06 eta: 2:28:28 time: 1.1973 data_time: 0.0132 memory: 10639 loss: 0.2097 2025/03/23 22:44:39 - mmengine - INFO - Iter(train) [13080/19176] lr: 4.8496e-06 eta: 2:28:12 time: 1.0579 data_time: 0.0123 memory: 10235 loss: 0.1742 2025/03/23 22:44:51 - mmengine - INFO - Iter(train) [13090/19176] lr: 4.8352e-06 eta: 2:27:56 time: 1.2001 data_time: 0.0121 memory: 16945 loss: 0.1700 2025/03/23 22:45:10 - mmengine - INFO - Iter(train) [13100/19176] lr: 4.8207e-06 eta: 2:27:43 time: 1.9212 data_time: 0.0145 memory: 14188 loss: 0.1575 2025/03/23 22:45:27 - mmengine - INFO - Iter(train) [13110/19176] lr: 4.8063e-06 eta: 2:27:30 time: 1.7399 data_time: 0.0144 memory: 11928 loss: 0.2127 2025/03/23 22:45:44 - mmengine - INFO - Iter(train) [13120/19176] lr: 4.7918e-06 eta: 2:27:17 time: 1.6901 data_time: 0.0154 memory: 11606 loss: 0.1709 2025/03/23 22:46:00 - mmengine - INFO - Iter(train) [13130/19176] lr: 4.7774e-06 eta: 2:27:03 time: 1.6154 data_time: 0.0143 memory: 11533 loss: 0.1665 2025/03/23 22:46:16 - mmengine - INFO - Iter(train) [13140/19176] lr: 4.7630e-06 eta: 2:26:49 time: 1.5482 data_time: 0.0143 memory: 11337 loss: 0.1825 2025/03/23 22:46:31 - mmengine - INFO - Iter(train) [13150/19176] lr: 4.7487e-06 eta: 2:26:34 time: 1.4890 data_time: 0.0144 memory: 11117 loss: 0.1515 2025/03/23 22:46:45 - mmengine - INFO - Iter(train) [13160/19176] lr: 4.7343e-06 eta: 2:26:19 time: 1.4423 data_time: 0.0143 memory: 11033 loss: 0.1489 2025/03/23 22:46:58 - mmengine - INFO - Iter(train) [13170/19176] lr: 4.7199e-06 eta: 2:26:04 time: 1.2823 data_time: 0.0134 memory: 10846 loss: 0.1650 2025/03/23 22:47:09 - mmengine - INFO - Iter(train) [13180/19176] lr: 4.7056e-06 eta: 2:25:48 time: 1.0793 data_time: 0.0126 memory: 10317 loss: 0.1710 2025/03/23 22:47:20 - mmengine - INFO - Iter(train) [13190/19176] lr: 4.6913e-06 eta: 2:25:32 time: 1.1423 data_time: 0.0115 memory: 14704 loss: 0.1671 2025/03/23 22:47:39 - mmengine - INFO - Iter(train) [13200/19176] lr: 4.6770e-06 eta: 2:25:19 time: 1.9185 data_time: 0.0145 memory: 13317 loss: 0.1537 2025/03/23 22:47:57 - mmengine - INFO - Iter(train) [13210/19176] lr: 4.6627e-06 eta: 2:25:06 time: 1.7584 data_time: 0.0140 memory: 11949 loss: 0.1409 2025/03/23 22:48:14 - mmengine - INFO - Iter(train) [13220/19176] lr: 4.6484e-06 eta: 2:24:52 time: 1.6971 data_time: 0.0142 memory: 11681 loss: 0.1482 2025/03/23 22:48:30 - mmengine - INFO - Iter(train) [13230/19176] lr: 4.6341e-06 eta: 2:24:39 time: 1.6547 data_time: 0.0141 memory: 11691 loss: 0.1678 2025/03/23 22:48:46 - mmengine - INFO - Iter(train) [13240/19176] lr: 4.6199e-06 eta: 2:24:25 time: 1.5900 data_time: 0.0143 memory: 11820 loss: 0.1532 2025/03/23 22:49:02 - mmengine - INFO - Iter(train) [13250/19176] lr: 4.6057e-06 eta: 2:24:10 time: 1.5333 data_time: 0.0142 memory: 11212 loss: 0.1724 2025/03/23 22:49:16 - mmengine - INFO - Iter(train) [13260/19176] lr: 4.5915e-06 eta: 2:23:56 time: 1.4458 data_time: 0.0139 memory: 11058 loss: 0.1857 2025/03/23 22:49:28 - mmengine - INFO - Iter(train) [13270/19176] lr: 4.5773e-06 eta: 2:23:40 time: 1.2378 data_time: 0.0132 memory: 10814 loss: 0.1749 2025/03/23 22:49:38 - mmengine - INFO - Iter(train) [13280/19176] lr: 4.5631e-06 eta: 2:23:24 time: 0.9852 data_time: 0.0118 memory: 10135 loss: 0.1759 2025/03/23 22:49:49 - mmengine - INFO - Iter(train) [13290/19176] lr: 4.5489e-06 eta: 2:23:07 time: 1.0380 data_time: 0.0117 memory: 13915 loss: 0.1806 2025/03/23 22:50:08 - mmengine - INFO - Iter(train) [13300/19176] lr: 4.5348e-06 eta: 2:22:54 time: 1.8890 data_time: 0.0142 memory: 13390 loss: 0.1954 2025/03/23 22:50:25 - mmengine - INFO - Iter(train) [13310/19176] lr: 4.5206e-06 eta: 2:22:41 time: 1.7129 data_time: 0.0142 memory: 11852 loss: 0.1796 2025/03/23 22:50:41 - mmengine - INFO - Iter(train) [13320/19176] lr: 4.5065e-06 eta: 2:22:27 time: 1.6621 data_time: 0.0142 memory: 11537 loss: 0.1520 2025/03/23 22:50:58 - mmengine - INFO - Iter(train) [13330/19176] lr: 4.4924e-06 eta: 2:22:13 time: 1.6213 data_time: 0.0143 memory: 11450 loss: 0.1454 2025/03/23 22:51:13 - mmengine - INFO - Iter(train) [13340/19176] lr: 4.4783e-06 eta: 2:21:59 time: 1.5419 data_time: 0.0141 memory: 11301 loss: 0.1806 2025/03/23 22:51:28 - mmengine - INFO - Iter(train) [13350/19176] lr: 4.4642e-06 eta: 2:21:45 time: 1.4952 data_time: 0.0140 memory: 11130 loss: 0.1459 2025/03/23 22:51:42 - mmengine - INFO - Iter(train) [13360/19176] lr: 4.4502e-06 eta: 2:21:30 time: 1.4122 data_time: 0.0140 memory: 11002 loss: 0.1696 2025/03/23 22:51:54 - mmengine - INFO - Iter(train) [13370/19176] lr: 4.4361e-06 eta: 2:21:14 time: 1.2442 data_time: 0.0135 memory: 10707 loss: 0.1573 2025/03/23 22:52:05 - mmengine - INFO - Iter(train) [13380/19176] lr: 4.4221e-06 eta: 2:20:58 time: 1.0338 data_time: 0.0121 memory: 10296 loss: 0.1636 2025/03/23 22:52:15 - mmengine - INFO - Iter(train) [13390/19176] lr: 4.4081e-06 eta: 2:20:41 time: 0.9957 data_time: 0.0112 memory: 13534 loss: 0.1761 2025/03/23 22:52:33 - mmengine - INFO - Iter(train) [13400/19176] lr: 4.3941e-06 eta: 2:20:28 time: 1.8748 data_time: 0.0142 memory: 12682 loss: 0.1774 2025/03/23 22:52:51 - mmengine - INFO - Iter(train) [13410/19176] lr: 4.3801e-06 eta: 2:20:15 time: 1.7335 data_time: 0.0144 memory: 11891 loss: 0.1446 2025/03/23 22:53:07 - mmengine - INFO - Iter(train) [13420/19176] lr: 4.3662e-06 eta: 2:20:01 time: 1.6623 data_time: 0.0145 memory: 11573 loss: 0.1575 2025/03/23 22:53:24 - mmengine - INFO - Iter(train) [13430/19176] lr: 4.3522e-06 eta: 2:19:47 time: 1.6217 data_time: 0.0140 memory: 11437 loss: 0.1561 2025/03/23 22:53:39 - mmengine - INFO - Iter(train) [13440/19176] lr: 4.3383e-06 eta: 2:19:33 time: 1.5701 data_time: 0.0146 memory: 11344 loss: 0.1584 2025/03/23 22:53:54 - mmengine - INFO - Iter(train) [13450/19176] lr: 4.3244e-06 eta: 2:19:19 time: 1.5080 data_time: 0.0142 memory: 11192 loss: 0.1612 2025/03/23 22:54:09 - mmengine - INFO - Iter(train) [13460/19176] lr: 4.3105e-06 eta: 2:19:04 time: 1.4073 data_time: 0.0143 memory: 11034 loss: 0.1880 2025/03/23 22:54:20 - mmengine - INFO - Iter(train) [13470/19176] lr: 4.2966e-06 eta: 2:18:48 time: 1.1950 data_time: 0.0132 memory: 10583 loss: 0.1367 2025/03/23 22:54:31 - mmengine - INFO - Iter(train) [13480/19176] lr: 4.2827e-06 eta: 2:18:32 time: 1.0461 data_time: 0.0123 memory: 10205 loss: 0.1727 2025/03/23 22:54:42 - mmengine - INFO - Iter(train) [13490/19176] lr: 4.2689e-06 eta: 2:18:16 time: 1.1042 data_time: 0.0119 memory: 12866 loss: 0.1650 2025/03/23 22:55:00 - mmengine - INFO - Iter(train) [13500/19176] lr: 4.2551e-06 eta: 2:18:03 time: 1.8362 data_time: 0.0138 memory: 13015 loss: 0.1613 2025/03/23 22:55:17 - mmengine - INFO - Iter(train) [13510/19176] lr: 4.2412e-06 eta: 2:17:49 time: 1.7092 data_time: 0.0141 memory: 11858 loss: 0.1655 2025/03/23 22:55:34 - mmengine - INFO - Iter(train) [13520/19176] lr: 4.2274e-06 eta: 2:17:36 time: 1.6499 data_time: 0.0140 memory: 11595 loss: 0.1528 2025/03/23 22:55:50 - mmengine - INFO - Iter(train) [13530/19176] lr: 4.2137e-06 eta: 2:17:22 time: 1.6074 data_time: 0.0146 memory: 11428 loss: 0.1749 2025/03/23 22:56:05 - mmengine - INFO - Iter(train) [13540/19176] lr: 4.1999e-06 eta: 2:17:07 time: 1.5473 data_time: 0.0140 memory: 11344 loss: 0.1707 2025/03/23 22:56:20 - mmengine - INFO - Iter(train) [13550/19176] lr: 4.1861e-06 eta: 2:16:53 time: 1.4179 data_time: 0.0145 memory: 11010 loss: 0.1531 2025/03/23 22:56:32 - mmengine - INFO - Iter(train) [13560/19176] lr: 4.1724e-06 eta: 2:16:37 time: 1.2747 data_time: 0.0144 memory: 10811 loss: 0.3073 2025/03/23 22:56:43 - mmengine - INFO - Iter(train) [13570/19176] lr: 4.1587e-06 eta: 2:16:21 time: 1.0800 data_time: 0.0125 memory: 10391 loss: 0.1904 2025/03/23 22:56:53 - mmengine - INFO - Iter(train) [13580/19176] lr: 4.1450e-06 eta: 2:16:04 time: 0.9533 data_time: 0.0120 memory: 9971 loss: 0.1672 2025/03/23 22:57:04 - mmengine - INFO - Iter(train) [13590/19176] lr: 4.1313e-06 eta: 2:15:48 time: 1.1036 data_time: 0.0112 memory: 18264 loss: 0.1778 2025/03/23 22:57:22 - mmengine - INFO - Iter(train) [13600/19176] lr: 4.1176e-06 eta: 2:15:35 time: 1.8127 data_time: 0.0143 memory: 13186 loss: 0.1585 2025/03/23 22:57:39 - mmengine - INFO - Iter(train) [13610/19176] lr: 4.1040e-06 eta: 2:15:22 time: 1.6874 data_time: 0.0144 memory: 11740 loss: 0.1682 2025/03/23 22:57:55 - mmengine - INFO - Iter(train) [13620/19176] lr: 4.0904e-06 eta: 2:15:08 time: 1.6243 data_time: 0.0141 memory: 11479 loss: 0.1542 2025/03/23 22:58:10 - mmengine - INFO - Iter(train) [13630/19176] lr: 4.0767e-06 eta: 2:14:53 time: 1.5447 data_time: 0.0144 memory: 11283 loss: 0.1780 2025/03/23 22:58:25 - mmengine - INFO - Iter(train) [13640/19176] lr: 4.0631e-06 eta: 2:14:39 time: 1.4863 data_time: 0.0140 memory: 11127 loss: 0.1721 2025/03/23 22:58:40 - mmengine - INFO - Iter(train) [13650/19176] lr: 4.0496e-06 eta: 2:14:24 time: 1.4220 data_time: 0.0134 memory: 11023 loss: 0.1740 2025/03/23 22:58:53 - mmengine - INFO - Iter(train) [13660/19176] lr: 4.0360e-06 eta: 2:14:09 time: 1.3571 data_time: 0.0141 memory: 10836 loss: 0.2883 2025/03/23 22:59:05 - mmengine - INFO - Iter(train) [13670/19176] lr: 4.0224e-06 eta: 2:13:54 time: 1.2135 data_time: 0.0135 memory: 10691 loss: 0.1592 2025/03/23 22:59:15 - mmengine - INFO - Iter(train) [13680/19176] lr: 4.0089e-06 eta: 2:13:37 time: 0.9725 data_time: 0.0121 memory: 10012 loss: 0.1692 2025/03/23 22:59:24 - mmengine - INFO - Iter(train) [13690/19176] lr: 3.9954e-06 eta: 2:13:20 time: 0.9442 data_time: 0.0116 memory: 12729 loss: 0.1434 2025/03/23 22:59:42 - mmengine - INFO - Iter(train) [13700/19176] lr: 3.9819e-06 eta: 2:13:07 time: 1.7508 data_time: 0.0143 memory: 12103 loss: 0.1579 2025/03/23 22:59:59 - mmengine - INFO - Iter(train) [13710/19176] lr: 3.9684e-06 eta: 2:12:53 time: 1.6778 data_time: 0.0145 memory: 11731 loss: 0.1788 2025/03/23 23:00:15 - mmengine - INFO - Iter(train) [13720/19176] lr: 3.9550e-06 eta: 2:12:39 time: 1.6171 data_time: 0.0142 memory: 11436 loss: 0.1552 2025/03/23 23:00:31 - mmengine - INFO - Iter(train) [13730/19176] lr: 3.9415e-06 eta: 2:12:25 time: 1.5720 data_time: 0.0143 memory: 11337 loss: 0.1738 2025/03/23 23:00:46 - mmengine - INFO - Iter(train) [13740/19176] lr: 3.9281e-06 eta: 2:12:11 time: 1.5460 data_time: 0.0144 memory: 11271 loss: 0.1780 2025/03/23 23:01:01 - mmengine - INFO - Iter(train) [13750/19176] lr: 3.9147e-06 eta: 2:11:56 time: 1.4871 data_time: 0.0144 memory: 11150 loss: 0.1873 2025/03/23 23:01:15 - mmengine - INFO - Iter(train) [13760/19176] lr: 3.9013e-06 eta: 2:11:41 time: 1.3809 data_time: 0.0135 memory: 10922 loss: 0.1869 2025/03/23 23:01:26 - mmengine - INFO - Iter(train) [13770/19176] lr: 3.8879e-06 eta: 2:11:26 time: 1.1413 data_time: 0.0126 memory: 10452 loss: 0.1512 2025/03/23 23:01:36 - mmengine - INFO - Iter(train) [13780/19176] lr: 3.8745e-06 eta: 2:11:09 time: 0.9611 data_time: 0.0122 memory: 10101 loss: 0.1775 2025/03/23 23:01:47 - mmengine - INFO - Iter(train) [13790/19176] lr: 3.8612e-06 eta: 2:10:53 time: 1.0964 data_time: 0.0107 memory: 18234 loss: 0.2344 2025/03/23 23:02:05 - mmengine - INFO - Iter(train) [13800/19176] lr: 3.8479e-06 eta: 2:10:40 time: 1.8421 data_time: 0.0141 memory: 13092 loss: 0.1638 2025/03/23 23:02:22 - mmengine - INFO - Iter(train) [13810/19176] lr: 3.8346e-06 eta: 2:10:26 time: 1.7067 data_time: 0.0145 memory: 11803 loss: 0.1377 2025/03/23 23:02:39 - mmengine - INFO - Iter(train) [13820/19176] lr: 3.8213e-06 eta: 2:10:13 time: 1.6613 data_time: 0.0142 memory: 11586 loss: 0.1537 2025/03/23 23:02:55 - mmengine - INFO - Iter(train) [13830/19176] lr: 3.8080e-06 eta: 2:09:59 time: 1.6176 data_time: 0.0138 memory: 11448 loss: 0.1762 2025/03/23 23:03:11 - mmengine - INFO - Iter(train) [13840/19176] lr: 3.7948e-06 eta: 2:09:45 time: 1.5813 data_time: 0.0138 memory: 11386 loss: 0.1418 2025/03/23 23:03:26 - mmengine - INFO - Iter(train) [13850/19176] lr: 3.7815e-06 eta: 2:09:30 time: 1.5304 data_time: 0.0146 memory: 11276 loss: 0.1791 2025/03/23 23:03:41 - mmengine - INFO - Iter(train) [13860/19176] lr: 3.7683e-06 eta: 2:09:16 time: 1.4805 data_time: 0.0147 memory: 11133 loss: 0.1581 2025/03/23 23:03:55 - mmengine - INFO - Iter(train) [13870/19176] lr: 3.7551e-06 eta: 2:09:01 time: 1.3613 data_time: 0.0141 memory: 11066 loss: 0.1844 2025/03/23 23:04:06 - mmengine - INFO - Iter(train) [13880/19176] lr: 3.7419e-06 eta: 2:08:45 time: 1.0982 data_time: 0.0128 memory: 10579 loss: 0.1563 2025/03/23 23:04:16 - mmengine - INFO - Iter(train) [13890/19176] lr: 3.7288e-06 eta: 2:08:29 time: 1.0417 data_time: 0.0130 memory: 13253 loss: 0.1622 2025/03/23 23:04:34 - mmengine - INFO - Iter(train) [13900/19176] lr: 3.7156e-06 eta: 2:08:15 time: 1.8408 data_time: 0.0158 memory: 12677 loss: 0.1550 2025/03/23 23:04:52 - mmengine - INFO - Iter(train) [13910/19176] lr: 3.7025e-06 eta: 2:08:02 time: 1.7398 data_time: 0.0144 memory: 11900 loss: 0.1522 2025/03/23 23:05:08 - mmengine - INFO - Iter(train) [13920/19176] lr: 3.6894e-06 eta: 2:07:48 time: 1.6476 data_time: 0.0155 memory: 11533 loss: 0.1662 2025/03/23 23:05:24 - mmengine - INFO - Iter(train) [13930/19176] lr: 3.6763e-06 eta: 2:07:34 time: 1.6100 data_time: 0.0150 memory: 11412 loss: 0.1929 2025/03/23 23:05:40 - mmengine - INFO - Iter(train) [13940/19176] lr: 3.6632e-06 eta: 2:07:20 time: 1.5386 data_time: 0.0146 memory: 11252 loss: 0.1717 2025/03/23 23:05:55 - mmengine - INFO - Iter(train) [13950/19176] lr: 3.6502e-06 eta: 2:07:05 time: 1.4999 data_time: 0.0151 memory: 11127 loss: 0.1643 2025/03/23 23:06:09 - mmengine - INFO - Iter(train) [13960/19176] lr: 3.6371e-06 eta: 2:06:51 time: 1.4233 data_time: 0.0145 memory: 11019 loss: 0.1732 2025/03/23 23:06:21 - mmengine - INFO - Iter(train) [13970/19176] lr: 3.6241e-06 eta: 2:06:35 time: 1.2418 data_time: 0.0136 memory: 10788 loss: 0.1583 2025/03/23 23:06:32 - mmengine - INFO - Iter(train) [13980/19176] lr: 3.6111e-06 eta: 2:06:19 time: 1.0212 data_time: 0.0126 memory: 10186 loss: 0.1664 2025/03/23 23:06:43 - mmengine - INFO - Iter(train) [13990/19176] lr: 3.5981e-06 eta: 2:06:03 time: 1.1579 data_time: 0.0120 memory: 18682 loss: 0.1852 2025/03/23 23:07:01 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 23:07:01 - mmengine - INFO - Iter(train) [14000/19176] lr: 3.5851e-06 eta: 2:05:50 time: 1.7728 data_time: 0.0147 memory: 12192 loss: 0.1433 2025/03/23 23:07:01 - mmengine - INFO - Saving checkpoint at 14000 iterations 2025/03/23 23:07:18 - mmengine - INFO - Iter(train) [14010/19176] lr: 3.5722e-06 eta: 2:05:36 time: 1.7195 data_time: 0.0925 memory: 11577 loss: 0.1552 2025/03/23 23:07:34 - mmengine - INFO - Iter(train) [14020/19176] lr: 3.5593e-06 eta: 2:05:22 time: 1.6348 data_time: 0.0147 memory: 11453 loss: 0.1784 2025/03/23 23:07:50 - mmengine - INFO - Iter(train) [14030/19176] lr: 3.5464e-06 eta: 2:05:08 time: 1.5946 data_time: 0.0153 memory: 11367 loss: 0.1671 2025/03/23 23:08:06 - mmengine - INFO - Iter(train) [14040/19176] lr: 3.5335e-06 eta: 2:04:54 time: 1.5503 data_time: 0.0149 memory: 11255 loss: 0.2122 2025/03/23 23:08:21 - mmengine - INFO - Iter(train) [14050/19176] lr: 3.5206e-06 eta: 2:04:40 time: 1.5197 data_time: 0.0158 memory: 11202 loss: 0.1697 2025/03/23 23:08:36 - mmengine - INFO - Iter(train) [14060/19176] lr: 3.5077e-06 eta: 2:04:25 time: 1.4614 data_time: 0.0156 memory: 11154 loss: 0.1647 2025/03/23 23:08:48 - mmengine - INFO - Iter(train) [14070/19176] lr: 3.4949e-06 eta: 2:04:10 time: 1.2393 data_time: 0.0143 memory: 10802 loss: 0.1585 2025/03/23 23:08:59 - mmengine - INFO - Iter(train) [14080/19176] lr: 3.4821e-06 eta: 2:03:53 time: 1.0427 data_time: 0.0123 memory: 10192 loss: 0.1629 2025/03/23 23:09:10 - mmengine - INFO - Iter(train) [14090/19176] lr: 3.4693e-06 eta: 2:03:38 time: 1.1226 data_time: 0.0119 memory: 14154 loss: 0.1801 2025/03/23 23:09:29 - mmengine - INFO - Iter(train) [14100/19176] lr: 3.4565e-06 eta: 2:03:25 time: 1.8941 data_time: 0.0153 memory: 12781 loss: 0.1482 2025/03/23 23:09:46 - mmengine - INFO - Iter(train) [14110/19176] lr: 3.4437e-06 eta: 2:03:11 time: 1.7292 data_time: 0.0143 memory: 11883 loss: 0.1818 2025/03/23 23:10:02 - mmengine - INFO - Iter(train) [14120/19176] lr: 3.4310e-06 eta: 2:02:57 time: 1.6523 data_time: 0.0141 memory: 11697 loss: 0.1657 2025/03/23 23:10:18 - mmengine - INFO - Iter(train) [14130/19176] lr: 3.4183e-06 eta: 2:02:43 time: 1.5890 data_time: 0.0144 memory: 11389 loss: 0.1449 2025/03/23 23:10:34 - mmengine - INFO - Iter(train) [14140/19176] lr: 3.4056e-06 eta: 2:02:29 time: 1.5178 data_time: 0.0141 memory: 11194 loss: 0.1677 2025/03/23 23:10:48 - mmengine - INFO - Iter(train) [14150/19176] lr: 3.3929e-06 eta: 2:02:14 time: 1.4583 data_time: 0.0142 memory: 11088 loss: 0.2026 2025/03/23 23:11:01 - mmengine - INFO - Iter(train) [14160/19176] lr: 3.3802e-06 eta: 2:01:59 time: 1.3330 data_time: 0.0141 memory: 10855 loss: 0.1697 2025/03/23 23:11:13 - mmengine - INFO - Iter(train) [14170/19176] lr: 3.3676e-06 eta: 2:01:43 time: 1.1694 data_time: 0.0126 memory: 10504 loss: 0.1549 2025/03/23 23:11:23 - mmengine - INFO - Iter(train) [14180/19176] lr: 3.3549e-06 eta: 2:01:27 time: 1.0132 data_time: 0.0120 memory: 10281 loss: 0.1682 2025/03/23 23:11:35 - mmengine - INFO - Iter(train) [14190/19176] lr: 3.3423e-06 eta: 2:01:12 time: 1.2116 data_time: 0.0119 memory: 16526 loss: 0.1548 2025/03/23 23:11:54 - mmengine - INFO - Iter(train) [14200/19176] lr: 3.3297e-06 eta: 2:00:59 time: 1.8801 data_time: 0.0147 memory: 13802 loss: 0.1639 2025/03/23 23:12:11 - mmengine - INFO - Iter(train) [14210/19176] lr: 3.3171e-06 eta: 2:00:45 time: 1.7271 data_time: 0.0138 memory: 11891 loss: 0.1767 2025/03/23 23:12:28 - mmengine - INFO - Iter(train) [14220/19176] lr: 3.3046e-06 eta: 2:00:31 time: 1.6787 data_time: 0.0142 memory: 11737 loss: 0.1644 2025/03/23 23:12:45 - mmengine - INFO - Iter(train) [14230/19176] lr: 3.2921e-06 eta: 2:00:17 time: 1.6350 data_time: 0.0140 memory: 11501 loss: 0.1462 2025/03/23 23:13:00 - mmengine - INFO - Iter(train) [14240/19176] lr: 3.2795e-06 eta: 2:00:03 time: 1.5741 data_time: 0.0152 memory: 11357 loss: 0.1585 2025/03/23 23:13:15 - mmengine - INFO - Iter(train) [14250/19176] lr: 3.2670e-06 eta: 1:59:49 time: 1.4928 data_time: 0.0145 memory: 11147 loss: 0.1734 2025/03/23 23:13:30 - mmengine - INFO - Iter(train) [14260/19176] lr: 3.2546e-06 eta: 1:59:34 time: 1.4343 data_time: 0.0140 memory: 11040 loss: 0.1820 2025/03/23 23:13:42 - mmengine - INFO - Iter(train) [14270/19176] lr: 3.2421e-06 eta: 1:59:18 time: 1.1914 data_time: 0.0134 memory: 10619 loss: 0.1905 2025/03/23 23:13:52 - mmengine - INFO - Iter(train) [14280/19176] lr: 3.2297e-06 eta: 1:59:02 time: 1.0799 data_time: 0.0132 memory: 10248 loss: 0.1429 2025/03/23 23:14:03 - mmengine - INFO - Iter(train) [14290/19176] lr: 3.2172e-06 eta: 1:58:47 time: 1.0936 data_time: 0.0118 memory: 13459 loss: 0.2051 2025/03/23 23:14:22 - mmengine - INFO - Iter(train) [14300/19176] lr: 3.2048e-06 eta: 1:58:34 time: 1.9134 data_time: 0.0147 memory: 13099 loss: 0.1719 2025/03/23 23:14:40 - mmengine - INFO - Iter(train) [14310/19176] lr: 3.1925e-06 eta: 1:58:20 time: 1.7368 data_time: 0.0148 memory: 11883 loss: 0.1351 2025/03/23 23:14:57 - mmengine - INFO - Iter(train) [14320/19176] lr: 3.1801e-06 eta: 1:58:06 time: 1.6772 data_time: 0.0145 memory: 11688 loss: 0.1598 2025/03/23 23:15:13 - mmengine - INFO - Iter(train) [14330/19176] lr: 3.1678e-06 eta: 1:57:52 time: 1.6132 data_time: 0.0143 memory: 11400 loss: 0.1539 2025/03/23 23:15:28 - mmengine - INFO - Iter(train) [14340/19176] lr: 3.1554e-06 eta: 1:57:38 time: 1.5251 data_time: 0.0145 memory: 11255 loss: 0.1538 2025/03/23 23:15:43 - mmengine - INFO - Iter(train) [14350/19176] lr: 3.1431e-06 eta: 1:57:23 time: 1.4729 data_time: 0.0143 memory: 11028 loss: 0.1624 2025/03/23 23:15:56 - mmengine - INFO - Iter(train) [14360/19176] lr: 3.1309e-06 eta: 1:57:08 time: 1.3552 data_time: 0.0140 memory: 10915 loss: 0.2035 2025/03/23 23:16:07 - mmengine - INFO - Iter(train) [14370/19176] lr: 3.1186e-06 eta: 1:56:52 time: 1.1069 data_time: 0.0125 memory: 10458 loss: 0.1676 2025/03/23 23:16:16 - mmengine - INFO - Iter(train) [14380/19176] lr: 3.1063e-06 eta: 1:56:36 time: 0.9090 data_time: 0.0117 memory: 10025 loss: 0.1790 2025/03/23 23:16:37 - mmengine - INFO - Iter(train) [14390/19176] lr: 3.0941e-06 eta: 1:56:23 time: 2.0581 data_time: 0.2605 memory: 19198 loss: 0.1663 2025/03/23 23:16:54 - mmengine - INFO - Iter(train) [14400/19176] lr: 3.0819e-06 eta: 1:56:10 time: 1.7302 data_time: 0.0149 memory: 11815 loss: 0.1626 2025/03/23 23:17:11 - mmengine - INFO - Iter(train) [14410/19176] lr: 3.0697e-06 eta: 1:55:56 time: 1.6700 data_time: 0.0144 memory: 11675 loss: 0.1397 2025/03/23 23:17:27 - mmengine - INFO - Iter(train) [14420/19176] lr: 3.0576e-06 eta: 1:55:42 time: 1.6103 data_time: 0.0147 memory: 11424 loss: 0.1409 2025/03/23 23:17:43 - mmengine - INFO - Iter(train) [14430/19176] lr: 3.0454e-06 eta: 1:55:27 time: 1.5537 data_time: 0.0139 memory: 11313 loss: 0.1444 2025/03/23 23:17:58 - mmengine - INFO - Iter(train) [14440/19176] lr: 3.0333e-06 eta: 1:55:13 time: 1.5091 data_time: 0.0152 memory: 11173 loss: 0.1329 2025/03/23 23:18:12 - mmengine - INFO - Iter(train) [14450/19176] lr: 3.0212e-06 eta: 1:54:58 time: 1.4660 data_time: 0.0152 memory: 11010 loss: 0.1481 2025/03/23 23:18:26 - mmengine - INFO - Iter(train) [14460/19176] lr: 3.0091e-06 eta: 1:54:43 time: 1.3726 data_time: 0.0140 memory: 10914 loss: 0.1255 2025/03/23 23:18:38 - mmengine - INFO - Iter(train) [14470/19176] lr: 2.9970e-06 eta: 1:54:28 time: 1.2016 data_time: 0.0133 memory: 10702 loss: 0.1612 2025/03/23 23:18:47 - mmengine - INFO - Iter(train) [14480/19176] lr: 2.9850e-06 eta: 1:54:12 time: 0.9340 data_time: 0.0127 memory: 10171 loss: 0.1484 2025/03/23 23:19:05 - mmengine - INFO - Iter(train) [14490/19176] lr: 2.9730e-06 eta: 1:53:58 time: 1.7682 data_time: 0.0141 memory: 14721 loss: 0.1623 2025/03/23 23:19:22 - mmengine - INFO - Iter(train) [14500/19176] lr: 2.9610e-06 eta: 1:53:44 time: 1.7341 data_time: 0.0145 memory: 11935 loss: 0.1706 2025/03/23 23:19:39 - mmengine - INFO - Iter(train) [14510/19176] lr: 2.9490e-06 eta: 1:53:31 time: 1.6797 data_time: 0.0146 memory: 11651 loss: 0.1425 2025/03/23 23:19:56 - mmengine - INFO - Iter(train) [14520/19176] lr: 2.9370e-06 eta: 1:53:16 time: 1.6355 data_time: 0.0146 memory: 11555 loss: 0.1346 2025/03/23 23:20:12 - mmengine - INFO - Iter(train) [14530/19176] lr: 2.9251e-06 eta: 1:53:02 time: 1.6029 data_time: 0.0145 memory: 11482 loss: 0.1385 2025/03/23 23:20:27 - mmengine - INFO - Iter(train) [14540/19176] lr: 2.9131e-06 eta: 1:52:48 time: 1.5462 data_time: 0.0142 memory: 11302 loss: 0.1520 2025/03/23 23:20:42 - mmengine - INFO - Iter(train) [14550/19176] lr: 2.9012e-06 eta: 1:52:34 time: 1.4912 data_time: 0.0146 memory: 11133 loss: 0.1545 2025/03/23 23:20:56 - mmengine - INFO - Iter(train) [14560/19176] lr: 2.8893e-06 eta: 1:52:19 time: 1.3884 data_time: 0.0141 memory: 10930 loss: 0.1492 2025/03/23 23:21:08 - mmengine - INFO - Iter(train) [14570/19176] lr: 2.8775e-06 eta: 1:52:03 time: 1.2242 data_time: 0.0128 memory: 10802 loss: 0.1451 2025/03/23 23:21:18 - mmengine - INFO - Iter(train) [14580/19176] lr: 2.8656e-06 eta: 1:51:47 time: 0.9905 data_time: 0.0123 memory: 10164 loss: 0.1311 2025/03/23 23:21:34 - mmengine - INFO - Iter(train) [14590/19176] lr: 2.8538e-06 eta: 1:51:33 time: 1.6054 data_time: 0.0129 memory: 12783 loss: 0.1449 2025/03/23 23:21:52 - mmengine - INFO - Iter(train) [14600/19176] lr: 2.8420e-06 eta: 1:51:19 time: 1.7405 data_time: 0.0140 memory: 11956 loss: 0.1677 2025/03/23 23:22:08 - mmengine - INFO - Iter(train) [14610/19176] lr: 2.8302e-06 eta: 1:51:05 time: 1.6519 data_time: 0.0143 memory: 11577 loss: 0.1473 2025/03/23 23:22:24 - mmengine - INFO - Iter(train) [14620/19176] lr: 2.8184e-06 eta: 1:50:51 time: 1.6218 data_time: 0.0146 memory: 11424 loss: 0.1420 2025/03/23 23:22:40 - mmengine - INFO - Iter(train) [14630/19176] lr: 2.8067e-06 eta: 1:50:37 time: 1.5590 data_time: 0.0146 memory: 11307 loss: 0.1699 2025/03/23 23:22:55 - mmengine - INFO - Iter(train) [14640/19176] lr: 2.7950e-06 eta: 1:50:23 time: 1.4925 data_time: 0.0145 memory: 11195 loss: 0.1592 2025/03/23 23:23:09 - mmengine - INFO - Iter(train) [14650/19176] lr: 2.7833e-06 eta: 1:50:08 time: 1.4563 data_time: 0.0156 memory: 11065 loss: 0.1356 2025/03/23 23:23:22 - mmengine - INFO - Iter(train) [14660/19176] lr: 2.7716e-06 eta: 1:49:53 time: 1.3114 data_time: 0.0140 memory: 10910 loss: 0.1458 2025/03/23 23:23:33 - mmengine - INFO - Iter(train) [14670/19176] lr: 2.7599e-06 eta: 1:49:37 time: 1.1017 data_time: 0.0129 memory: 10377 loss: 0.1495 2025/03/23 23:23:42 - mmengine - INFO - Iter(train) [14680/19176] lr: 2.7483e-06 eta: 1:49:21 time: 0.8419 data_time: 0.0115 memory: 9859 loss: 0.1720 2025/03/23 23:24:00 - mmengine - INFO - Iter(train) [14690/19176] lr: 2.7367e-06 eta: 1:49:07 time: 1.8013 data_time: 0.0133 memory: 15167 loss: 0.1827 2025/03/23 23:24:17 - mmengine - INFO - Iter(train) [14700/19176] lr: 2.7251e-06 eta: 1:48:53 time: 1.7344 data_time: 0.0148 memory: 12353 loss: 0.1565 2025/03/23 23:24:34 - mmengine - INFO - Iter(train) [14710/19176] lr: 2.7135e-06 eta: 1:48:39 time: 1.6895 data_time: 0.0146 memory: 12029 loss: 0.1381 2025/03/23 23:24:50 - mmengine - INFO - Iter(train) [14720/19176] lr: 2.7019e-06 eta: 1:48:25 time: 1.6089 data_time: 0.0144 memory: 11395 loss: 0.1434 2025/03/23 23:25:06 - mmengine - INFO - Iter(train) [14730/19176] lr: 2.6904e-06 eta: 1:48:11 time: 1.5752 data_time: 0.0147 memory: 11440 loss: 0.1331 2025/03/23 23:25:21 - mmengine - INFO - Iter(train) [14740/19176] lr: 2.6789e-06 eta: 1:47:57 time: 1.4907 data_time: 0.0140 memory: 11182 loss: 0.1455 2025/03/23 23:25:35 - mmengine - INFO - Iter(train) [14750/19176] lr: 2.6674e-06 eta: 1:47:42 time: 1.3618 data_time: 0.0140 memory: 10904 loss: 0.1368 2025/03/23 23:25:47 - mmengine - INFO - Iter(train) [14760/19176] lr: 2.6559e-06 eta: 1:47:26 time: 1.2016 data_time: 0.0129 memory: 10676 loss: 0.1487 2025/03/23 23:25:57 - mmengine - INFO - Iter(train) [14770/19176] lr: 2.6445e-06 eta: 1:47:11 time: 1.0680 data_time: 0.0132 memory: 10229 loss: 0.1560 2025/03/23 23:26:06 - mmengine - INFO - Iter(train) [14780/19176] lr: 2.6330e-06 eta: 1:46:54 time: 0.9242 data_time: 0.0124 memory: 10016 loss: 0.1283 2025/03/23 23:26:24 - mmengine - INFO - Iter(train) [14790/19176] lr: 2.6216e-06 eta: 1:46:41 time: 1.7953 data_time: 0.0142 memory: 15882 loss: 0.1653 2025/03/23 23:26:42 - mmengine - INFO - Iter(train) [14800/19176] lr: 2.6102e-06 eta: 1:46:27 time: 1.7433 data_time: 0.0145 memory: 11995 loss: 0.1441 2025/03/23 23:26:59 - mmengine - INFO - Iter(train) [14810/19176] lr: 2.5989e-06 eta: 1:46:13 time: 1.6770 data_time: 0.0149 memory: 11617 loss: 0.1411 2025/03/23 23:27:15 - mmengine - INFO - Iter(train) [14820/19176] lr: 2.5875e-06 eta: 1:45:59 time: 1.6380 data_time: 0.0150 memory: 11477 loss: 0.1552 2025/03/23 23:27:31 - mmengine - INFO - Iter(train) [14830/19176] lr: 2.5762e-06 eta: 1:45:45 time: 1.5789 data_time: 0.0150 memory: 11358 loss: 0.1211 2025/03/23 23:27:46 - mmengine - INFO - Iter(train) [14840/19176] lr: 2.5649e-06 eta: 1:45:30 time: 1.4980 data_time: 0.0145 memory: 11157 loss: 0.1350 2025/03/23 23:28:00 - mmengine - INFO - Iter(train) [14850/19176] lr: 2.5536e-06 eta: 1:45:15 time: 1.4190 data_time: 0.0138 memory: 11015 loss: 0.1367 2025/03/23 23:28:12 - mmengine - INFO - Iter(train) [14860/19176] lr: 2.5424e-06 eta: 1:45:00 time: 1.2162 data_time: 0.0135 memory: 10798 loss: 0.1587 2025/03/23 23:28:23 - mmengine - INFO - Iter(train) [14870/19176] lr: 2.5311e-06 eta: 1:44:44 time: 1.0594 data_time: 0.0129 memory: 10258 loss: 0.1559 2025/03/23 23:28:31 - mmengine - INFO - Iter(train) [14880/19176] lr: 2.5199e-06 eta: 1:44:28 time: 0.7967 data_time: 0.0117 memory: 9847 loss: 0.1538 2025/03/23 23:28:50 - mmengine - INFO - Iter(train) [14890/19176] lr: 2.5087e-06 eta: 1:44:15 time: 1.9374 data_time: 0.0133 memory: 18264 loss: 0.1501 2025/03/23 23:29:08 - mmengine - INFO - Iter(train) [14900/19176] lr: 2.4975e-06 eta: 1:44:01 time: 1.7803 data_time: 0.0149 memory: 12171 loss: 0.1438 2025/03/23 23:29:25 - mmengine - INFO - Iter(train) [14910/19176] lr: 2.4864e-06 eta: 1:43:47 time: 1.6863 data_time: 0.0149 memory: 11767 loss: 0.1445 2025/03/23 23:29:41 - mmengine - INFO - Iter(train) [14920/19176] lr: 2.4752e-06 eta: 1:43:33 time: 1.6195 data_time: 0.0145 memory: 11437 loss: 0.1315 2025/03/23 23:29:57 - mmengine - INFO - Iter(train) [14930/19176] lr: 2.4641e-06 eta: 1:43:19 time: 1.5601 data_time: 0.0147 memory: 11322 loss: 0.1697 2025/03/23 23:30:11 - mmengine - INFO - Iter(train) [14940/19176] lr: 2.4530e-06 eta: 1:43:04 time: 1.4817 data_time: 0.0140 memory: 11205 loss: 0.1632 2025/03/23 23:30:25 - mmengine - INFO - Iter(train) [14950/19176] lr: 2.4420e-06 eta: 1:42:49 time: 1.3801 data_time: 0.0141 memory: 10905 loss: 0.1414 2025/03/23 23:30:37 - mmengine - INFO - Iter(train) [14960/19176] lr: 2.4309e-06 eta: 1:42:34 time: 1.1735 data_time: 0.0131 memory: 10588 loss: 0.1421 2025/03/23 23:30:47 - mmengine - INFO - Iter(train) [14970/19176] lr: 2.4199e-06 eta: 1:42:18 time: 1.0520 data_time: 0.0125 memory: 10273 loss: 0.1388 2025/03/23 23:30:56 - mmengine - INFO - Iter(train) [14980/19176] lr: 2.4089e-06 eta: 1:42:02 time: 0.8295 data_time: 0.0122 memory: 9885 loss: 0.1319 2025/03/23 23:31:11 - mmengine - INFO - Iter(train) [14990/19176] lr: 2.3979e-06 eta: 1:41:47 time: 1.5772 data_time: 0.0135 memory: 12811 loss: 0.1399 2025/03/23 23:31:29 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 23:31:29 - mmengine - INFO - Iter(train) [15000/19176] lr: 2.3869e-06 eta: 1:41:34 time: 1.7120 data_time: 0.0147 memory: 12047 loss: 0.1365 2025/03/23 23:31:29 - mmengine - INFO - Saving checkpoint at 15000 iterations 2025/03/23 23:31:46 - mmengine - INFO - Iter(train) [15010/19176] lr: 2.3760e-06 eta: 1:41:20 time: 1.7331 data_time: 0.0925 memory: 11490 loss: 0.1431 2025/03/23 23:32:02 - mmengine - INFO - Iter(train) [15020/19176] lr: 2.3651e-06 eta: 1:41:06 time: 1.5974 data_time: 0.0142 memory: 11413 loss: 0.1432 2025/03/23 23:32:17 - mmengine - INFO - Iter(train) [15030/19176] lr: 2.3542e-06 eta: 1:40:51 time: 1.5391 data_time: 0.0150 memory: 11245 loss: 0.1436 2025/03/23 23:32:32 - mmengine - INFO - Iter(train) [15040/19176] lr: 2.3433e-06 eta: 1:40:37 time: 1.4892 data_time: 0.0145 memory: 11120 loss: 0.1563 2025/03/23 23:32:46 - mmengine - INFO - Iter(train) [15050/19176] lr: 2.3324e-06 eta: 1:40:22 time: 1.4060 data_time: 0.0142 memory: 11021 loss: 0.1745 2025/03/23 23:32:59 - mmengine - INFO - Iter(train) [15060/19176] lr: 2.3216e-06 eta: 1:40:07 time: 1.2623 data_time: 0.0135 memory: 10682 loss: 0.1496 2025/03/23 23:33:09 - mmengine - INFO - Iter(train) [15070/19176] lr: 2.3108e-06 eta: 1:39:51 time: 1.0618 data_time: 0.0127 memory: 10269 loss: 0.1476 2025/03/23 23:33:18 - mmengine - INFO - Iter(train) [15080/19176] lr: 2.3000e-06 eta: 1:39:35 time: 0.8599 data_time: 0.0124 memory: 9953 loss: 0.1531 2025/03/23 23:33:36 - mmengine - INFO - Iter(train) [15090/19176] lr: 2.2893e-06 eta: 1:39:21 time: 1.7578 data_time: 0.0138 memory: 15316 loss: 0.1587 2025/03/23 23:33:54 - mmengine - INFO - Iter(train) [15100/19176] lr: 2.2785e-06 eta: 1:39:07 time: 1.7978 data_time: 0.0147 memory: 12353 loss: 0.1506 2025/03/23 23:34:11 - mmengine - INFO - Iter(train) [15110/19176] lr: 2.2678e-06 eta: 1:38:54 time: 1.7189 data_time: 0.0152 memory: 11833 loss: 0.1434 2025/03/23 23:34:27 - mmengine - INFO - Iter(train) [15120/19176] lr: 2.2571e-06 eta: 1:38:39 time: 1.6420 data_time: 0.0149 memory: 11490 loss: 0.1432 2025/03/23 23:34:43 - mmengine - INFO - Iter(train) [15130/19176] lr: 2.2464e-06 eta: 1:38:25 time: 1.5875 data_time: 0.0150 memory: 11348 loss: 0.1405 2025/03/23 23:34:58 - mmengine - INFO - Iter(train) [15140/19176] lr: 2.2358e-06 eta: 1:38:11 time: 1.5323 data_time: 0.0148 memory: 11316 loss: 0.1332 2025/03/23 23:35:13 - mmengine - INFO - Iter(train) [15150/19176] lr: 2.2251e-06 eta: 1:37:56 time: 1.4836 data_time: 0.0146 memory: 11115 loss: 0.1349 2025/03/23 23:35:28 - mmengine - INFO - Iter(train) [15160/19176] lr: 2.2145e-06 eta: 1:37:42 time: 1.4300 data_time: 0.0145 memory: 10989 loss: 0.1606 2025/03/23 23:35:39 - mmengine - INFO - Iter(train) [15170/19176] lr: 2.2039e-06 eta: 1:37:26 time: 1.1927 data_time: 0.0135 memory: 10813 loss: 0.1523 2025/03/23 23:35:48 - mmengine - INFO - Iter(train) [15180/19176] lr: 2.1934e-06 eta: 1:37:10 time: 0.8297 data_time: 0.0121 memory: 9967 loss: 0.1490 2025/03/23 23:36:04 - mmengine - INFO - Iter(train) [15190/19176] lr: 2.1828e-06 eta: 1:36:56 time: 1.5812 data_time: 0.0130 memory: 13030 loss: 0.1295 2025/03/23 23:36:21 - mmengine - INFO - Iter(train) [15200/19176] lr: 2.1723e-06 eta: 1:36:42 time: 1.7641 data_time: 0.0147 memory: 12058 loss: 0.1380 2025/03/23 23:36:38 - mmengine - INFO - Iter(train) [15210/19176] lr: 2.1618e-06 eta: 1:36:28 time: 1.6591 data_time: 0.0145 memory: 11593 loss: 0.1551 2025/03/23 23:36:54 - mmengine - INFO - Iter(train) [15220/19176] lr: 2.1513e-06 eta: 1:36:14 time: 1.6127 data_time: 0.0144 memory: 11402 loss: 0.1307 2025/03/23 23:37:09 - mmengine - INFO - Iter(train) [15230/19176] lr: 2.1409e-06 eta: 1:35:59 time: 1.5294 data_time: 0.0144 memory: 11208 loss: 0.1345 2025/03/23 23:37:24 - mmengine - INFO - Iter(train) [15240/19176] lr: 2.1304e-06 eta: 1:35:45 time: 1.4805 data_time: 0.0144 memory: 11112 loss: 0.1498 2025/03/23 23:37:38 - mmengine - INFO - Iter(train) [15250/19176] lr: 2.1200e-06 eta: 1:35:30 time: 1.4142 data_time: 0.0141 memory: 11027 loss: 0.1544 2025/03/23 23:37:51 - mmengine - INFO - Iter(train) [15260/19176] lr: 2.1096e-06 eta: 1:35:15 time: 1.2477 data_time: 0.0132 memory: 10743 loss: 0.1439 2025/03/23 23:38:01 - mmengine - INFO - Iter(train) [15270/19176] lr: 2.0993e-06 eta: 1:34:59 time: 1.0597 data_time: 0.0125 memory: 10285 loss: 0.1254 2025/03/23 23:38:09 - mmengine - INFO - Iter(train) [15280/19176] lr: 2.0889e-06 eta: 1:34:43 time: 0.7930 data_time: 0.0121 memory: 9898 loss: 0.1617 2025/03/23 23:38:27 - mmengine - INFO - Iter(train) [15290/19176] lr: 2.0786e-06 eta: 1:34:29 time: 1.7980 data_time: 0.0135 memory: 16945 loss: 0.1721 2025/03/23 23:38:45 - mmengine - INFO - Iter(train) [15300/19176] lr: 2.0683e-06 eta: 1:34:15 time: 1.7534 data_time: 0.0147 memory: 12019 loss: 0.1620 2025/03/23 23:39:01 - mmengine - INFO - Iter(train) [15310/19176] lr: 2.0580e-06 eta: 1:34:01 time: 1.6714 data_time: 0.0144 memory: 11640 loss: 0.1437 2025/03/23 23:39:18 - mmengine - INFO - Iter(train) [15320/19176] lr: 2.0478e-06 eta: 1:33:47 time: 1.6337 data_time: 0.0141 memory: 11655 loss: 0.1351 2025/03/23 23:39:33 - mmengine - INFO - Iter(train) [15330/19176] lr: 2.0376e-06 eta: 1:33:33 time: 1.5362 data_time: 0.0138 memory: 11260 loss: 0.1473 2025/03/23 23:39:48 - mmengine - INFO - Iter(train) [15340/19176] lr: 2.0273e-06 eta: 1:33:18 time: 1.4597 data_time: 0.0140 memory: 11111 loss: 0.1385 2025/03/23 23:40:02 - mmengine - INFO - Iter(train) [15350/19176] lr: 2.0172e-06 eta: 1:33:03 time: 1.3822 data_time: 0.0132 memory: 10998 loss: 0.1461 2025/03/23 23:40:14 - mmengine - INFO - Iter(train) [15360/19176] lr: 2.0070e-06 eta: 1:32:48 time: 1.2583 data_time: 0.0124 memory: 10672 loss: 0.1532 2025/03/23 23:40:25 - mmengine - INFO - Iter(train) [15370/19176] lr: 1.9969e-06 eta: 1:32:33 time: 1.0619 data_time: 0.0119 memory: 10359 loss: 0.1653 2025/03/23 23:40:33 - mmengine - INFO - Iter(train) [15380/19176] lr: 1.9868e-06 eta: 1:32:17 time: 0.8075 data_time: 0.0117 memory: 9925 loss: 0.1417 2025/03/23 23:40:49 - mmengine - INFO - Iter(train) [15390/19176] lr: 1.9767e-06 eta: 1:32:02 time: 1.6219 data_time: 0.0125 memory: 13317 loss: 0.2046 2025/03/23 23:41:07 - mmengine - INFO - Iter(train) [15400/19176] lr: 1.9666e-06 eta: 1:31:48 time: 1.7620 data_time: 0.0137 memory: 12077 loss: 0.1412 2025/03/23 23:41:23 - mmengine - INFO - Iter(train) [15410/19176] lr: 1.9565e-06 eta: 1:31:34 time: 1.6766 data_time: 0.0137 memory: 11679 loss: 0.1465 2025/03/23 23:41:40 - mmengine - INFO - Iter(train) [15420/19176] lr: 1.9465e-06 eta: 1:31:20 time: 1.6396 data_time: 0.0149 memory: 11465 loss: 0.1591 2025/03/23 23:41:56 - mmengine - INFO - Iter(train) [15430/19176] lr: 1.9365e-06 eta: 1:31:06 time: 1.5723 data_time: 0.0152 memory: 11314 loss: 0.1522 2025/03/23 23:42:11 - mmengine - INFO - Iter(train) [15440/19176] lr: 1.9265e-06 eta: 1:30:51 time: 1.5079 data_time: 0.0147 memory: 11186 loss: 0.1360 2025/03/23 23:42:25 - mmengine - INFO - Iter(train) [15450/19176] lr: 1.9166e-06 eta: 1:30:37 time: 1.4386 data_time: 0.0148 memory: 11064 loss: 0.1389 2025/03/23 23:42:38 - mmengine - INFO - Iter(train) [15460/19176] lr: 1.9067e-06 eta: 1:30:22 time: 1.2559 data_time: 0.0148 memory: 10736 loss: 0.1634 2025/03/23 23:42:49 - mmengine - INFO - Iter(train) [15470/19176] lr: 1.8967e-06 eta: 1:30:06 time: 1.0913 data_time: 0.0126 memory: 10359 loss: 0.1566 2025/03/23 23:42:57 - mmengine - INFO - Iter(train) [15480/19176] lr: 1.8869e-06 eta: 1:29:50 time: 0.8827 data_time: 0.0120 memory: 9988 loss: 0.1608 2025/03/23 23:43:13 - mmengine - INFO - Iter(train) [15490/19176] lr: 1.8770e-06 eta: 1:29:36 time: 1.5913 data_time: 0.0133 memory: 12751 loss: 0.1878 2025/03/23 23:43:31 - mmengine - INFO - Iter(train) [15500/19176] lr: 1.8672e-06 eta: 1:29:22 time: 1.7750 data_time: 0.0148 memory: 12079 loss: 0.1454 2025/03/23 23:43:48 - mmengine - INFO - Iter(train) [15510/19176] lr: 1.8573e-06 eta: 1:29:08 time: 1.6847 data_time: 0.0149 memory: 11753 loss: 0.1518 2025/03/23 23:44:04 - mmengine - INFO - Iter(train) [15520/19176] lr: 1.8476e-06 eta: 1:28:54 time: 1.6367 data_time: 0.0148 memory: 11453 loss: 0.1568 2025/03/23 23:44:20 - mmengine - INFO - Iter(train) [15530/19176] lr: 1.8378e-06 eta: 1:28:40 time: 1.5453 data_time: 0.0146 memory: 11263 loss: 0.1427 2025/03/23 23:44:34 - mmengine - INFO - Iter(train) [15540/19176] lr: 1.8280e-06 eta: 1:28:25 time: 1.4790 data_time: 0.0145 memory: 11135 loss: 0.1635 2025/03/23 23:44:48 - mmengine - INFO - Iter(train) [15550/19176] lr: 1.8183e-06 eta: 1:28:10 time: 1.3919 data_time: 0.0142 memory: 10965 loss: 0.1438 2025/03/23 23:45:01 - mmengine - INFO - Iter(train) [15560/19176] lr: 1.8086e-06 eta: 1:27:55 time: 1.2403 data_time: 0.0134 memory: 10803 loss: 0.1629 2025/03/23 23:45:11 - mmengine - INFO - Iter(train) [15570/19176] lr: 1.7989e-06 eta: 1:27:40 time: 1.0257 data_time: 0.0124 memory: 10128 loss: 0.1569 2025/03/23 23:45:19 - mmengine - INFO - Iter(train) [15580/19176] lr: 1.7893e-06 eta: 1:27:24 time: 0.8398 data_time: 0.0121 memory: 9925 loss: 0.1679 2025/03/23 23:45:36 - mmengine - INFO - Iter(train) [15590/19176] lr: 1.7797e-06 eta: 1:27:09 time: 1.6427 data_time: 0.0131 memory: 13636 loss: 0.1406 2025/03/23 23:45:53 - mmengine - INFO - Iter(train) [15600/19176] lr: 1.7701e-06 eta: 1:26:55 time: 1.7361 data_time: 0.0148 memory: 11965 loss: 0.1401 2025/03/23 23:46:10 - mmengine - INFO - Iter(train) [15610/19176] lr: 1.7605e-06 eta: 1:26:41 time: 1.7044 data_time: 0.0150 memory: 11765 loss: 0.1580 2025/03/23 23:46:27 - mmengine - INFO - Iter(train) [15620/19176] lr: 1.7509e-06 eta: 1:26:27 time: 1.6568 data_time: 0.0151 memory: 11537 loss: 0.1428 2025/03/23 23:46:43 - mmengine - INFO - Iter(train) [15630/19176] lr: 1.7414e-06 eta: 1:26:13 time: 1.5961 data_time: 0.0146 memory: 11371 loss: 0.1530 2025/03/23 23:46:58 - mmengine - INFO - Iter(train) [15640/19176] lr: 1.7319e-06 eta: 1:25:59 time: 1.5517 data_time: 0.0146 memory: 11284 loss: 0.1340 2025/03/23 23:47:13 - mmengine - INFO - Iter(train) [15650/19176] lr: 1.7224e-06 eta: 1:25:44 time: 1.4914 data_time: 0.0148 memory: 11160 loss: 0.1527 2025/03/23 23:47:27 - mmengine - INFO - Iter(train) [15660/19176] lr: 1.7129e-06 eta: 1:25:29 time: 1.3931 data_time: 0.0150 memory: 10913 loss: 0.1258 2025/03/23 23:47:39 - mmengine - INFO - Iter(train) [15670/19176] lr: 1.7035e-06 eta: 1:25:14 time: 1.1838 data_time: 0.0132 memory: 10589 loss: 0.1439 2025/03/23 23:47:49 - mmengine - INFO - Iter(train) [15680/19176] lr: 1.6941e-06 eta: 1:24:59 time: 1.0118 data_time: 0.0128 memory: 10326 loss: 0.1780 2025/03/23 23:48:06 - mmengine - INFO - Iter(train) [15690/19176] lr: 1.6847e-06 eta: 1:24:45 time: 1.7182 data_time: 0.0140 memory: 14179 loss: 0.1414 2025/03/23 23:48:24 - mmengine - INFO - Iter(train) [15700/19176] lr: 1.6753e-06 eta: 1:24:31 time: 1.7249 data_time: 0.0146 memory: 11926 loss: 0.1483 2025/03/23 23:48:40 - mmengine - INFO - Iter(train) [15710/19176] lr: 1.6659e-06 eta: 1:24:17 time: 1.6822 data_time: 0.0149 memory: 11564 loss: 0.1559 2025/03/23 23:48:56 - mmengine - INFO - Iter(train) [15720/19176] lr: 1.6566e-06 eta: 1:24:02 time: 1.6021 data_time: 0.0146 memory: 11399 loss: 0.1237 2025/03/23 23:49:12 - mmengine - INFO - Iter(train) [15730/19176] lr: 1.6473e-06 eta: 1:23:48 time: 1.5168 data_time: 0.0143 memory: 11208 loss: 0.1523 2025/03/23 23:49:26 - mmengine - INFO - Iter(train) [15740/19176] lr: 1.6380e-06 eta: 1:23:33 time: 1.4464 data_time: 0.0140 memory: 11063 loss: 0.1464 2025/03/23 23:49:39 - mmengine - INFO - Iter(train) [15750/19176] lr: 1.6288e-06 eta: 1:23:18 time: 1.3338 data_time: 0.0149 memory: 10864 loss: 0.1347 2025/03/23 23:49:51 - mmengine - INFO - Iter(train) [15760/19176] lr: 1.6196e-06 eta: 1:23:03 time: 1.1582 data_time: 0.0131 memory: 10445 loss: 0.1456 2025/03/23 23:50:02 - mmengine - INFO - Iter(train) [15770/19176] lr: 1.6104e-06 eta: 1:22:48 time: 1.0890 data_time: 0.0131 memory: 10391 loss: 0.1557 2025/03/23 23:50:11 - mmengine - INFO - Iter(train) [15780/19176] lr: 1.6012e-06 eta: 1:22:32 time: 0.9209 data_time: 0.0124 memory: 10021 loss: 0.1515 2025/03/23 23:50:28 - mmengine - INFO - Iter(train) [15790/19176] lr: 1.5920e-06 eta: 1:22:18 time: 1.7079 data_time: 0.0137 memory: 14090 loss: 0.1601 2025/03/23 23:50:45 - mmengine - INFO - Iter(train) [15800/19176] lr: 1.5829e-06 eta: 1:22:04 time: 1.7201 data_time: 0.0147 memory: 11854 loss: 0.1299 2025/03/23 23:51:02 - mmengine - INFO - Iter(train) [15810/19176] lr: 1.5738e-06 eta: 1:21:50 time: 1.6502 data_time: 0.0150 memory: 11506 loss: 0.1443 2025/03/23 23:51:18 - mmengine - INFO - Iter(train) [15820/19176] lr: 1.5647e-06 eta: 1:21:35 time: 1.5964 data_time: 0.0145 memory: 11384 loss: 0.1463 2025/03/23 23:51:34 - mmengine - INFO - Iter(train) [15830/19176] lr: 1.5557e-06 eta: 1:21:21 time: 1.5748 data_time: 0.0146 memory: 11321 loss: 0.1515 2025/03/23 23:51:49 - mmengine - INFO - Iter(train) [15840/19176] lr: 1.5466e-06 eta: 1:21:07 time: 1.5134 data_time: 0.0142 memory: 11188 loss: 0.1429 2025/03/23 23:52:03 - mmengine - INFO - Iter(train) [15850/19176] lr: 1.5376e-06 eta: 1:20:52 time: 1.4600 data_time: 0.0142 memory: 11192 loss: 0.1336 2025/03/23 23:52:17 - mmengine - INFO - Iter(train) [15860/19176] lr: 1.5286e-06 eta: 1:20:37 time: 1.3339 data_time: 0.0143 memory: 10886 loss: 0.1309 2025/03/23 23:52:27 - mmengine - INFO - Iter(train) [15870/19176] lr: 1.5197e-06 eta: 1:20:22 time: 1.0340 data_time: 0.0126 memory: 10240 loss: 0.1313 2025/03/23 23:52:35 - mmengine - INFO - Iter(train) [15880/19176] lr: 1.5107e-06 eta: 1:20:06 time: 0.7731 data_time: 0.0113 memory: 9505 loss: 0.1490 2025/03/23 23:52:54 - mmengine - INFO - Iter(train) [15890/19176] lr: 1.5018e-06 eta: 1:19:52 time: 1.9276 data_time: 0.0136 memory: 18041 loss: 0.2330 2025/03/23 23:53:12 - mmengine - INFO - Iter(train) [15900/19176] lr: 1.4929e-06 eta: 1:19:38 time: 1.7594 data_time: 0.0145 memory: 12105 loss: 0.1336 2025/03/23 23:53:28 - mmengine - INFO - Iter(train) [15910/19176] lr: 1.4841e-06 eta: 1:19:24 time: 1.6928 data_time: 0.0146 memory: 12229 loss: 0.1267 2025/03/23 23:53:45 - mmengine - INFO - Iter(train) [15920/19176] lr: 1.4752e-06 eta: 1:19:10 time: 1.6354 data_time: 0.0145 memory: 11466 loss: 0.1280 2025/03/23 23:54:01 - mmengine - INFO - Iter(train) [15930/19176] lr: 1.4664e-06 eta: 1:18:55 time: 1.5932 data_time: 0.0144 memory: 11361 loss: 0.1458 2025/03/23 23:54:16 - mmengine - INFO - Iter(train) [15940/19176] lr: 1.4576e-06 eta: 1:18:41 time: 1.5244 data_time: 0.0139 memory: 11303 loss: 0.1341 2025/03/23 23:54:31 - mmengine - INFO - Iter(train) [15950/19176] lr: 1.4488e-06 eta: 1:18:26 time: 1.4718 data_time: 0.0145 memory: 11095 loss: 0.1243 2025/03/23 23:54:44 - mmengine - INFO - Iter(train) [15960/19176] lr: 1.4401e-06 eta: 1:18:12 time: 1.3425 data_time: 0.0141 memory: 10881 loss: 0.1529 2025/03/23 23:54:55 - mmengine - INFO - Iter(train) [15970/19176] lr: 1.4314e-06 eta: 1:17:56 time: 1.1215 data_time: 0.0130 memory: 10482 loss: 0.1207 2025/03/23 23:55:05 - mmengine - INFO - Iter(train) [15980/19176] lr: 1.4227e-06 eta: 1:17:41 time: 0.9312 data_time: 0.0125 memory: 10013 loss: 0.1518 2025/03/23 23:55:21 - mmengine - INFO - Iter(train) [15990/19176] lr: 1.4140e-06 eta: 1:17:26 time: 1.6511 data_time: 0.0136 memory: 12923 loss: 0.1309 2025/03/23 23:55:39 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/23 23:55:39 - mmengine - INFO - Iter(train) [16000/19176] lr: 1.4054e-06 eta: 1:17:12 time: 1.7617 data_time: 0.0140 memory: 11986 loss: 0.1395 2025/03/23 23:55:39 - mmengine - INFO - Saving checkpoint at 16000 iterations 2025/03/23 23:55:56 - mmengine - INFO - Iter(train) [16010/19176] lr: 1.3967e-06 eta: 1:16:58 time: 1.7628 data_time: 0.0926 memory: 11760 loss: 0.1406 2025/03/23 23:56:13 - mmengine - INFO - Iter(train) [16020/19176] lr: 1.3881e-06 eta: 1:16:44 time: 1.6312 data_time: 0.0140 memory: 11458 loss: 0.1375 2025/03/23 23:56:29 - mmengine - INFO - Iter(train) [16030/19176] lr: 1.3796e-06 eta: 1:16:30 time: 1.5764 data_time: 0.0142 memory: 11347 loss: 0.1483 2025/03/23 23:56:44 - mmengine - INFO - Iter(train) [16040/19176] lr: 1.3710e-06 eta: 1:16:15 time: 1.5155 data_time: 0.0140 memory: 11198 loss: 0.1304 2025/03/23 23:56:58 - mmengine - INFO - Iter(train) [16050/19176] lr: 1.3625e-06 eta: 1:16:01 time: 1.4562 data_time: 0.0140 memory: 11088 loss: 0.1534 2025/03/23 23:57:12 - mmengine - INFO - Iter(train) [16060/19176] lr: 1.3540e-06 eta: 1:15:46 time: 1.3699 data_time: 0.0149 memory: 10949 loss: 0.1325 2025/03/23 23:57:23 - mmengine - INFO - Iter(train) [16070/19176] lr: 1.3455e-06 eta: 1:15:31 time: 1.0774 data_time: 0.0121 memory: 10360 loss: 0.1553 2025/03/23 23:57:30 - mmengine - INFO - Iter(train) [16080/19176] lr: 1.3371e-06 eta: 1:15:15 time: 0.7424 data_time: 0.0116 memory: 9531 loss: 0.1347 2025/03/23 23:57:47 - mmengine - INFO - Iter(train) [16090/19176] lr: 1.3286e-06 eta: 1:15:01 time: 1.7262 data_time: 0.0145 memory: 14276 loss: 0.2110 2025/03/23 23:58:04 - mmengine - INFO - Iter(train) [16100/19176] lr: 1.3202e-06 eta: 1:14:47 time: 1.7054 data_time: 0.0149 memory: 11719 loss: 0.1311 2025/03/23 23:58:21 - mmengine - INFO - Iter(train) [16110/19176] lr: 1.3119e-06 eta: 1:14:32 time: 1.6529 data_time: 0.0145 memory: 11485 loss: 0.1367 2025/03/23 23:58:37 - mmengine - INFO - Iter(train) [16120/19176] lr: 1.3035e-06 eta: 1:14:18 time: 1.5770 data_time: 0.0143 memory: 11337 loss: 0.1484 2025/03/23 23:58:52 - mmengine - INFO - Iter(train) [16130/19176] lr: 1.2952e-06 eta: 1:14:04 time: 1.5275 data_time: 0.0146 memory: 11244 loss: 0.1349 2025/03/23 23:59:07 - mmengine - INFO - Iter(train) [16140/19176] lr: 1.2869e-06 eta: 1:13:49 time: 1.4727 data_time: 0.0142 memory: 11105 loss: 0.1380 2025/03/23 23:59:21 - mmengine - INFO - Iter(train) [16150/19176] lr: 1.2786e-06 eta: 1:13:34 time: 1.3953 data_time: 0.0136 memory: 10942 loss: 0.1362 2025/03/23 23:59:33 - mmengine - INFO - Iter(train) [16160/19176] lr: 1.2704e-06 eta: 1:13:19 time: 1.2703 data_time: 0.0130 memory: 10754 loss: 0.1571 2025/03/23 23:59:44 - mmengine - INFO - Iter(train) [16170/19176] lr: 1.2621e-06 eta: 1:13:04 time: 1.1040 data_time: 0.0122 memory: 10367 loss: 0.1472 2025/03/23 23:59:53 - mmengine - INFO - Iter(train) [16180/19176] lr: 1.2539e-06 eta: 1:12:48 time: 0.8838 data_time: 0.0130 memory: 9965 loss: 0.1550 2025/03/24 00:00:11 - mmengine - INFO - Iter(train) [16190/19176] lr: 1.2458e-06 eta: 1:12:34 time: 1.7724 data_time: 0.0130 memory: 15756 loss: 0.1499 2025/03/24 00:00:29 - mmengine - INFO - Iter(train) [16200/19176] lr: 1.2376e-06 eta: 1:12:20 time: 1.7638 data_time: 0.0148 memory: 12056 loss: 0.1482 2025/03/24 00:00:46 - mmengine - INFO - Iter(train) [16210/19176] lr: 1.2295e-06 eta: 1:12:06 time: 1.7021 data_time: 0.0141 memory: 11845 loss: 0.1426 2025/03/24 00:01:02 - mmengine - INFO - Iter(train) [16220/19176] lr: 1.2214e-06 eta: 1:11:52 time: 1.6282 data_time: 0.0140 memory: 11564 loss: 0.1436 2025/03/24 00:01:18 - mmengine - INFO - Iter(train) [16230/19176] lr: 1.2133e-06 eta: 1:11:38 time: 1.5611 data_time: 0.0155 memory: 11325 loss: 0.1306 2025/03/24 00:01:33 - mmengine - INFO - Iter(train) [16240/19176] lr: 1.2053e-06 eta: 1:11:23 time: 1.5011 data_time: 0.0149 memory: 11183 loss: 0.1549 2025/03/24 00:01:47 - mmengine - INFO - Iter(train) [16250/19176] lr: 1.1972e-06 eta: 1:11:08 time: 1.4469 data_time: 0.0145 memory: 11024 loss: 0.1401 2025/03/24 00:02:00 - mmengine - INFO - Iter(train) [16260/19176] lr: 1.1892e-06 eta: 1:10:53 time: 1.2647 data_time: 0.0145 memory: 10746 loss: 0.1321 2025/03/24 00:02:10 - mmengine - INFO - Iter(train) [16270/19176] lr: 1.1813e-06 eta: 1:10:38 time: 1.0677 data_time: 0.0128 memory: 10269 loss: 0.1568 2025/03/24 00:02:19 - mmengine - INFO - Iter(train) [16280/19176] lr: 1.1733e-06 eta: 1:10:22 time: 0.8264 data_time: 0.0120 memory: 9881 loss: 0.1369 2025/03/24 00:02:34 - mmengine - INFO - Iter(train) [16290/19176] lr: 1.1654e-06 eta: 1:10:08 time: 1.5225 data_time: 0.0131 memory: 12923 loss: 0.1364 2025/03/24 00:02:51 - mmengine - INFO - Iter(train) [16300/19176] lr: 1.1575e-06 eta: 1:09:54 time: 1.7492 data_time: 0.0146 memory: 11909 loss: 0.1422 2025/03/24 00:03:08 - mmengine - INFO - Iter(train) [16310/19176] lr: 1.1496e-06 eta: 1:09:40 time: 1.6714 data_time: 0.0143 memory: 11686 loss: 0.1405 2025/03/24 00:03:24 - mmengine - INFO - Iter(train) [16320/19176] lr: 1.1418e-06 eta: 1:09:25 time: 1.6258 data_time: 0.0148 memory: 11501 loss: 0.1664 2025/03/24 00:03:40 - mmengine - INFO - Iter(train) [16330/19176] lr: 1.1339e-06 eta: 1:09:11 time: 1.6035 data_time: 0.0159 memory: 11369 loss: 0.1458 2025/03/24 00:03:56 - mmengine - INFO - Iter(train) [16340/19176] lr: 1.1261e-06 eta: 1:08:57 time: 1.5553 data_time: 0.0151 memory: 11282 loss: 0.1814 2025/03/24 00:04:11 - mmengine - INFO - Iter(train) [16350/19176] lr: 1.1184e-06 eta: 1:08:42 time: 1.4987 data_time: 0.0147 memory: 11163 loss: 0.1514 2025/03/24 00:04:25 - mmengine - INFO - Iter(train) [16360/19176] lr: 1.1106e-06 eta: 1:08:28 time: 1.4280 data_time: 0.0147 memory: 11022 loss: 0.1622 2025/03/24 00:04:39 - mmengine - INFO - Iter(train) [16370/19176] lr: 1.1029e-06 eta: 1:08:13 time: 1.3388 data_time: 0.0147 memory: 10860 loss: 0.2168 2025/03/24 00:04:50 - mmengine - INFO - Iter(train) [16380/19176] lr: 1.0952e-06 eta: 1:07:58 time: 1.1256 data_time: 0.0130 memory: 10458 loss: 0.1247 2025/03/24 00:05:09 - mmengine - INFO - Iter(train) [16390/19176] lr: 1.0875e-06 eta: 1:07:44 time: 1.9415 data_time: 0.0140 memory: 18970 loss: 0.1597 2025/03/24 00:05:27 - mmengine - INFO - Iter(train) [16400/19176] lr: 1.0799e-06 eta: 1:07:30 time: 1.7340 data_time: 0.0149 memory: 11900 loss: 0.1415 2025/03/24 00:05:43 - mmengine - INFO - Iter(train) [16410/19176] lr: 1.0723e-06 eta: 1:07:15 time: 1.6811 data_time: 0.0143 memory: 11659 loss: 0.1409 2025/03/24 00:06:00 - mmengine - INFO - Iter(train) [16420/19176] lr: 1.0647e-06 eta: 1:07:01 time: 1.6146 data_time: 0.0147 memory: 11501 loss: 0.1568 2025/03/24 00:06:15 - mmengine - INFO - Iter(train) [16430/19176] lr: 1.0571e-06 eta: 1:06:47 time: 1.5742 data_time: 0.0147 memory: 11307 loss: 0.1627 2025/03/24 00:06:30 - mmengine - INFO - Iter(train) [16440/19176] lr: 1.0495e-06 eta: 1:06:32 time: 1.5013 data_time: 0.0144 memory: 11281 loss: 0.1376 2025/03/24 00:06:45 - mmengine - INFO - Iter(train) [16450/19176] lr: 1.0420e-06 eta: 1:06:18 time: 1.4315 data_time: 0.0139 memory: 11028 loss: 0.1719 2025/03/24 00:06:57 - mmengine - INFO - Iter(train) [16460/19176] lr: 1.0345e-06 eta: 1:06:03 time: 1.2720 data_time: 0.0136 memory: 10828 loss: 0.1534 2025/03/24 00:07:08 - mmengine - INFO - Iter(train) [16470/19176] lr: 1.0271e-06 eta: 1:05:47 time: 1.0749 data_time: 0.0128 memory: 10259 loss: 0.1549 2025/03/24 00:07:18 - mmengine - INFO - Iter(train) [16480/19176] lr: 1.0196e-06 eta: 1:05:32 time: 0.9547 data_time: 0.0123 memory: 10047 loss: 0.1413 2025/03/24 00:07:36 - mmengine - INFO - Iter(train) [16490/19176] lr: 1.0122e-06 eta: 1:05:18 time: 1.8348 data_time: 0.0136 memory: 15904 loss: 0.1617 2025/03/24 00:07:54 - mmengine - INFO - Iter(train) [16500/19176] lr: 1.0048e-06 eta: 1:05:04 time: 1.7801 data_time: 0.0149 memory: 12246 loss: 0.1215 2025/03/24 00:08:11 - mmengine - INFO - Iter(train) [16510/19176] lr: 9.9744e-07 eta: 1:04:50 time: 1.6928 data_time: 0.0149 memory: 11787 loss: 0.1399 2025/03/24 00:08:27 - mmengine - INFO - Iter(train) [16520/19176] lr: 9.9010e-07 eta: 1:04:35 time: 1.6443 data_time: 0.0147 memory: 11548 loss: 0.1398 2025/03/24 00:08:43 - mmengine - INFO - Iter(train) [16530/19176] lr: 9.8279e-07 eta: 1:04:21 time: 1.5856 data_time: 0.0145 memory: 11532 loss: 0.1487 2025/03/24 00:08:58 - mmengine - INFO - Iter(train) [16540/19176] lr: 9.7550e-07 eta: 1:04:07 time: 1.4961 data_time: 0.0148 memory: 11233 loss: 0.1548 2025/03/24 00:09:12 - mmengine - INFO - Iter(train) [16550/19176] lr: 9.6824e-07 eta: 1:03:52 time: 1.4395 data_time: 0.0145 memory: 10991 loss: 0.1193 2025/03/24 00:09:25 - mmengine - INFO - Iter(train) [16560/19176] lr: 9.6100e-07 eta: 1:03:37 time: 1.2953 data_time: 0.0143 memory: 10817 loss: 0.2460 2025/03/24 00:09:36 - mmengine - INFO - Iter(train) [16570/19176] lr: 9.5379e-07 eta: 1:03:22 time: 1.0713 data_time: 0.0121 memory: 10381 loss: 0.1343 2025/03/24 00:09:44 - mmengine - INFO - Iter(train) [16580/19176] lr: 9.4660e-07 eta: 1:03:06 time: 0.8345 data_time: 0.0120 memory: 9937 loss: 0.1722 2025/03/24 00:10:02 - mmengine - INFO - Iter(train) [16590/19176] lr: 9.3944e-07 eta: 1:02:52 time: 1.7568 data_time: 0.0129 memory: 16557 loss: 0.1528 2025/03/24 00:10:20 - mmengine - INFO - Iter(train) [16600/19176] lr: 9.3231e-07 eta: 1:02:38 time: 1.7977 data_time: 0.0148 memory: 12300 loss: 0.1400 2025/03/24 00:10:37 - mmengine - INFO - Iter(train) [16610/19176] lr: 9.2520e-07 eta: 1:02:24 time: 1.7235 data_time: 0.0144 memory: 11863 loss: 0.1296 2025/03/24 00:10:54 - mmengine - INFO - Iter(train) [16620/19176] lr: 9.1812e-07 eta: 1:02:10 time: 1.6467 data_time: 0.0150 memory: 11737 loss: 0.1492 2025/03/24 00:11:09 - mmengine - INFO - Iter(train) [16630/19176] lr: 9.1106e-07 eta: 1:01:55 time: 1.5856 data_time: 0.0154 memory: 11301 loss: 0.1270 2025/03/24 00:11:25 - mmengine - INFO - Iter(train) [16640/19176] lr: 9.0403e-07 eta: 1:01:41 time: 1.5356 data_time: 0.0144 memory: 11251 loss: 0.1514 2025/03/24 00:11:40 - mmengine - INFO - Iter(train) [16650/19176] lr: 8.9703e-07 eta: 1:01:26 time: 1.4674 data_time: 0.0148 memory: 11130 loss: 0.1458 2025/03/24 00:11:53 - mmengine - INFO - Iter(train) [16660/19176] lr: 8.9005e-07 eta: 1:01:11 time: 1.3472 data_time: 0.0142 memory: 10914 loss: 0.1647 2025/03/24 00:12:04 - mmengine - INFO - Iter(train) [16670/19176] lr: 8.8310e-07 eta: 1:00:56 time: 1.0891 data_time: 0.0126 memory: 10296 loss: 0.1627 2025/03/24 00:12:13 - mmengine - INFO - Iter(train) [16680/19176] lr: 8.7617e-07 eta: 1:00:41 time: 0.8741 data_time: 0.0119 memory: 10005 loss: 0.1506 2025/03/24 00:12:29 - mmengine - INFO - Iter(train) [16690/19176] lr: 8.6927e-07 eta: 1:00:26 time: 1.6244 data_time: 0.0130 memory: 13516 loss: 0.1393 2025/03/24 00:12:46 - mmengine - INFO - Iter(train) [16700/19176] lr: 8.6239e-07 eta: 1:00:12 time: 1.7188 data_time: 0.0141 memory: 11865 loss: 0.1647 2025/03/24 00:13:03 - mmengine - INFO - Iter(train) [16710/19176] lr: 8.5555e-07 eta: 0:59:58 time: 1.6517 data_time: 0.0144 memory: 11501 loss: 0.1391 2025/03/24 00:13:19 - mmengine - INFO - Iter(train) [16720/19176] lr: 8.4872e-07 eta: 0:59:44 time: 1.6023 data_time: 0.0143 memory: 11533 loss: 0.1435 2025/03/24 00:13:34 - mmengine - INFO - Iter(train) [16730/19176] lr: 8.4193e-07 eta: 0:59:29 time: 1.5392 data_time: 0.0142 memory: 11277 loss: 0.1608 2025/03/24 00:13:49 - mmengine - INFO - Iter(train) [16740/19176] lr: 8.3516e-07 eta: 0:59:15 time: 1.4749 data_time: 0.0144 memory: 11230 loss: 0.1461 2025/03/24 00:14:02 - mmengine - INFO - Iter(train) [16750/19176] lr: 8.2841e-07 eta: 0:59:00 time: 1.3354 data_time: 0.0142 memory: 10862 loss: 0.1395 2025/03/24 00:14:14 - mmengine - INFO - Iter(train) [16760/19176] lr: 8.2170e-07 eta: 0:58:45 time: 1.1772 data_time: 0.0134 memory: 10584 loss: 0.1445 2025/03/24 00:14:23 - mmengine - INFO - Iter(train) [16770/19176] lr: 8.1500e-07 eta: 0:58:29 time: 0.9026 data_time: 0.0122 memory: 9982 loss: 0.1379 2025/03/24 00:14:30 - mmengine - INFO - Iter(train) [16780/19176] lr: 8.0834e-07 eta: 0:58:14 time: 0.6623 data_time: 0.0111 memory: 9389 loss: 0.1570 2025/03/24 00:14:47 - mmengine - INFO - Iter(train) [16790/19176] lr: 8.0170e-07 eta: 0:57:59 time: 1.7070 data_time: 0.0130 memory: 14354 loss: 0.1539 2025/03/24 00:15:05 - mmengine - INFO - Iter(train) [16800/19176] lr: 7.9509e-07 eta: 0:57:45 time: 1.8216 data_time: 0.0145 memory: 12326 loss: 0.1627 2025/03/24 00:15:22 - mmengine - INFO - Iter(train) [16810/19176] lr: 7.8850e-07 eta: 0:57:31 time: 1.7338 data_time: 0.0145 memory: 11803 loss: 0.1312 2025/03/24 00:15:39 - mmengine - INFO - Iter(train) [16820/19176] lr: 7.8194e-07 eta: 0:57:17 time: 1.6813 data_time: 0.0144 memory: 11541 loss: 0.1320 2025/03/24 00:15:55 - mmengine - INFO - Iter(train) [16830/19176] lr: 7.7541e-07 eta: 0:57:03 time: 1.6445 data_time: 0.0146 memory: 11472 loss: 0.1387 2025/03/24 00:16:11 - mmengine - INFO - Iter(train) [16840/19176] lr: 7.6890e-07 eta: 0:56:48 time: 1.5588 data_time: 0.0147 memory: 11301 loss: 0.1422 2025/03/24 00:16:26 - mmengine - INFO - Iter(train) [16850/19176] lr: 7.6242e-07 eta: 0:56:34 time: 1.4952 data_time: 0.0149 memory: 11375 loss: 0.1393 2025/03/24 00:16:39 - mmengine - INFO - Iter(train) [16860/19176] lr: 7.5596e-07 eta: 0:56:19 time: 1.3282 data_time: 0.0142 memory: 10943 loss: 0.1371 2025/03/24 00:16:50 - mmengine - INFO - Iter(train) [16870/19176] lr: 7.4953e-07 eta: 0:56:04 time: 1.0654 data_time: 0.0128 memory: 10248 loss: 0.1510 2025/03/24 00:16:57 - mmengine - INFO - Iter(train) [16880/19176] lr: 7.4313e-07 eta: 0:55:48 time: 0.7575 data_time: 0.0115 memory: 9855 loss: 0.1401 2025/03/24 00:17:14 - mmengine - INFO - Iter(train) [16890/19176] lr: 7.3676e-07 eta: 0:55:34 time: 1.6058 data_time: 0.0134 memory: 13001 loss: 0.1285 2025/03/24 00:17:31 - mmengine - INFO - Iter(train) [16900/19176] lr: 7.3041e-07 eta: 0:55:20 time: 1.7477 data_time: 0.0152 memory: 12065 loss: 0.1343 2025/03/24 00:17:48 - mmengine - INFO - Iter(train) [16910/19176] lr: 7.2408e-07 eta: 0:55:05 time: 1.6829 data_time: 0.0150 memory: 11642 loss: 0.1664 2025/03/24 00:18:04 - mmengine - INFO - Iter(train) [16920/19176] lr: 7.1779e-07 eta: 0:54:51 time: 1.5928 data_time: 0.0144 memory: 11448 loss: 0.1602 2025/03/24 00:18:19 - mmengine - INFO - Iter(train) [16930/19176] lr: 7.1152e-07 eta: 0:54:36 time: 1.5552 data_time: 0.0149 memory: 11325 loss: 0.1597 2025/03/24 00:18:34 - mmengine - INFO - Iter(train) [16940/19176] lr: 7.0527e-07 eta: 0:54:22 time: 1.4976 data_time: 0.0149 memory: 11161 loss: 0.1632 2025/03/24 00:18:49 - mmengine - INFO - Iter(train) [16950/19176] lr: 6.9906e-07 eta: 0:54:07 time: 1.4251 data_time: 0.0151 memory: 11067 loss: 0.1413 2025/03/24 00:19:01 - mmengine - INFO - Iter(train) [16960/19176] lr: 6.9286e-07 eta: 0:53:52 time: 1.2853 data_time: 0.0146 memory: 10855 loss: 0.1447 2025/03/24 00:19:12 - mmengine - INFO - Iter(train) [16970/19176] lr: 6.8670e-07 eta: 0:53:37 time: 1.0558 data_time: 0.0138 memory: 10247 loss: 0.1441 2025/03/24 00:19:21 - mmengine - INFO - Iter(train) [16980/19176] lr: 6.8056e-07 eta: 0:53:22 time: 0.9233 data_time: 0.0133 memory: 10025 loss: 0.1733 2025/03/24 00:19:39 - mmengine - INFO - Iter(train) [16990/19176] lr: 6.7445e-07 eta: 0:53:08 time: 1.8160 data_time: 0.0148 memory: 16526 loss: 0.1671 2025/03/24 00:19:57 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/24 00:19:57 - mmengine - INFO - Iter(train) [17000/19176] lr: 6.6837e-07 eta: 0:52:54 time: 1.8049 data_time: 0.0159 memory: 12291 loss: 0.1539 2025/03/24 00:19:57 - mmengine - INFO - Saving checkpoint at 17000 iterations 2025/03/24 00:20:15 - mmengine - INFO - Iter(train) [17010/19176] lr: 6.6231e-07 eta: 0:52:40 time: 1.7802 data_time: 0.0953 memory: 11774 loss: 0.1584 2025/03/24 00:20:32 - mmengine - INFO - Iter(train) [17020/19176] lr: 6.5628e-07 eta: 0:52:25 time: 1.6458 data_time: 0.0150 memory: 11458 loss: 0.1480 2025/03/24 00:20:48 - mmengine - INFO - Iter(train) [17030/19176] lr: 6.5028e-07 eta: 0:52:11 time: 1.5880 data_time: 0.0147 memory: 11330 loss: 0.1528 2025/03/24 00:21:03 - mmengine - INFO - Iter(train) [17040/19176] lr: 6.4430e-07 eta: 0:51:56 time: 1.5314 data_time: 0.0149 memory: 11218 loss: 0.1415 2025/03/24 00:21:17 - mmengine - INFO - Iter(train) [17050/19176] lr: 6.3835e-07 eta: 0:51:42 time: 1.4366 data_time: 0.0149 memory: 11006 loss: 0.1635 2025/03/24 00:21:30 - mmengine - INFO - Iter(train) [17060/19176] lr: 6.3242e-07 eta: 0:51:27 time: 1.2961 data_time: 0.0146 memory: 10834 loss: 0.1273 2025/03/24 00:21:41 - mmengine - INFO - Iter(train) [17070/19176] lr: 6.2652e-07 eta: 0:51:12 time: 1.1270 data_time: 0.0129 memory: 10364 loss: 0.1427 2025/03/24 00:21:50 - mmengine - INFO - Iter(train) [17080/19176] lr: 6.2065e-07 eta: 0:50:57 time: 0.8894 data_time: 0.0126 memory: 9993 loss: 0.1336 2025/03/24 00:22:09 - mmengine - INFO - Iter(train) [17090/19176] lr: 6.1481e-07 eta: 0:50:43 time: 1.9105 data_time: 0.0135 memory: 16441 loss: 0.1904 2025/03/24 00:22:28 - mmengine - INFO - Iter(train) [17100/19176] lr: 6.0899e-07 eta: 0:50:28 time: 1.8255 data_time: 0.0160 memory: 12411 loss: 0.1345 2025/03/24 00:22:45 - mmengine - INFO - Iter(train) [17110/19176] lr: 6.0320e-07 eta: 0:50:14 time: 1.7153 data_time: 0.0150 memory: 11921 loss: 0.1296 2025/03/24 00:23:02 - mmengine - INFO - Iter(train) [17120/19176] lr: 5.9744e-07 eta: 0:50:00 time: 1.6682 data_time: 0.0153 memory: 11645 loss: 0.1608 2025/03/24 00:23:17 - mmengine - INFO - Iter(train) [17130/19176] lr: 5.9170e-07 eta: 0:49:45 time: 1.5959 data_time: 0.0156 memory: 11372 loss: 0.1581 2025/03/24 00:23:33 - mmengine - INFO - Iter(train) [17140/19176] lr: 5.8599e-07 eta: 0:49:31 time: 1.5446 data_time: 0.0160 memory: 11281 loss: 0.1511 2025/03/24 00:23:48 - mmengine - INFO - Iter(train) [17150/19176] lr: 5.8031e-07 eta: 0:49:16 time: 1.4620 data_time: 0.0148 memory: 11152 loss: 0.1401 2025/03/24 00:24:00 - mmengine - INFO - Iter(train) [17160/19176] lr: 5.7465e-07 eta: 0:49:01 time: 1.2759 data_time: 0.0145 memory: 10738 loss: 0.1399 2025/03/24 00:24:11 - mmengine - INFO - Iter(train) [17170/19176] lr: 5.6902e-07 eta: 0:48:46 time: 1.0565 data_time: 0.0135 memory: 10334 loss: 0.1413 2025/03/24 00:24:19 - mmengine - INFO - Iter(train) [17180/19176] lr: 5.6342e-07 eta: 0:48:31 time: 0.8619 data_time: 0.0124 memory: 9889 loss: 0.1495 2025/03/24 00:24:39 - mmengine - INFO - Iter(train) [17190/19176] lr: 5.5784e-07 eta: 0:48:17 time: 1.9436 data_time: 0.0144 memory: 18234 loss: 0.1762 2025/03/24 00:24:57 - mmengine - INFO - Iter(train) [17200/19176] lr: 5.5230e-07 eta: 0:48:03 time: 1.7844 data_time: 0.0162 memory: 12176 loss: 0.1377 2025/03/24 00:25:13 - mmengine - INFO - Iter(train) [17210/19176] lr: 5.4677e-07 eta: 0:47:48 time: 1.6706 data_time: 0.0155 memory: 11593 loss: 0.1440 2025/03/24 00:25:30 - mmengine - INFO - Iter(train) [17220/19176] lr: 5.4128e-07 eta: 0:47:34 time: 1.6281 data_time: 0.0156 memory: 11657 loss: 0.1505 2025/03/24 00:25:46 - mmengine - INFO - Iter(train) [17230/19176] lr: 5.3581e-07 eta: 0:47:20 time: 1.5881 data_time: 0.0156 memory: 11363 loss: 0.1394 2025/03/24 00:26:01 - mmengine - INFO - Iter(train) [17240/19176] lr: 5.3037e-07 eta: 0:47:05 time: 1.5348 data_time: 0.0147 memory: 11237 loss: 0.1231 2025/03/24 00:26:16 - mmengine - INFO - Iter(train) [17250/19176] lr: 5.2496e-07 eta: 0:46:51 time: 1.4521 data_time: 0.0138 memory: 11066 loss: 0.1668 2025/03/24 00:26:29 - mmengine - INFO - Iter(train) [17260/19176] lr: 5.1957e-07 eta: 0:46:36 time: 1.3813 data_time: 0.0141 memory: 10919 loss: 0.1464 2025/03/24 00:26:41 - mmengine - INFO - Iter(train) [17270/19176] lr: 5.1421e-07 eta: 0:46:21 time: 1.1387 data_time: 0.0131 memory: 10565 loss: 0.1445 2025/03/24 00:26:50 - mmengine - INFO - Iter(train) [17280/19176] lr: 5.0888e-07 eta: 0:46:06 time: 0.9605 data_time: 0.0123 memory: 10111 loss: 0.1680 2025/03/24 00:27:06 - mmengine - INFO - Iter(train) [17290/19176] lr: 5.0357e-07 eta: 0:45:51 time: 1.6150 data_time: 0.0140 memory: 12967 loss: 0.1584 2025/03/24 00:27:24 - mmengine - INFO - Iter(train) [17300/19176] lr: 4.9829e-07 eta: 0:45:37 time: 1.7420 data_time: 0.0146 memory: 12105 loss: 0.1253 2025/03/24 00:27:41 - mmengine - INFO - Iter(train) [17310/19176] lr: 4.9304e-07 eta: 0:45:23 time: 1.6650 data_time: 0.0157 memory: 11624 loss: 0.1499 2025/03/24 00:27:57 - mmengine - INFO - Iter(train) [17320/19176] lr: 4.8782e-07 eta: 0:45:08 time: 1.6215 data_time: 0.0147 memory: 11435 loss: 0.1469 2025/03/24 00:28:12 - mmengine - INFO - Iter(train) [17330/19176] lr: 4.8262e-07 eta: 0:44:54 time: 1.5542 data_time: 0.0146 memory: 11257 loss: 0.1476 2025/03/24 00:28:28 - mmengine - INFO - Iter(train) [17340/19176] lr: 4.7745e-07 eta: 0:44:39 time: 1.5278 data_time: 0.0146 memory: 11250 loss: 0.1483 2025/03/24 00:28:42 - mmengine - INFO - Iter(train) [17350/19176] lr: 4.7231e-07 eta: 0:44:25 time: 1.4306 data_time: 0.0144 memory: 11043 loss: 0.1468 2025/03/24 00:28:54 - mmengine - INFO - Iter(train) [17360/19176] lr: 4.6719e-07 eta: 0:44:10 time: 1.1752 data_time: 0.0132 memory: 10656 loss: 0.1232 2025/03/24 00:29:04 - mmengine - INFO - Iter(train) [17370/19176] lr: 4.6210e-07 eta: 0:43:55 time: 1.0353 data_time: 0.0122 memory: 10193 loss: 0.1398 2025/03/24 00:29:13 - mmengine - INFO - Iter(train) [17380/19176] lr: 4.5704e-07 eta: 0:43:40 time: 0.8967 data_time: 0.0119 memory: 9883 loss: 0.1401 2025/03/24 00:29:32 - mmengine - INFO - Iter(train) [17390/19176] lr: 4.5201e-07 eta: 0:43:25 time: 1.8823 data_time: 0.0131 memory: 16526 loss: 0.1474 2025/03/24 00:29:49 - mmengine - INFO - Iter(train) [17400/19176] lr: 4.4700e-07 eta: 0:43:11 time: 1.7505 data_time: 0.0149 memory: 12124 loss: 0.1308 2025/03/24 00:30:06 - mmengine - INFO - Iter(train) [17410/19176] lr: 4.4202e-07 eta: 0:42:57 time: 1.6685 data_time: 0.0146 memory: 11621 loss: 0.1531 2025/03/24 00:30:22 - mmengine - INFO - Iter(train) [17420/19176] lr: 4.3707e-07 eta: 0:42:42 time: 1.6010 data_time: 0.0147 memory: 11445 loss: 0.1457 2025/03/24 00:30:37 - mmengine - INFO - Iter(train) [17430/19176] lr: 4.3215e-07 eta: 0:42:28 time: 1.5400 data_time: 0.0149 memory: 11281 loss: 0.1497 2025/03/24 00:30:52 - mmengine - INFO - Iter(train) [17440/19176] lr: 4.2725e-07 eta: 0:42:13 time: 1.5061 data_time: 0.0144 memory: 11191 loss: 0.1645 2025/03/24 00:31:07 - mmengine - INFO - Iter(train) [17450/19176] lr: 4.2238e-07 eta: 0:41:59 time: 1.4437 data_time: 0.0161 memory: 11034 loss: 0.1366 2025/03/24 00:31:20 - mmengine - INFO - Iter(train) [17460/19176] lr: 4.1753e-07 eta: 0:41:44 time: 1.3069 data_time: 0.0143 memory: 10805 loss: 0.1363 2025/03/24 00:31:31 - mmengine - INFO - Iter(train) [17470/19176] lr: 4.1272e-07 eta: 0:41:29 time: 1.1168 data_time: 0.0130 memory: 10291 loss: 0.1416 2025/03/24 00:31:41 - mmengine - INFO - Iter(train) [17480/19176] lr: 4.0793e-07 eta: 0:41:14 time: 0.9480 data_time: 0.0124 memory: 10174 loss: 0.1287 2025/03/24 00:31:58 - mmengine - INFO - Iter(train) [17490/19176] lr: 4.0317e-07 eta: 0:40:59 time: 1.7096 data_time: 0.0131 memory: 14533 loss: 0.1454 2025/03/24 00:32:15 - mmengine - INFO - Iter(train) [17500/19176] lr: 3.9844e-07 eta: 0:40:45 time: 1.7518 data_time: 0.0148 memory: 12042 loss: 0.1595 2025/03/24 00:32:32 - mmengine - INFO - Iter(train) [17510/19176] lr: 3.9373e-07 eta: 0:40:31 time: 1.6765 data_time: 0.0143 memory: 11642 loss: 0.1632 2025/03/24 00:32:48 - mmengine - INFO - Iter(train) [17520/19176] lr: 3.8905e-07 eta: 0:40:16 time: 1.6325 data_time: 0.0145 memory: 11451 loss: 0.1374 2025/03/24 00:33:04 - mmengine - INFO - Iter(train) [17530/19176] lr: 3.8440e-07 eta: 0:40:02 time: 1.5783 data_time: 0.0157 memory: 11375 loss: 0.1322 2025/03/24 00:33:19 - mmengine - INFO - Iter(train) [17540/19176] lr: 3.7977e-07 eta: 0:39:47 time: 1.5347 data_time: 0.0154 memory: 11456 loss: 0.1267 2025/03/24 00:33:34 - mmengine - INFO - Iter(train) [17550/19176] lr: 3.7518e-07 eta: 0:39:33 time: 1.4448 data_time: 0.0144 memory: 11057 loss: 0.1508 2025/03/24 00:33:48 - mmengine - INFO - Iter(train) [17560/19176] lr: 3.7061e-07 eta: 0:39:18 time: 1.3710 data_time: 0.0147 memory: 10930 loss: 0.1429 2025/03/24 00:33:59 - mmengine - INFO - Iter(train) [17570/19176] lr: 3.6607e-07 eta: 0:39:03 time: 1.1199 data_time: 0.0131 memory: 10534 loss: 0.1742 2025/03/24 00:34:07 - mmengine - INFO - Iter(train) [17580/19176] lr: 3.6155e-07 eta: 0:38:48 time: 0.8648 data_time: 0.0125 memory: 9952 loss: 0.1605 2025/03/24 00:34:24 - mmengine - INFO - Iter(train) [17590/19176] lr: 3.5707e-07 eta: 0:38:34 time: 1.6401 data_time: 0.0132 memory: 13346 loss: 0.1550 2025/03/24 00:34:42 - mmengine - INFO - Iter(train) [17600/19176] lr: 3.5261e-07 eta: 0:38:19 time: 1.7765 data_time: 0.0149 memory: 12005 loss: 0.1338 2025/03/24 00:34:59 - mmengine - INFO - Iter(train) [17610/19176] lr: 3.4818e-07 eta: 0:38:05 time: 1.7185 data_time: 0.0150 memory: 11790 loss: 0.1516 2025/03/24 00:35:15 - mmengine - INFO - Iter(train) [17620/19176] lr: 3.4377e-07 eta: 0:37:50 time: 1.6336 data_time: 0.0149 memory: 11586 loss: 0.1507 2025/03/24 00:35:31 - mmengine - INFO - Iter(train) [17630/19176] lr: 3.3940e-07 eta: 0:37:36 time: 1.5851 data_time: 0.0149 memory: 11339 loss: 0.1581 2025/03/24 00:35:46 - mmengine - INFO - Iter(train) [17640/19176] lr: 3.3505e-07 eta: 0:37:21 time: 1.5347 data_time: 0.0150 memory: 11303 loss: 0.1437 2025/03/24 00:36:01 - mmengine - INFO - Iter(train) [17650/19176] lr: 3.3072e-07 eta: 0:37:07 time: 1.4807 data_time: 0.0146 memory: 11104 loss: 0.1610 2025/03/24 00:36:14 - mmengine - INFO - Iter(train) [17660/19176] lr: 3.2643e-07 eta: 0:36:52 time: 1.2667 data_time: 0.0142 memory: 10716 loss: 0.1157 2025/03/24 00:36:25 - mmengine - INFO - Iter(train) [17670/19176] lr: 3.2216e-07 eta: 0:36:37 time: 1.1005 data_time: 0.0126 memory: 10287 loss: 0.1407 2025/03/24 00:36:34 - mmengine - INFO - Iter(train) [17680/19176] lr: 3.1793e-07 eta: 0:36:22 time: 0.8926 data_time: 0.0121 memory: 10000 loss: 0.1392 2025/03/24 00:36:52 - mmengine - INFO - Iter(train) [17690/19176] lr: 3.1371e-07 eta: 0:36:08 time: 1.7948 data_time: 0.0135 memory: 15096 loss: 0.1627 2025/03/24 00:37:09 - mmengine - INFO - Iter(train) [17700/19176] lr: 3.0953e-07 eta: 0:35:53 time: 1.7678 data_time: 0.0149 memory: 12367 loss: 0.1598 2025/03/24 00:37:26 - mmengine - INFO - Iter(train) [17710/19176] lr: 3.0538e-07 eta: 0:35:39 time: 1.6884 data_time: 0.0149 memory: 11760 loss: 0.1482 2025/03/24 00:37:43 - mmengine - INFO - Iter(train) [17720/19176] lr: 3.0125e-07 eta: 0:35:25 time: 1.6370 data_time: 0.0149 memory: 11979 loss: 0.1300 2025/03/24 00:37:58 - mmengine - INFO - Iter(train) [17730/19176] lr: 2.9715e-07 eta: 0:35:10 time: 1.5473 data_time: 0.0142 memory: 11283 loss: 0.1482 2025/03/24 00:38:13 - mmengine - INFO - Iter(train) [17740/19176] lr: 2.9307e-07 eta: 0:34:56 time: 1.4937 data_time: 0.0146 memory: 11129 loss: 0.1506 2025/03/24 00:38:27 - mmengine - INFO - Iter(train) [17750/19176] lr: 2.8903e-07 eta: 0:34:41 time: 1.4314 data_time: 0.0149 memory: 11034 loss: 0.1436 2025/03/24 00:38:41 - mmengine - INFO - Iter(train) [17760/19176] lr: 2.8501e-07 eta: 0:34:26 time: 1.3857 data_time: 0.0143 memory: 10894 loss: 0.1467 2025/03/24 00:38:53 - mmengine - INFO - Iter(train) [17770/19176] lr: 2.8102e-07 eta: 0:34:11 time: 1.1777 data_time: 0.0137 memory: 10538 loss: 0.1249 2025/03/24 00:39:03 - mmengine - INFO - Iter(train) [17780/19176] lr: 2.7706e-07 eta: 0:33:56 time: 0.9975 data_time: 0.0124 memory: 10128 loss: 0.1367 2025/03/24 00:39:20 - mmengine - INFO - Iter(train) [17790/19176] lr: 2.7313e-07 eta: 0:33:42 time: 1.7261 data_time: 0.0135 memory: 15361 loss: 0.2243 2025/03/24 00:39:38 - mmengine - INFO - Iter(train) [17800/19176] lr: 2.6922e-07 eta: 0:33:28 time: 1.7372 data_time: 0.0148 memory: 11867 loss: 0.1567 2025/03/24 00:39:55 - mmengine - INFO - Iter(train) [17810/19176] lr: 2.6534e-07 eta: 0:33:13 time: 1.6924 data_time: 0.0147 memory: 11651 loss: 0.1434 2025/03/24 00:40:11 - mmengine - INFO - Iter(train) [17820/19176] lr: 2.6149e-07 eta: 0:32:59 time: 1.6329 data_time: 0.0144 memory: 11471 loss: 0.1320 2025/03/24 00:40:27 - mmengine - INFO - Iter(train) [17830/19176] lr: 2.5767e-07 eta: 0:32:44 time: 1.6199 data_time: 0.0150 memory: 11399 loss: 0.1333 2025/03/24 00:40:42 - mmengine - INFO - Iter(train) [17840/19176] lr: 2.5387e-07 eta: 0:32:30 time: 1.5313 data_time: 0.0146 memory: 11192 loss: 0.1425 2025/03/24 00:40:57 - mmengine - INFO - Iter(train) [17850/19176] lr: 2.5010e-07 eta: 0:32:15 time: 1.4684 data_time: 0.0144 memory: 11108 loss: 0.1450 2025/03/24 00:41:10 - mmengine - INFO - Iter(train) [17860/19176] lr: 2.4636e-07 eta: 0:32:01 time: 1.3258 data_time: 0.0142 memory: 10924 loss: 0.1393 2025/03/24 00:41:21 - mmengine - INFO - Iter(train) [17870/19176] lr: 2.4265e-07 eta: 0:31:46 time: 1.0389 data_time: 0.0127 memory: 10245 loss: 0.1440 2025/03/24 00:41:28 - mmengine - INFO - Iter(train) [17880/19176] lr: 2.3897e-07 eta: 0:31:31 time: 0.7630 data_time: 0.0116 memory: 9768 loss: 0.1303 2025/03/24 00:41:46 - mmengine - INFO - Iter(train) [17890/19176] lr: 2.3531e-07 eta: 0:31:16 time: 1.7603 data_time: 0.0134 memory: 15963 loss: 0.1332 2025/03/24 00:42:04 - mmengine - INFO - Iter(train) [17900/19176] lr: 2.3168e-07 eta: 0:31:02 time: 1.8132 data_time: 0.0149 memory: 12312 loss: 0.1396 2025/03/24 00:42:21 - mmengine - INFO - Iter(train) [17910/19176] lr: 2.2808e-07 eta: 0:30:47 time: 1.7156 data_time: 0.0148 memory: 11894 loss: 0.1407 2025/03/24 00:42:38 - mmengine - INFO - Iter(train) [17920/19176] lr: 2.2451e-07 eta: 0:30:33 time: 1.6688 data_time: 0.0147 memory: 11553 loss: 0.1495 2025/03/24 00:42:54 - mmengine - INFO - Iter(train) [17930/19176] lr: 2.2097e-07 eta: 0:30:18 time: 1.6055 data_time: 0.0144 memory: 11391 loss: 0.1410 2025/03/24 00:43:09 - mmengine - INFO - Iter(train) [17940/19176] lr: 2.1745e-07 eta: 0:30:04 time: 1.5534 data_time: 0.0145 memory: 11304 loss: 0.1469 2025/03/24 00:43:24 - mmengine - INFO - Iter(train) [17950/19176] lr: 2.1396e-07 eta: 0:29:49 time: 1.4741 data_time: 0.0147 memory: 11183 loss: 0.1479 2025/03/24 00:43:38 - mmengine - INFO - Iter(train) [17960/19176] lr: 2.1050e-07 eta: 0:29:35 time: 1.3328 data_time: 0.0142 memory: 10961 loss: 0.1386 2025/03/24 00:43:49 - mmengine - INFO - Iter(train) [17970/19176] lr: 2.0707e-07 eta: 0:29:20 time: 1.1176 data_time: 0.0133 memory: 10401 loss: 0.1258 2025/03/24 00:43:58 - mmengine - INFO - Iter(train) [17980/19176] lr: 2.0366e-07 eta: 0:29:05 time: 0.9015 data_time: 0.0121 memory: 9999 loss: 0.1408 2025/03/24 00:44:14 - mmengine - INFO - Iter(train) [17990/19176] lr: 2.0028e-07 eta: 0:28:50 time: 1.6635 data_time: 0.0136 memory: 13236 loss: 0.1485 2025/03/24 00:44:32 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/24 00:44:32 - mmengine - INFO - Iter(train) [18000/19176] lr: 1.9693e-07 eta: 0:28:36 time: 1.7365 data_time: 0.0148 memory: 11891 loss: 0.1250 2025/03/24 00:44:32 - mmengine - INFO - Saving checkpoint at 18000 iterations 2025/03/24 00:44:49 - mmengine - INFO - Iter(train) [18010/19176] lr: 1.9361e-07 eta: 0:28:22 time: 1.7710 data_time: 0.0964 memory: 11744 loss: 0.1444 2025/03/24 00:45:06 - mmengine - INFO - Iter(train) [18020/19176] lr: 1.9032e-07 eta: 0:28:07 time: 1.6180 data_time: 0.0148 memory: 11472 loss: 0.1313 2025/03/24 00:45:21 - mmengine - INFO - Iter(train) [18030/19176] lr: 1.8705e-07 eta: 0:27:53 time: 1.5666 data_time: 0.0151 memory: 11428 loss: 0.1457 2025/03/24 00:45:36 - mmengine - INFO - Iter(train) [18040/19176] lr: 1.8382e-07 eta: 0:27:38 time: 1.5009 data_time: 0.0147 memory: 11276 loss: 0.1568 2025/03/24 00:45:50 - mmengine - INFO - Iter(train) [18050/19176] lr: 1.8061e-07 eta: 0:27:23 time: 1.3602 data_time: 0.0142 memory: 10971 loss: 0.1418 2025/03/24 00:46:02 - mmengine - INFO - Iter(train) [18060/19176] lr: 1.7743e-07 eta: 0:27:09 time: 1.1603 data_time: 0.0130 memory: 10550 loss: 0.1389 2025/03/24 00:46:12 - mmengine - INFO - Iter(train) [18070/19176] lr: 1.7427e-07 eta: 0:26:54 time: 1.0520 data_time: 0.0125 memory: 10262 loss: 0.1512 2025/03/24 00:46:21 - mmengine - INFO - Iter(train) [18080/19176] lr: 1.7115e-07 eta: 0:26:39 time: 0.8549 data_time: 0.0116 memory: 9814 loss: 0.1361 2025/03/24 00:46:42 - mmengine - INFO - Iter(train) [18090/19176] lr: 1.6805e-07 eta: 0:26:25 time: 2.0916 data_time: 0.0139 memory: 18970 loss: 0.1769 2025/03/24 00:46:59 - mmengine - INFO - Iter(train) [18100/19176] lr: 1.6498e-07 eta: 0:26:10 time: 1.7538 data_time: 0.0139 memory: 12077 loss: 0.1416 2025/03/24 00:47:16 - mmengine - INFO - Iter(train) [18110/19176] lr: 1.6194e-07 eta: 0:25:56 time: 1.6740 data_time: 0.0136 memory: 11645 loss: 0.1508 2025/03/24 00:47:32 - mmengine - INFO - Iter(train) [18120/19176] lr: 1.5893e-07 eta: 0:25:41 time: 1.6104 data_time: 0.0138 memory: 11380 loss: 0.1455 2025/03/24 00:47:47 - mmengine - INFO - Iter(train) [18130/19176] lr: 1.5594e-07 eta: 0:25:27 time: 1.5245 data_time: 0.0136 memory: 11225 loss: 0.1496 2025/03/24 00:48:02 - mmengine - INFO - Iter(train) [18140/19176] lr: 1.5298e-07 eta: 0:25:12 time: 1.4629 data_time: 0.0138 memory: 11102 loss: 0.1523 2025/03/24 00:48:16 - mmengine - INFO - Iter(train) [18150/19176] lr: 1.5005e-07 eta: 0:24:57 time: 1.4046 data_time: 0.0134 memory: 11006 loss: 0.1468 2025/03/24 00:48:29 - mmengine - INFO - Iter(train) [18160/19176] lr: 1.4715e-07 eta: 0:24:43 time: 1.3046 data_time: 0.0137 memory: 10716 loss: 0.1613 2025/03/24 00:48:39 - mmengine - INFO - Iter(train) [18170/19176] lr: 1.4428e-07 eta: 0:24:28 time: 1.0565 data_time: 0.0124 memory: 10367 loss: 0.1519 2025/03/24 00:48:48 - mmengine - INFO - Iter(train) [18180/19176] lr: 1.4144e-07 eta: 0:24:13 time: 0.8475 data_time: 0.0120 memory: 9909 loss: 0.1431 2025/03/24 00:49:05 - mmengine - INFO - Iter(train) [18190/19176] lr: 1.3862e-07 eta: 0:23:58 time: 1.7432 data_time: 0.0142 memory: 15665 loss: 0.1254 2025/03/24 00:49:22 - mmengine - INFO - Iter(train) [18200/19176] lr: 1.3583e-07 eta: 0:23:44 time: 1.7171 data_time: 0.0149 memory: 11749 loss: 0.1379 2025/03/24 00:49:39 - mmengine - INFO - Iter(train) [18210/19176] lr: 1.3307e-07 eta: 0:23:30 time: 1.6880 data_time: 0.0150 memory: 11664 loss: 0.1501 2025/03/24 00:49:56 - mmengine - INFO - Iter(train) [18220/19176] lr: 1.3034e-07 eta: 0:23:15 time: 1.6376 data_time: 0.0149 memory: 11499 loss: 0.1519 2025/03/24 00:50:12 - mmengine - INFO - Iter(train) [18230/19176] lr: 1.2764e-07 eta: 0:23:00 time: 1.5936 data_time: 0.0147 memory: 11357 loss: 0.1330 2025/03/24 00:50:27 - mmengine - INFO - Iter(train) [18240/19176] lr: 1.2496e-07 eta: 0:22:46 time: 1.5164 data_time: 0.0145 memory: 11176 loss: 0.1368 2025/03/24 00:50:42 - mmengine - INFO - Iter(train) [18250/19176] lr: 1.2231e-07 eta: 0:22:31 time: 1.4797 data_time: 0.0146 memory: 11089 loss: 0.1271 2025/03/24 00:50:56 - mmengine - INFO - Iter(train) [18260/19176] lr: 1.1969e-07 eta: 0:22:17 time: 1.4076 data_time: 0.0143 memory: 11020 loss: 0.1419 2025/03/24 00:51:07 - mmengine - INFO - Iter(train) [18270/19176] lr: 1.1710e-07 eta: 0:22:02 time: 1.1579 data_time: 0.0135 memory: 10595 loss: 0.1792 2025/03/24 00:51:16 - mmengine - INFO - Iter(train) [18280/19176] lr: 1.1454e-07 eta: 0:21:47 time: 0.8403 data_time: 0.0121 memory: 10065 loss: 0.1283 2025/03/24 00:51:32 - mmengine - INFO - Iter(train) [18290/19176] lr: 1.1200e-07 eta: 0:21:33 time: 1.6400 data_time: 0.0136 memory: 12965 loss: 0.1480 2025/03/24 00:51:50 - mmengine - INFO - Iter(train) [18300/19176] lr: 1.0950e-07 eta: 0:21:18 time: 1.7728 data_time: 0.0148 memory: 12101 loss: 0.1509 2025/03/24 00:52:07 - mmengine - INFO - Iter(train) [18310/19176] lr: 1.0702e-07 eta: 0:21:04 time: 1.6783 data_time: 0.0151 memory: 11681 loss: 0.1313 2025/03/24 00:52:23 - mmengine - INFO - Iter(train) [18320/19176] lr: 1.0457e-07 eta: 0:20:49 time: 1.6087 data_time: 0.0148 memory: 11408 loss: 0.1596 2025/03/24 00:52:38 - mmengine - INFO - Iter(train) [18330/19176] lr: 1.0215e-07 eta: 0:20:35 time: 1.5676 data_time: 0.0151 memory: 11376 loss: 0.1388 2025/03/24 00:52:53 - mmengine - INFO - Iter(train) [18340/19176] lr: 9.9753e-08 eta: 0:20:20 time: 1.5096 data_time: 0.0147 memory: 11240 loss: 0.1691 2025/03/24 00:53:07 - mmengine - INFO - Iter(train) [18350/19176] lr: 9.7387e-08 eta: 0:20:05 time: 1.3711 data_time: 0.0143 memory: 10918 loss: 0.1633 2025/03/24 00:53:19 - mmengine - INFO - Iter(train) [18360/19176] lr: 9.5050e-08 eta: 0:19:51 time: 1.1730 data_time: 0.0133 memory: 10646 loss: 0.1448 2025/03/24 00:53:29 - mmengine - INFO - Iter(train) [18370/19176] lr: 9.2741e-08 eta: 0:19:36 time: 1.0139 data_time: 0.0127 memory: 10093 loss: 0.1547 2025/03/24 00:53:37 - mmengine - INFO - Iter(train) [18380/19176] lr: 9.0460e-08 eta: 0:19:21 time: 0.8189 data_time: 0.0121 memory: 9821 loss: 0.1551 2025/03/24 00:53:54 - mmengine - INFO - Iter(train) [18390/19176] lr: 8.8208e-08 eta: 0:19:06 time: 1.7151 data_time: 0.0134 memory: 14162 loss: 0.1511 2025/03/24 00:54:12 - mmengine - INFO - Iter(train) [18400/19176] lr: 8.5984e-08 eta: 0:18:52 time: 1.7234 data_time: 0.0145 memory: 12075 loss: 0.1360 2025/03/24 00:54:28 - mmengine - INFO - Iter(train) [18410/19176] lr: 8.3788e-08 eta: 0:18:37 time: 1.6463 data_time: 0.0149 memory: 11559 loss: 0.1480 2025/03/24 00:54:44 - mmengine - INFO - Iter(train) [18420/19176] lr: 8.1620e-08 eta: 0:18:23 time: 1.6137 data_time: 0.0149 memory: 11465 loss: 0.1393 2025/03/24 00:54:59 - mmengine - INFO - Iter(train) [18430/19176] lr: 7.9481e-08 eta: 0:18:08 time: 1.4980 data_time: 0.0145 memory: 11263 loss: 0.1460 2025/03/24 00:55:14 - mmengine - INFO - Iter(train) [18440/19176] lr: 7.7370e-08 eta: 0:17:54 time: 1.4319 data_time: 0.0146 memory: 11034 loss: 0.1346 2025/03/24 00:55:27 - mmengine - INFO - Iter(train) [18450/19176] lr: 7.5287e-08 eta: 0:17:39 time: 1.3549 data_time: 0.0141 memory: 10912 loss: 0.1602 2025/03/24 00:55:40 - mmengine - INFO - Iter(train) [18460/19176] lr: 7.3233e-08 eta: 0:17:24 time: 1.2443 data_time: 0.0138 memory: 10696 loss: 0.1300 2025/03/24 00:55:50 - mmengine - INFO - Iter(train) [18470/19176] lr: 7.1207e-08 eta: 0:17:10 time: 1.0493 data_time: 0.0124 memory: 10186 loss: 0.1495 2025/03/24 00:55:58 - mmengine - INFO - Iter(train) [18480/19176] lr: 6.9209e-08 eta: 0:16:55 time: 0.8049 data_time: 0.0118 memory: 9903 loss: 0.1504 2025/03/24 00:56:15 - mmengine - INFO - Iter(train) [18490/19176] lr: 6.7239e-08 eta: 0:16:40 time: 1.6492 data_time: 0.0128 memory: 13511 loss: 0.1485 2025/03/24 00:56:32 - mmengine - INFO - Iter(train) [18500/19176] lr: 6.5298e-08 eta: 0:16:26 time: 1.7743 data_time: 0.0137 memory: 12061 loss: 0.1675 2025/03/24 00:56:49 - mmengine - INFO - Iter(train) [18510/19176] lr: 6.3385e-08 eta: 0:16:11 time: 1.7110 data_time: 0.0138 memory: 11749 loss: 0.1535 2025/03/24 00:57:06 - mmengine - INFO - Iter(train) [18520/19176] lr: 6.1501e-08 eta: 0:15:57 time: 1.6641 data_time: 0.0146 memory: 11493 loss: 0.1436 2025/03/24 00:57:22 - mmengine - INFO - Iter(train) [18530/19176] lr: 5.9645e-08 eta: 0:15:42 time: 1.5871 data_time: 0.0139 memory: 11393 loss: 0.1439 2025/03/24 00:57:37 - mmengine - INFO - Iter(train) [18540/19176] lr: 5.7817e-08 eta: 0:15:28 time: 1.5386 data_time: 0.0158 memory: 11233 loss: 0.1423 2025/03/24 00:57:52 - mmengine - INFO - Iter(train) [18550/19176] lr: 5.6018e-08 eta: 0:15:13 time: 1.4769 data_time: 0.0141 memory: 11152 loss: 0.1558 2025/03/24 00:58:06 - mmengine - INFO - Iter(train) [18560/19176] lr: 5.4247e-08 eta: 0:14:58 time: 1.3578 data_time: 0.0141 memory: 10891 loss: 0.1570 2025/03/24 00:58:18 - mmengine - INFO - Iter(train) [18570/19176] lr: 5.2504e-08 eta: 0:14:44 time: 1.2013 data_time: 0.0142 memory: 10547 loss: 0.1587 2025/03/24 00:58:28 - mmengine - INFO - Iter(train) [18580/19176] lr: 5.0790e-08 eta: 0:14:29 time: 1.0313 data_time: 0.0135 memory: 10248 loss: 0.1606 2025/03/24 00:58:46 - mmengine - INFO - Iter(train) [18590/19176] lr: 4.9104e-08 eta: 0:14:15 time: 1.8002 data_time: 0.0148 memory: 14488 loss: 0.1584 2025/03/24 00:59:04 - mmengine - INFO - Iter(train) [18600/19176] lr: 4.7447e-08 eta: 0:14:00 time: 1.7647 data_time: 0.0148 memory: 12169 loss: 0.1503 2025/03/24 00:59:20 - mmengine - INFO - Iter(train) [18610/19176] lr: 4.5817e-08 eta: 0:13:46 time: 1.6846 data_time: 0.0147 memory: 11695 loss: 0.1345 2025/03/24 00:59:37 - mmengine - INFO - Iter(train) [18620/19176] lr: 4.4217e-08 eta: 0:13:31 time: 1.6398 data_time: 0.0146 memory: 11533 loss: 0.1303 2025/03/24 00:59:53 - mmengine - INFO - Iter(train) [18630/19176] lr: 4.2644e-08 eta: 0:13:16 time: 1.5860 data_time: 0.0154 memory: 11358 loss: 0.1483 2025/03/24 01:00:08 - mmengine - INFO - Iter(train) [18640/19176] lr: 4.1101e-08 eta: 0:13:02 time: 1.5193 data_time: 0.0146 memory: 11204 loss: 0.1662 2025/03/24 01:00:22 - mmengine - INFO - Iter(train) [18650/19176] lr: 3.9585e-08 eta: 0:12:47 time: 1.4392 data_time: 0.0144 memory: 11155 loss: 0.1468 2025/03/24 01:00:35 - mmengine - INFO - Iter(train) [18660/19176] lr: 3.8098e-08 eta: 0:12:33 time: 1.3058 data_time: 0.0144 memory: 10813 loss: 0.1390 2025/03/24 01:00:46 - mmengine - INFO - Iter(train) [18670/19176] lr: 3.6639e-08 eta: 0:12:18 time: 1.0877 data_time: 0.0127 memory: 10317 loss: 0.1583 2025/03/24 01:00:55 - mmengine - INFO - Iter(train) [18680/19176] lr: 3.5209e-08 eta: 0:12:03 time: 0.8457 data_time: 0.0121 memory: 9876 loss: 0.1567 2025/03/24 01:01:11 - mmengine - INFO - Iter(train) [18690/19176] lr: 3.3807e-08 eta: 0:11:49 time: 1.6158 data_time: 0.0134 memory: 12724 loss: 0.1482 2025/03/24 01:01:28 - mmengine - INFO - Iter(train) [18700/19176] lr: 3.2434e-08 eta: 0:11:34 time: 1.7532 data_time: 0.0147 memory: 12075 loss: 0.1341 2025/03/24 01:01:45 - mmengine - INFO - Iter(train) [18710/19176] lr: 3.1089e-08 eta: 0:11:20 time: 1.6566 data_time: 0.0144 memory: 11817 loss: 0.1344 2025/03/24 01:02:01 - mmengine - INFO - Iter(train) [18720/19176] lr: 2.9772e-08 eta: 0:11:05 time: 1.5942 data_time: 0.0148 memory: 11374 loss: 0.1354 2025/03/24 01:02:16 - mmengine - INFO - Iter(train) [18730/19176] lr: 2.8484e-08 eta: 0:10:50 time: 1.5419 data_time: 0.0148 memory: 11238 loss: 0.1745 2025/03/24 01:02:31 - mmengine - INFO - Iter(train) [18740/19176] lr: 2.7225e-08 eta: 0:10:36 time: 1.4978 data_time: 0.0149 memory: 11224 loss: 0.1494 2025/03/24 01:02:46 - mmengine - INFO - Iter(train) [18750/19176] lr: 2.5993e-08 eta: 0:10:21 time: 1.4448 data_time: 0.0149 memory: 11019 loss: 0.1541 2025/03/24 01:02:58 - mmengine - INFO - Iter(train) [18760/19176] lr: 2.4791e-08 eta: 0:10:07 time: 1.2716 data_time: 0.0143 memory: 10761 loss: 0.1570 2025/03/24 01:03:10 - mmengine - INFO - Iter(train) [18770/19176] lr: 2.3616e-08 eta: 0:09:52 time: 1.1199 data_time: 0.0132 memory: 10345 loss: 0.1503 2025/03/24 01:03:20 - mmengine - INFO - Iter(train) [18780/19176] lr: 2.2471e-08 eta: 0:09:37 time: 0.9918 data_time: 0.0122 memory: 10076 loss: 0.1424 2025/03/24 01:03:38 - mmengine - INFO - Iter(train) [18790/19176] lr: 2.1353e-08 eta: 0:09:23 time: 1.8069 data_time: 0.0137 memory: 14750 loss: 0.1545 2025/03/24 01:03:55 - mmengine - INFO - Iter(train) [18800/19176] lr: 2.0264e-08 eta: 0:09:08 time: 1.7349 data_time: 0.0150 memory: 11840 loss: 0.1353 2025/03/24 01:04:12 - mmengine - INFO - Iter(train) [18810/19176] lr: 1.9204e-08 eta: 0:08:54 time: 1.6689 data_time: 0.0148 memory: 11555 loss: 0.1364 2025/03/24 01:04:28 - mmengine - INFO - Iter(train) [18820/19176] lr: 1.8172e-08 eta: 0:08:39 time: 1.6240 data_time: 0.0148 memory: 11429 loss: 0.1503 2025/03/24 01:04:44 - mmengine - INFO - Iter(train) [18830/19176] lr: 1.7168e-08 eta: 0:08:25 time: 1.5648 data_time: 0.0147 memory: 11309 loss: 0.1370 2025/03/24 01:04:59 - mmengine - INFO - Iter(train) [18840/19176] lr: 1.6193e-08 eta: 0:08:10 time: 1.5293 data_time: 0.0147 memory: 11231 loss: 0.1451 2025/03/24 01:05:14 - mmengine - INFO - Iter(train) [18850/19176] lr: 1.5247e-08 eta: 0:07:55 time: 1.4900 data_time: 0.0149 memory: 11127 loss: 0.1392 2025/03/24 01:05:27 - mmengine - INFO - Iter(train) [18860/19176] lr: 1.4329e-08 eta: 0:07:41 time: 1.3175 data_time: 0.0140 memory: 10925 loss: 0.1361 2025/03/24 01:05:37 - mmengine - INFO - Iter(train) [18870/19176] lr: 1.3439e-08 eta: 0:07:26 time: 1.0444 data_time: 0.0130 memory: 10301 loss: 0.1448 2025/03/24 01:05:45 - mmengine - INFO - Iter(train) [18880/19176] lr: 1.2578e-08 eta: 0:07:11 time: 0.7925 data_time: 0.0123 memory: 9815 loss: 0.1526 2025/03/24 01:06:02 - mmengine - INFO - Iter(train) [18890/19176] lr: 1.1746e-08 eta: 0:06:57 time: 1.6648 data_time: 0.0135 memory: 15218 loss: 0.1391 2025/03/24 01:06:20 - mmengine - INFO - Iter(train) [18900/19176] lr: 1.0942e-08 eta: 0:06:42 time: 1.7729 data_time: 0.0151 memory: 12047 loss: 0.1563 2025/03/24 01:06:37 - mmengine - INFO - Iter(train) [18910/19176] lr: 1.0166e-08 eta: 0:06:28 time: 1.6890 data_time: 0.0150 memory: 11653 loss: 0.1516 2025/03/24 01:06:53 - mmengine - INFO - Iter(train) [18920/19176] lr: 9.4188e-09 eta: 0:06:13 time: 1.6369 data_time: 0.0151 memory: 11506 loss: 0.1587 2025/03/24 01:07:09 - mmengine - INFO - Iter(train) [18930/19176] lr: 8.7002e-09 eta: 0:05:59 time: 1.5789 data_time: 0.0150 memory: 11344 loss: 0.1481 2025/03/24 01:07:24 - mmengine - INFO - Iter(train) [18940/19176] lr: 8.0101e-09 eta: 0:05:44 time: 1.5228 data_time: 0.0149 memory: 11245 loss: 0.1525 2025/03/24 01:07:39 - mmengine - INFO - Iter(train) [18950/19176] lr: 7.3484e-09 eta: 0:05:29 time: 1.4687 data_time: 0.0150 memory: 11136 loss: 0.1228 2025/03/24 01:07:53 - mmengine - INFO - Iter(train) [18960/19176] lr: 6.7153e-09 eta: 0:05:15 time: 1.3947 data_time: 0.0147 memory: 10901 loss: 0.1532 2025/03/24 01:08:04 - mmengine - INFO - Iter(train) [18970/19176] lr: 6.1107e-09 eta: 0:05:00 time: 1.1753 data_time: 0.0129 memory: 10640 loss: 0.1512 2025/03/24 01:08:13 - mmengine - INFO - Iter(train) [18980/19176] lr: 5.5346e-09 eta: 0:04:45 time: 0.8297 data_time: 0.0116 memory: 9953 loss: 0.1244 2025/03/24 01:08:29 - mmengine - INFO - Iter(train) [18990/19176] lr: 4.9871e-09 eta: 0:04:31 time: 1.6172 data_time: 0.0133 memory: 13545 loss: 0.1712 2025/03/24 01:08:46 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20250323_172626 2025/03/24 01:08:46 - mmengine - INFO - Iter(train) [19000/19176] lr: 4.4680e-09 eta: 0:04:16 time: 1.7352 data_time: 0.0146 memory: 11872 loss: 0.1443 2025/03/24 01:08:46 - mmengine - INFO - Saving checkpoint at 19000 iterations 2025/03/24 01:09:04 - mmengine - INFO - Iter(train) [19010/19176] lr: 3.9774e-09 eta: 0:04:02 time: 1.7504 data_time: 0.0923 memory: 11541 loss: 0.1357 2025/03/24 01:09:20 - mmengine - INFO - Iter(train) [19020/19176] lr: 3.5154e-09 eta: 0:03:47 time: 1.6260 data_time: 0.0146 memory: 11450 loss: 0.1401 2025/03/24 01:09:36 - mmengine - INFO - Iter(train) [19030/19176] lr: 3.0818e-09 eta: 0:03:33 time: 1.5727 data_time: 0.0142 memory: 11322 loss: 0.1476 2025/03/24 01:09:51 - mmengine - INFO - Iter(train) [19040/19176] lr: 2.6768e-09 eta: 0:03:18 time: 1.5201 data_time: 0.0146 memory: 11229 loss: 0.1563 2025/03/24 01:10:06 - mmengine - INFO - Iter(train) [19050/19176] lr: 2.3003e-09 eta: 0:03:03 time: 1.4631 data_time: 0.0138 memory: 11136 loss: 0.1470 2025/03/24 01:10:19 - mmengine - INFO - Iter(train) [19060/19176] lr: 1.9523e-09 eta: 0:02:49 time: 1.3577 data_time: 0.0141 memory: 10916 loss: 0.1513 2025/03/24 01:10:31 - mmengine - INFO - Iter(train) [19070/19176] lr: 1.6329e-09 eta: 0:02:34 time: 1.1443 data_time: 0.0130 memory: 10560 loss: 0.1491 2025/03/24 01:10:39 - mmengine - INFO - Iter(train) [19080/19176] lr: 1.3419e-09 eta: 0:02:20 time: 0.8117 data_time: 0.0120 memory: 10041 loss: 0.1365 2025/03/24 01:10:55 - mmengine - INFO - Iter(train) [19090/19176] lr: 1.0795e-09 eta: 0:02:05 time: 1.6435 data_time: 0.0135 memory: 13524 loss: 0.1409 2025/03/24 01:11:12 - mmengine - INFO - Iter(train) [19100/19176] lr: 8.4561e-10 eta: 0:01:50 time: 1.7362 data_time: 0.0144 memory: 11946 loss: 0.1473 2025/03/24 01:11:29 - mmengine - INFO - Iter(train) [19110/19176] lr: 6.4024e-10 eta: 0:01:36 time: 1.6822 data_time: 0.0145 memory: 11704 loss: 0.1429 2025/03/24 01:11:46 - mmengine - INFO - Iter(train) [19120/19176] lr: 4.6339e-10 eta: 0:01:21 time: 1.6222 data_time: 0.0146 memory: 11436 loss: 0.1554 2025/03/24 01:12:01 - mmengine - INFO - Iter(train) [19130/19176] lr: 3.1506e-10 eta: 0:01:07 time: 1.5626 data_time: 0.0143 memory: 11342 loss: 0.1592 2025/03/24 01:12:16 - mmengine - INFO - Iter(train) [19140/19176] lr: 1.9525e-10 eta: 0:00:52 time: 1.4758 data_time: 0.0140 memory: 11213 loss: 0.1332 2025/03/24 01:12:30 - mmengine - INFO - Iter(train) [19150/19176] lr: 1.0397e-10 eta: 0:00:37 time: 1.3601 data_time: 0.0144 memory: 10973 loss: 0.1531 2025/03/24 01:12:41 - mmengine - INFO - Iter(train) [19160/19176] lr: 4.1219e-11 eta: 0:00:23 time: 1.1645 data_time: 0.0131 memory: 10452 loss: 0.1306 2025/03/24 01:12:51 - mmengine - INFO - Iter(train) [19170/19176] lr: 6.9886e-12 eta: 0:00:08 time: 0.9722 data_time: 0.0124 memory: 10176 loss: 0.1692 2025/03/24 01:12:55 - mmengine - INFO - Saving checkpoint at 19176 iterations