2024/07/24 14:25:33 - mmengine - DEBUG - An `DeepSpeedStrategy` instance is built from registry, and its implementation can be found in xtuner.engine._strategy.deepspeed 2024/07/24 14:25:33 - mmengine - INFO - ------------------------------------------------------------ System environment: sys.platform: linux Python: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] CUDA available: True MUSA available: False numpy_random_seed: 1567780313 GPU 0: NVIDIA A100-SXM4-80GB CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.2, V12.2.140 GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 PyTorch: 2.3.1+cu121 PyTorch compiling details: PyTorch built with: - GCC 9.3 - C++ Version: 201703 - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) - OpenMP 201511 (a.k.a. OpenMP 4.5) - LAPACK is enabled (usually provided by MKL) - NNPACK is enabled - CPU capability usage: AVX512 - CUDA Runtime 12.1 - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90 - CuDNN 8.9.2 - Magma 2.6.1 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.3.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, TorchVision: 0.18.1+cu121 OpenCV: 4.9.0 MMEngine: 0.10.3 Runtime environment: launcher: none randomness: {'seed': None, 'deterministic': False} cudnn_benchmark: False mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: None deterministic: False Distributed launcher: none Distributed training: False GPU number: 1 ------------------------------------------------------------ 2024/07/24 14:25:33 - mmengine - INFO - Config: accumulative_counts = 2 batch_size = 1 betas = ( 0.9, 0.999, ) custom_hooks = [ dict( tokenizer=dict( pretrained_model_name_or_path='/root/models/InternVL2_2B', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.engine.hooks.DatasetInfoHook'), ] data_path = '/root/data/screenshot_od/layout_ocr_multi.json' data_root = '/root/data/' dataloader_num_workers = 4 default_hooks = dict( checkpoint=dict( by_epoch=False, interval=1000, max_keep_ckpts=-1, save_optimizer=False, type='mmengine.hooks.CheckpointHook'), logger=dict( interval=10, log_metric_by_epoch=False, type='mmengine.hooks.LoggerHook'), param_scheduler=dict(type='mmengine.hooks.ParamSchedulerHook'), sampler_seed=dict(type='mmengine.hooks.DistSamplerSeedHook'), timer=dict(type='mmengine.hooks.IterTimerHook')) env_cfg = dict( cudnn_benchmark=False, dist_cfg=dict(backend='nccl'), mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) image_folder = '/root/data/llava_images' launcher = 'none' llava_dataset = dict( data_paths='/root/data/screenshot_od/layout_ocr_multi.json', image_folders='/root/data/llava_images', max_length=8192, model_path='/root/models/InternVL2_2B', template='xtuner.utils.PROMPT_TEMPLATE.internlm2_chat', type='xtuner.dataset.InternVL_V1_5_Dataset') load_from = None log_level = 'DEBUG' log_processor = dict(by_epoch=False) lr = 2e-05 max_epochs = 4 max_length = 8192 max_norm = 1 model = dict( freeze_llm=True, freeze_visual_encoder=True, llm_lora=dict( lora_alpha=256, lora_dropout=0.05, r=128, target_modules=None, task_type='CAUSAL_LM', type='peft.LoraConfig'), model_path='/root/models/InternVL2_2B', quantization_llm=True, quantization_vit=False, type='xtuner.model.InternVL_V1_5') optim_type = 'torch.optim.AdamW' optim_wrapper = dict( optimizer=dict( betas=( 0.9, 0.999, ), lr=2e-05, type='torch.optim.AdamW', weight_decay=0.05), type='DeepSpeedOptimWrapper') param_scheduler = [ dict( begin=0, by_epoch=True, convert_to_iter_based=True, end=0.12, start_factor=1e-05, type='mmengine.optim.LinearLR'), dict( begin=0.12, by_epoch=True, convert_to_iter_based=True, end=4, eta_min=0.0, type='mmengine.optim.CosineAnnealingLR'), ] path = '/root/models/InternVL2_2B' prompt_template = 'xtuner.utils.PROMPT_TEMPLATE.internlm2_chat' randomness = dict(deterministic=False, seed=None) resume = False runner_type = 'FlexibleRunner' save_steps = 1000 save_total_limit = -1 strategy = dict( config=dict( bf16=dict(enabled=True), fp16=dict(enabled=False, initial_scale_power=16), gradient_accumulation_steps='auto', gradient_clipping='auto', train_micro_batch_size_per_gpu='auto', zero_allow_untested_optimizer=True, zero_force_ds_cpu_optimizer=False, zero_optimization=dict(overlap_comm=True, stage=2)), exclude_frozen_parameters=True, gradient_accumulation_steps=2, gradient_clipping=1, sequence_parallel_size=1, train_micro_batch_size_per_gpu=1, type='xtuner.engine.DeepSpeedStrategy') tokenizer = dict( pretrained_model_name_or_path='/root/models/InternVL2_2B', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained') train_cfg = dict(max_epochs=4, type='xtuner.engine.runner.TrainLoop') train_dataloader = dict( batch_size=1, collate_fn=dict(type='xtuner.dataset.collate_fns.default_collate_fn'), dataset=dict( data_paths='/root/data/screenshot_od/layout_ocr_multi.json', image_folders='/root/data/llava_images', max_length=8192, model_path='/root/models/InternVL2_2B', template='xtuner.utils.PROMPT_TEMPLATE.internlm2_chat', type='xtuner.dataset.InternVL_V1_5_Dataset'), num_workers=4, sampler=dict( length_property='modality_length', per_device_batch_size=2, type='xtuner.dataset.samplers.LengthGroupedSampler')) visualizer = dict( type='mmengine.visualization.Visualizer', vis_backends=[ dict(type='mmengine.visualization.TensorboardVisBackend'), ]) warmup_ratio = 0.03 weight_decay = 0.05 work_dir = '/root/wangqun/work_dirs/internvl_ft_run_6_filter' 2024/07/24 14:25:33 - mmengine - DEBUG - An `TensorboardVisBackend` instance is built from registry, and its implementation can be found in mmengine.visualization.vis_backend 2024/07/24 14:25:33 - mmengine - DEBUG - An `Visualizer` instance is built from registry, and its implementation can be found in mmengine.visualization.visualizer 2024/07/24 14:25:33 - mmengine - DEBUG - Attribute `_env_initialized` is not defined in or `._env_initialized is False, `_init_env` will be called and ._env_initialized will be set to True 2024/07/24 14:25:36 - mmengine - DEBUG - Get class `RuntimeInfoHook` from "hook" registry in "mmengine" 2024/07/24 14:25:36 - mmengine - DEBUG - An `RuntimeInfoHook` instance is built from registry, and its implementation can be found in mmengine.hooks.runtime_info_hook 2024/07/24 14:25:36 - mmengine - DEBUG - An `IterTimerHook` instance is built from registry, and its implementation can be found in mmengine.hooks.iter_timer_hook 2024/07/24 14:25:36 - mmengine - DEBUG - An `DistSamplerSeedHook` instance is built from registry, and its implementation can be found in mmengine.hooks.sampler_seed_hook 2024/07/24 14:25:36 - mmengine - DEBUG - An `LoggerHook` instance is built from registry, and its implementation can be found in mmengine.hooks.logger_hook 2024/07/24 14:25:36 - mmengine - DEBUG - An `ParamSchedulerHook` instance is built from registry, and its implementation can be found in mmengine.hooks.param_scheduler_hook 2024/07/24 14:25:36 - mmengine - DEBUG - An `CheckpointHook` instance is built from registry, and its implementation can be found in mmengine.hooks.checkpoint_hook 2024/07/24 14:25:36 - mmengine - WARNING - Failed to search registry with scope "mmengine" in the "builder" registry tree. As a workaround, the current "builder" registry in "xtuner" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmengine" is a correct scope, or whether the registry is initialized. 2024/07/24 14:25:36 - mmengine - DEBUG - An `from_pretrained` instance is built from registry, and its implementation can be found in transformers.models.auto.tokenization_auto 2024/07/24 14:25:36 - mmengine - DEBUG - An `DatasetInfoHook` instance is built from registry, and its implementation can be found in xtuner.engine.hooks.dataset_info_hook 2024/07/24 14:25:36 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook (BELOW_NORMAL) LoggerHook -------------------- before_train: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DatasetInfoHook (VERY_LOW ) CheckpointHook -------------------- before_train_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DistSamplerSeedHook -------------------- before_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook -------------------- after_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_train_epoch: (NORMAL ) IterTimerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- before_val: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) DatasetInfoHook -------------------- before_val_epoch: (NORMAL ) IterTimerHook -------------------- before_val_iter: (NORMAL ) IterTimerHook -------------------- after_val_iter: (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook -------------------- after_val_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook -------------------- after_val: (VERY_HIGH ) RuntimeInfoHook -------------------- after_train: (VERY_HIGH ) RuntimeInfoHook (VERY_LOW ) CheckpointHook -------------------- before_test: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) DatasetInfoHook -------------------- before_test_epoch: (NORMAL ) IterTimerHook -------------------- before_test_iter: (NORMAL ) IterTimerHook -------------------- after_test_iter: (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook -------------------- after_test_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook -------------------- after_test: (VERY_HIGH ) RuntimeInfoHook -------------------- after_run: (BELOW_NORMAL) LoggerHook -------------------- 2024/07/24 14:25:36 - mmengine - DEBUG - An `FlexibleRunner` instance is built from registry, its implementation can be found inmmengine.runner._flexible_runner 2024/07/24 14:25:36 - mmengine - INFO - Starting to loading data and calc length 2024/07/24 14:25:36 - mmengine - INFO - =======Starting to process /root/data/screenshot_od/layout_ocr_multi.json ======= 2024/07/24 14:25:43 - mmengine - INFO - =======total 4806 samples of /root/data/screenshot_od/layout_ocr_multi.json======= 2024/07/24 14:25:43 - mmengine - INFO - end loading data and calc length 2024/07/24 14:25:43 - mmengine - INFO - =======total 4806 samples======= 2024/07/24 14:25:43 - mmengine - DEBUG - An `InternVL_V1_5_Dataset` instance is built from registry, and its implementation can be found in xtuner.dataset.internvl_dataset 2024/07/24 14:25:43 - mmengine - INFO - LengthGroupedSampler is used. 2024/07/24 14:25:43 - mmengine - INFO - LengthGroupedSampler construction is complete, and the selected attribute is modality_length 2024/07/24 14:25:43 - mmengine - DEBUG - An `LengthGroupedSampler` instance is built from registry, and its implementation can be found in xtuner.dataset.samplers.length_grouped 2024/07/24 14:25:43 - mmengine - WARNING - Dataset InternVL_V1_5_Dataset has no metainfo. ``dataset_meta`` in visualizer will be None. 2024/07/24 14:25:43 - mmengine - DEBUG - An `TrainLoop` instance is built from registry, and its implementation can be found in xtuner.engine.runner.loops 2024/07/24 14:25:43 - mmengine - INFO - Start to load InternVL_V1_5 model. 2024/07/24 14:25:43 - mmengine - DEBUG - Get class `BaseDataPreprocessor` from "model" registry in "mmengine" 2024/07/24 14:25:43 - mmengine - DEBUG - An `BaseDataPreprocessor` instance is built from registry, and its implementation can be found in mmengine.model.base_model.data_preprocessor 2024/07/24 14:25:57 - mmengine - DEBUG - An `LoraConfig` instance is built from registry, and its implementation can be found in peft.tuners.lora.config 2024/07/24 14:26:03 - mmengine - INFO - InternVL_V1_5( (data_preprocessor): BaseDataPreprocessor() (model): InternVLChatModel( (vision_model): InternVisionModel( (embeddings): InternVisionEmbeddings( (patch_embedding): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14)) ) (encoder): InternVisionEncoder( (layers): ModuleList( (0-23): 24 x InternVisionEncoderLayer( (attn): InternAttention( (qkv): Linear(in_features=1024, out_features=3072, bias=True) (attn_drop): Dropout(p=0.0, inplace=False) (proj_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=1024, out_features=1024, bias=True) ) (mlp): InternMLP( (act): GELUActivation() (fc1): Linear(in_features=1024, out_features=4096, bias=True) (fc2): Linear(in_features=4096, out_features=1024, bias=True) ) (norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True) (norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True) (drop_path1): Identity() (drop_path2): Identity() ) ) ) ) (language_model): PeftModelForCausalLM( (base_model): LoraModel( (model): InternLM2ForCausalLM( (model): InternLM2Model( (tok_embeddings): Embedding(92553, 2048, padding_idx=2) (layers): ModuleList( (0-23): 24 x InternLM2DecoderLayer( (attention): InternLM2Attention( (wqkv): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=4096, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=4096, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (wo): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (rotary_emb): InternLM2DynamicNTKScalingRotaryEmbedding() ) (feed_forward): InternLM2MLP( (w1): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=8192, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=8192, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (w3): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=8192, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=8192, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (w2): lora.Linear( (base_layer): Linear4bit(in_features=8192, out_features=2048, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=8192, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (act_fn): SiLU() ) (attention_norm): InternLM2RMSNorm() (ffn_norm): InternLM2RMSNorm() ) ) (norm): InternLM2RMSNorm() ) (output): lora.Linear( (base_layer): Linear4bit(in_features=2048, out_features=92553, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=128, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=128, out_features=92553, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) ) ) ) (mlp1): Sequential( (0): LayerNorm((4096,), eps=1e-05, elementwise_affine=True) (1): Linear(in_features=4096, out_features=2048, bias=True) (2): GELU(approximate='none') (3): Linear(in_features=2048, out_features=2048, bias=True) ) ) ) 2024/07/24 14:26:03 - mmengine - INFO - InternVL_V1_5 construction is complete 2024/07/24 14:26:03 - mmengine - DEBUG - An `InternVL_V1_5` instance is built from registry, and its implementation can be found in xtuner.model.internvl 2024/07/24 14:26:03 - mmengine - DEBUG - Get class `DefaultOptimWrapperConstructor` from "optimizer wrapper constructor" registry in "mmengine" 2024/07/24 14:26:03 - mmengine - DEBUG - An `DefaultOptimWrapperConstructor` instance is built from registry, and its implementation can be found in mmengine.optim.optimizer.default_constructor 2024/07/24 14:26:03 - mmengine - DEBUG - An `AdamW` instance is built from registry, and its implementation can be found in torch.optim.adamw 2024/07/24 14:26:03 - mmengine - DEBUG - Get class `DeepSpeedOptimWrapper` from "optim_wrapper" registry in "mmengine" 2024/07/24 14:26:03 - mmengine - DEBUG - An `DeepSpeedOptimWrapper` instance is built from registry, and its implementation can be found in mmengine._strategy.deepspeed 2024/07/24 14:26:07 - mmengine - DEBUG - The `end` of is not set. Use the max epochs/iters of train loop as default. 2024/07/24 14:26:07 - mmengine - DEBUG - The `end` of is not set. Use the max epochs/iters of train loop as default. 2024/07/24 14:26:07 - mmengine - INFO - Num train samples 4806 2024/07/24 14:26:07 - mmengine - INFO - train example: 2024/07/24 14:26:08 - mmengine - INFO - <|im_start|> system You are an AI assistant whose name is InternLM (书生·浦语).<|im_end|><|im_start|>user 请从这张聊天截图中提取结构化信息<|im_end|><|im_start|> assistant { "dialog_name": "<对方正在输入...", "conversation": [ { "timestamp": "", "speaker": "<对方正在输入...", "content": "不是", "message_bbox": { "min_x": 917, "max_x": 989, "min_y": 253, "max_y": 289 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "<对方正在输入...", "content": "在淘宝里", "message_bbox": { "min_x": 839, "max_x": 987, "min_y": 370, "max_y": 404 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "<对方正在输入...", "content": "不能发微信", "message_bbox": { "min_x": 801, "max_x": 989, "min_y": 485, "max_y": 521 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "<对方正在输入...", "content": "两字", "message_bbox": { "min_x": 915, "max_x": 988, "min_y": 601, "max_y": 637 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "<对方正在输入...", "content": "微信", "message_bbox": { "min_x": 916, "max_x": 990, "min_y": 718, "max_y": 753 }, "image": "", "transfer": [], "file": [] }, { "timestamp": "", "speaker": "<对方正在输入...", "content": "①微信", "message_bbox": { "min_x": 845, "max_x": 988, "min_y": 833, "max_y": 869 }, "image": "", "transfer": [], "file": [] } ] }<|im_end|> 2024/07/24 14:26:08 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io 2024/07/24 14:26:08 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future. 2024/07/24 14:26:08 - mmengine - INFO - Checkpoints will be saved to /root/wangqun/work_dirs/internvl_ft_run_6_filter. 2024/07/24 14:26:32 - mmengine - INFO - Iter(train) [ 10/19224] lr: 3.1324e-07 eta: 12:46:46 time: 2.3944 data_time: 0.0099 memory: 19416 loss: 0.4263 2024/07/24 14:26:49 - mmengine - INFO - Iter(train) [ 20/19224] lr: 6.6106e-07 eta: 10:50:05 time: 1.6678 data_time: 0.0180 memory: 11980 loss: 0.4896 2024/07/24 14:27:06 - mmengine - INFO - Iter(train) [ 30/19224] lr: 1.0089e-06 eta: 10:11:17 time: 1.6705 data_time: 0.0107 memory: 11736 loss: 0.5741 2024/07/24 14:27:21 - mmengine - INFO - Iter(train) [ 40/19224] lr: 1.3567e-06 eta: 9:39:17 time: 1.5145 data_time: 0.0122 memory: 11368 loss: 0.6068 2024/07/24 14:27:37 - mmengine - INFO - Iter(train) [ 50/19224] lr: 1.7045e-06 eta: 9:25:24 time: 1.5992 data_time: 0.0103 memory: 11408 loss: 0.6258 2024/07/24 14:27:52 - mmengine - INFO - Iter(train) [ 60/19224] lr: 2.0524e-06 eta: 9:12:07 time: 1.5253 data_time: 0.0103 memory: 11085 loss: 0.7429 2024/07/24 14:28:06 - mmengine - INFO - Iter(train) [ 70/19224] lr: 2.4002e-06 eta: 8:58:39 time: 1.4397 data_time: 0.0100 memory: 10975 loss: 0.9743 2024/07/24 14:28:19 - mmengine - INFO - Iter(train) [ 80/19224] lr: 2.7480e-06 eta: 8:43:05 time: 1.3039 data_time: 0.0095 memory: 10614 loss: 0.9012 2024/07/24 14:28:31 - mmengine - INFO - Iter(train) [ 90/19224] lr: 3.0958e-06 eta: 8:25:46 time: 1.1587 data_time: 0.0091 memory: 10324 loss: 0.4682 2024/07/24 14:28:39 - mmengine - INFO - Iter(train) [ 100/19224] lr: 3.4436e-06 eta: 8:01:29 time: 0.8325 data_time: 0.0087 memory: 9704 loss: 0.6222 2024/07/24 14:29:02 - mmengine - INFO - Iter(train) [ 110/19224] lr: 3.7915e-06 eta: 8:23:27 time: 2.2774 data_time: 0.0105 memory: 15347 loss: 0.3598 2024/07/24 14:29:24 - mmengine - INFO - Iter(train) [ 120/19224] lr: 4.1393e-06 eta: 8:40:11 time: 2.2211 data_time: 0.0112 memory: 12196 loss: 0.3465 2024/07/24 14:29:46 - mmengine - INFO - Iter(train) [ 130/19224] lr: 4.4871e-06 eta: 8:53:24 time: 2.1849 data_time: 0.0107 memory: 12031 loss: 0.3585 2024/07/24 14:30:12 - mmengine - INFO - Iter(train) [ 140/19224] lr: 4.8349e-06 eta: 9:12:58 time: 2.5493 data_time: 0.0111 memory: 11573 loss: 0.3554 2024/07/24 14:30:33 - mmengine - INFO - Iter(train) [ 150/19224] lr: 5.1828e-06 eta: 9:21:28 time: 2.1532 data_time: 0.0099 memory: 11323 loss: 0.3964 2024/07/24 14:30:55 - mmengine - INFO - Iter(train) [ 160/19224] lr: 5.5306e-06 eta: 9:28:59 time: 2.1596 data_time: 0.0101 memory: 11076 loss: 0.3903 2024/07/24 14:31:15 - mmengine - INFO - Iter(train) [ 170/19224] lr: 5.8784e-06 eta: 9:33:04 time: 2.0261 data_time: 0.0097 memory: 10978 loss: 0.4011 2024/07/24 14:31:34 - mmengine - INFO - Iter(train) [ 180/19224] lr: 6.2262e-06 eta: 9:34:41 time: 1.9132 data_time: 0.0096 memory: 10500 loss: 0.3448 2024/07/24 14:31:51 - mmengine - INFO - Iter(train) [ 190/19224] lr: 6.5740e-06 eta: 9:32:14 time: 1.6818 data_time: 0.0093 memory: 10118 loss: 0.3491 2024/07/24 14:32:04 - mmengine - INFO - Iter(train) [ 200/19224] lr: 6.9219e-06 eta: 9:24:43 time: 1.3487 data_time: 0.0086 memory: 9651 loss: 0.4036 2024/07/24 14:32:40 - mmengine - INFO - Iter(train) [ 210/19224] lr: 7.2697e-06 eta: 9:50:37 time: 3.5165 data_time: 0.0102 memory: 17914 loss: 0.3835 2024/07/24 14:33:09 - mmengine - INFO - Iter(train) [ 220/19224] lr: 7.6175e-06 eta: 10:05:38 time: 2.9283 data_time: 0.0104 memory: 12222 loss: 0.2832 2024/07/24 14:33:37 - mmengine - INFO - Iter(train) [ 230/19224] lr: 7.9653e-06 eta: 10:17:53 time: 2.8252 data_time: 0.0107 memory: 11901 loss: 0.2752 2024/07/24 14:34:05 - mmengine - INFO - Iter(train) [ 240/19224] lr: 8.3132e-06 eta: 10:28:01 time: 2.7463 data_time: 0.0104 memory: 11499 loss: 0.3208 2024/07/24 14:34:31 - mmengine - INFO - Iter(train) [ 250/19224] lr: 8.6610e-06 eta: 10:35:30 time: 2.6018 data_time: 0.0107 memory: 11299 loss: 0.3103 2024/07/24 14:34:57 - mmengine - INFO - Iter(train) [ 260/19224] lr: 9.0088e-06 eta: 10:42:30 time: 2.6136 data_time: 0.0112 memory: 11299 loss: 0.3273 2024/07/24 14:35:21 - mmengine - INFO - Iter(train) [ 270/19224] lr: 9.3566e-06 eta: 10:46:20 time: 2.3894 data_time: 0.0107 memory: 10983 loss: 0.3744 2024/07/24 14:35:42 - mmengine - INFO - Iter(train) [ 280/19224] lr: 9.7045e-06 eta: 10:47:16 time: 2.1585 data_time: 0.0100 memory: 10687 loss: 0.3348 2024/07/24 14:36:00 - mmengine - INFO - Iter(train) [ 290/19224] lr: 1.0052e-05 eta: 10:44:11 time: 1.7988 data_time: 0.0102 memory: 10216 loss: 0.3753 2024/07/24 14:36:13 - mmengine - INFO - Iter(train) [ 300/19224] lr: 1.0400e-05 eta: 10:35:49 time: 1.2771 data_time: 0.0088 memory: 9704 loss: 0.4567 2024/07/24 14:36:49 - mmengine - INFO - Iter(train) [ 310/19224] lr: 1.0748e-05 eta: 10:51:09 time: 3.5580 data_time: 0.0104 memory: 15840 loss: 0.2912 2024/07/24 14:37:19 - mmengine - INFO - Iter(train) [ 320/19224] lr: 1.1096e-05 eta: 11:00:26 time: 3.0428 data_time: 0.0108 memory: 12001 loss: 0.2693 2024/07/24 14:37:48 - mmengine - INFO - Iter(train) [ 330/19224] lr: 1.1444e-05 eta: 11:07:47 time: 2.9029 data_time: 0.0107 memory: 11938 loss: 0.2948 2024/07/24 14:38:15 - mmengine - INFO - Iter(train) [ 340/19224] lr: 1.1791e-05 eta: 11:13:09 time: 2.7378 data_time: 0.0108 memory: 11600 loss: 0.3047 2024/07/24 14:38:42 - mmengine - INFO - Iter(train) [ 350/19224] lr: 1.2139e-05 eta: 11:17:39 time: 2.6800 data_time: 0.0107 memory: 11324 loss: 0.3160 2024/07/24 14:39:08 - mmengine - INFO - Iter(train) [ 360/19224] lr: 1.2487e-05 eta: 11:20:59 time: 2.5766 data_time: 0.0110 memory: 11152 loss: 0.2811 2024/07/24 14:39:33 - mmengine - INFO - Iter(train) [ 370/19224] lr: 1.2835e-05 eta: 11:23:13 time: 2.4718 data_time: 0.0113 memory: 11042 loss: 0.3384 2024/07/24 14:39:56 - mmengine - INFO - Iter(train) [ 380/19224] lr: 1.3183e-05 eta: 11:24:19 time: 2.3504 data_time: 0.0109 memory: 10924 loss: 0.4106 2024/07/24 14:40:17 - mmengine - INFO - Iter(train) [ 390/19224] lr: 1.3530e-05 eta: 11:23:13 time: 2.0881 data_time: 0.0099 memory: 10609 loss: 0.3305 2024/07/24 14:40:34 - mmengine - INFO - Iter(train) [ 400/19224] lr: 1.3878e-05 eta: 11:18:47 time: 1.6585 data_time: 0.0090 memory: 9980 loss: 0.3305 2024/07/24 14:41:11 - mmengine - INFO - Iter(train) [ 410/19224] lr: 1.4226e-05 eta: 11:30:17 time: 3.7131 data_time: 0.0103 memory: 16719 loss: 0.2640 2024/07/24 14:41:41 - mmengine - INFO - Iter(train) [ 420/19224] lr: 1.4574e-05 eta: 11:35:41 time: 2.9744 data_time: 0.0116 memory: 12130 loss: 0.3081 2024/07/24 14:42:10 - mmengine - INFO - Iter(train) [ 430/19224] lr: 1.4922e-05 eta: 11:40:32 time: 2.9366 data_time: 0.0105 memory: 11956 loss: 0.2817 2024/07/24 14:42:37 - mmengine - INFO - Iter(train) [ 440/19224] lr: 1.5270e-05 eta: 11:43:52 time: 2.7577 data_time: 0.0106 memory: 11504 loss: 0.2849 2024/07/24 14:43:04 - mmengine - INFO - Iter(train) [ 450/19224] lr: 1.5617e-05 eta: 11:46:07 time: 2.6254 data_time: 0.0105 memory: 11275 loss: 0.2791 2024/07/24 14:43:29 - mmengine - INFO - Iter(train) [ 460/19224] lr: 1.5965e-05 eta: 11:47:36 time: 2.5311 data_time: 0.0104 memory: 11178 loss: 0.3238 2024/07/24 14:43:54 - mmengine - INFO - Iter(train) [ 470/19224] lr: 1.6313e-05 eta: 11:48:45 time: 2.4914 data_time: 0.0104 memory: 11149 loss: 0.3433 2024/07/24 14:44:16 - mmengine - INFO - Iter(train) [ 480/19224] lr: 1.6661e-05 eta: 11:47:51 time: 2.1866 data_time: 0.0102 memory: 10717 loss: 0.4587 2024/07/24 14:44:35 - mmengine - INFO - Iter(train) [ 490/19224] lr: 1.7009e-05 eta: 11:45:07 time: 1.8980 data_time: 0.0099 memory: 10339 loss: 0.3986 2024/07/24 14:44:49 - mmengine - INFO - Iter(train) [ 500/19224] lr: 1.7357e-05 eta: 11:39:30 time: 1.4178 data_time: 0.0087 memory: 9859 loss: 0.3277 2024/07/24 14:45:27 - mmengine - INFO - Iter(train) [ 510/19224] lr: 1.7704e-05 eta: 11:48:53 time: 3.8359 data_time: 0.0103 memory: 15599 loss: 0.3380 2024/07/24 14:45:58 - mmengine - INFO - Iter(train) [ 520/19224] lr: 1.8052e-05 eta: 11:53:26 time: 3.0955 data_time: 0.0106 memory: 12349 loss: 0.2441 2024/07/24 14:46:27 - mmengine - INFO - Iter(train) [ 530/19224] lr: 1.8400e-05 eta: 11:56:38 time: 2.8980 data_time: 0.0103 memory: 11703 loss: 0.2439 2024/07/24 14:46:56 - mmengine - INFO - Iter(train) [ 540/19224] lr: 1.8748e-05 eta: 11:59:35 time: 2.8793 data_time: 0.0109 memory: 11564 loss: 0.2883 2024/07/24 14:47:23 - mmengine - INFO - Iter(train) [ 550/19224] lr: 1.9096e-05 eta: 12:01:28 time: 2.7101 data_time: 0.0109 memory: 11357 loss: 0.2712 2024/07/24 14:47:49 - mmengine - INFO - Iter(train) [ 560/19224] lr: 1.9443e-05 eta: 12:02:44 time: 2.6152 data_time: 0.0109 memory: 11336 loss: 0.3218 2024/07/24 14:48:15 - mmengine - INFO - Iter(train) [ 570/19224] lr: 1.9791e-05 eta: 12:03:24 time: 2.5192 data_time: 0.0106 memory: 11154 loss: 0.3871 2024/07/24 14:48:38 - mmengine - INFO - Iter(train) [ 580/19224] lr: 2.0000e-05 eta: 12:03:08 time: 2.3488 data_time: 0.0098 memory: 10895 loss: 0.3693 2024/07/24 14:48:58 - mmengine - INFO - Iter(train) [ 590/19224] lr: 2.0000e-05 eta: 12:00:52 time: 1.9700 data_time: 0.0095 memory: 10358 loss: 0.2722 2024/07/24 14:49:13 - mmengine - INFO - Iter(train) [ 600/19224] lr: 2.0000e-05 eta: 11:56:11 time: 1.4897 data_time: 0.0088 memory: 9948 loss: 0.3064 2024/07/24 14:49:46 - mmengine - INFO - Iter(train) [ 610/19224] lr: 2.0000e-05 eta: 12:01:09 time: 3.3606 data_time: 0.0107 memory: 13614 loss: 0.2538 2024/07/24 14:50:16 - mmengine - INFO - Iter(train) [ 620/19224] lr: 2.0000e-05 eta: 12:03:50 time: 2.9379 data_time: 0.0105 memory: 11873 loss: 0.2335 2024/07/24 14:50:44 - mmengine - INFO - Iter(train) [ 630/19224] lr: 2.0000e-05 eta: 12:05:50 time: 2.8213 data_time: 0.0110 memory: 11526 loss: 0.2563 2024/07/24 14:51:12 - mmengine - INFO - Iter(train) [ 640/19224] lr: 1.9999e-05 eta: 12:07:33 time: 2.7759 data_time: 0.0104 memory: 11389 loss: 0.2909 2024/07/24 14:51:38 - mmengine - INFO - Iter(train) [ 650/19224] lr: 1.9999e-05 eta: 12:08:43 time: 2.6784 data_time: 0.0112 memory: 11332 loss: 0.2862 2024/07/24 14:52:04 - mmengine - INFO - Iter(train) [ 660/19224] lr: 1.9999e-05 eta: 12:09:18 time: 2.5601 data_time: 0.0103 memory: 11185 loss: 0.2547 2024/07/24 14:52:28 - mmengine - INFO - Iter(train) [ 670/19224] lr: 1.9999e-05 eta: 12:09:03 time: 2.3901 data_time: 0.0106 memory: 11091 loss: 0.3279 2024/07/24 14:52:49 - mmengine - INFO - Iter(train) [ 680/19224] lr: 1.9998e-05 eta: 12:07:24 time: 2.0788 data_time: 0.0116 memory: 10591 loss: 0.3276 2024/07/24 14:53:08 - mmengine - INFO - Iter(train) [ 690/19224] lr: 1.9998e-05 eta: 12:04:55 time: 1.8877 data_time: 0.0095 memory: 10245 loss: 0.2537 2024/07/24 14:53:22 - mmengine - INFO - Iter(train) [ 700/19224] lr: 1.9998e-05 eta: 12:00:33 time: 1.4456 data_time: 0.0091 memory: 9989 loss: 0.2935 2024/07/24 14:53:56 - mmengine - INFO - Iter(train) [ 710/19224] lr: 1.9997e-05 eta: 12:04:54 time: 3.4253 data_time: 0.0100 memory: 14357 loss: 0.2196 2024/07/24 14:54:26 - mmengine - INFO - Iter(train) [ 720/19224] lr: 1.9997e-05 eta: 12:07:23 time: 3.0188 data_time: 0.0106 memory: 12068 loss: 0.2324 2024/07/24 14:54:56 - mmengine - INFO - Iter(train) [ 730/19224] lr: 1.9997e-05 eta: 12:09:29 time: 2.9503 data_time: 0.0106 memory: 12005 loss: 0.2348 2024/07/24 14:55:23 - mmengine - INFO - Iter(train) [ 740/19224] lr: 1.9996e-05 eta: 12:10:43 time: 2.7557 data_time: 0.0107 memory: 11573 loss: 0.3293 2024/07/24 14:55:50 - mmengine - INFO - Iter(train) [ 750/19224] lr: 1.9996e-05 eta: 12:11:25 time: 2.6389 data_time: 0.0105 memory: 11282 loss: 0.3002 2024/07/24 14:56:16 - mmengine - INFO - Iter(train) [ 760/19224] lr: 1.9995e-05 eta: 12:12:01 time: 2.6205 data_time: 0.0106 memory: 11211 loss: 0.2534 2024/07/24 14:56:41 - mmengine - INFO - Iter(train) [ 770/19224] lr: 1.9995e-05 eta: 12:12:03 time: 2.4878 data_time: 0.0103 memory: 11044 loss: 0.2944 2024/07/24 14:57:05 - mmengine - INFO - Iter(train) [ 780/19224] lr: 1.9994e-05 eta: 12:11:35 time: 2.3631 data_time: 0.0103 memory: 10963 loss: 0.2925 2024/07/24 14:57:23 - mmengine - INFO - Iter(train) [ 790/19224] lr: 1.9994e-05 eta: 12:08:55 time: 1.7954 data_time: 0.0096 memory: 10239 loss: 0.2568 2024/07/24 14:57:37 - mmengine - INFO - Iter(train) [ 800/19224] lr: 1.9993e-05 eta: 12:05:04 time: 1.4716 data_time: 0.0087 memory: 9758 loss: 0.2739 2024/07/24 14:58:13 - mmengine - INFO - Iter(train) [ 810/19224] lr: 1.9992e-05 eta: 12:09:09 time: 3.5451 data_time: 0.0103 memory: 15692 loss: 0.2415 2024/07/24 14:58:46 - mmengine - INFO - Iter(train) [ 820/19224] lr: 1.9992e-05 eta: 12:12:13 time: 3.3022 data_time: 0.0107 memory: 12194 loss: 0.2613 2024/07/24 14:59:16 - mmengine - INFO - Iter(train) [ 830/19224] lr: 1.9991e-05 eta: 12:14:15 time: 3.0444 data_time: 0.0105 memory: 12663 loss: 0.2188 2024/07/24 14:59:45 - mmengine - INFO - Iter(train) [ 840/19224] lr: 1.9990e-05 eta: 12:15:41 time: 2.8978 data_time: 0.0119 memory: 11676 loss: 0.2296 2024/07/24 15:00:15 - mmengine - INFO - Iter(train) [ 850/19224] lr: 1.9989e-05 eta: 12:17:32 time: 3.0229 data_time: 0.0110 memory: 11461 loss: 0.2464 2024/07/24 15:00:42 - mmengine - INFO - Iter(train) [ 860/19224] lr: 1.9989e-05 eta: 12:18:08 time: 2.6893 data_time: 0.0106 memory: 11332 loss: 0.2431 2024/07/24 15:01:08 - mmengine - INFO - Iter(train) [ 870/19224] lr: 1.9988e-05 eta: 12:18:14 time: 2.5569 data_time: 0.0109 memory: 11184 loss: 0.2594 2024/07/24 15:01:32 - mmengine - INFO - Iter(train) [ 880/19224] lr: 1.9987e-05 eta: 12:17:59 time: 2.4562 data_time: 0.0104 memory: 10995 loss: 0.3077 2024/07/24 15:01:52 - mmengine - INFO - Iter(train) [ 890/19224] lr: 1.9986e-05 eta: 12:15:59 time: 1.9496 data_time: 0.0096 memory: 10501 loss: 0.6502 2024/07/24 15:02:07 - mmengine - INFO - Iter(train) [ 900/19224] lr: 1.9985e-05 eta: 12:12:34 time: 1.5201 data_time: 0.0087 memory: 9892 loss: 0.3012 2024/07/24 15:02:42 - mmengine - INFO - Iter(train) [ 910/19224] lr: 1.9984e-05 eta: 12:15:53 time: 3.5051 data_time: 0.0103 memory: 14917 loss: 0.2691 2024/07/24 15:03:14 - mmengine - INFO - Iter(train) [ 920/19224] lr: 1.9983e-05 eta: 12:17:53 time: 3.1373 data_time: 0.0117 memory: 12199 loss: 0.2301 2024/07/24 15:03:43 - mmengine - INFO - Iter(train) [ 930/19224] lr: 1.9982e-05 eta: 12:19:16 time: 2.9626 data_time: 0.0107 memory: 11915 loss: 0.2432 2024/07/24 15:04:11 - mmengine - INFO - Iter(train) [ 940/19224] lr: 1.9981e-05 eta: 12:20:12 time: 2.8350 data_time: 0.0106 memory: 11528 loss: 0.2904 2024/07/24 15:04:39 - mmengine - INFO - Iter(train) [ 950/19224] lr: 1.9980e-05 eta: 12:20:44 time: 2.7248 data_time: 0.0112 memory: 11352 loss: 0.3088 2024/07/24 15:05:05 - mmengine - INFO - Iter(train) [ 960/19224] lr: 1.9979e-05 eta: 12:20:50 time: 2.5924 data_time: 0.0111 memory: 11189 loss: 0.2757 2024/07/24 15:05:30 - mmengine - INFO - Iter(train) [ 970/19224] lr: 1.9978e-05 eta: 12:20:43 time: 2.5220 data_time: 0.0108 memory: 11178 loss: 0.2776 2024/07/24 15:05:52 - mmengine - INFO - Iter(train) [ 980/19224] lr: 1.9977e-05 eta: 12:19:37 time: 2.2130 data_time: 0.0109 memory: 10726 loss: 0.2897 2024/07/24 15:06:11 - mmengine - INFO - Iter(train) [ 990/19224] lr: 1.9976e-05 eta: 12:17:41 time: 1.9337 data_time: 0.0096 memory: 10426 loss: 0.2820 2024/07/24 15:06:25 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 15:06:25 - mmengine - INFO - Iter(train) [ 1000/19224] lr: 1.9975e-05 eta: 12:13:57 time: 1.3321 data_time: 0.0087 memory: 10049 loss: 0.2568 2024/07/24 15:06:25 - mmengine - INFO - Saving checkpoint at 1000 iterations 2024/07/24 15:07:07 - mmengine - INFO - Iter(train) [ 1010/19224] lr: 1.9973e-05 eta: 12:18:57 time: 4.2125 data_time: 0.2069 memory: 18443 loss: 0.2569 2024/07/24 15:07:38 - mmengine - INFO - Iter(train) [ 1020/19224] lr: 1.9972e-05 eta: 12:20:38 time: 3.1363 data_time: 0.0103 memory: 12354 loss: 0.2173 2024/07/24 15:08:08 - mmengine - INFO - Iter(train) [ 1030/19224] lr: 1.9971e-05 eta: 12:21:48 time: 2.9808 data_time: 0.0115 memory: 11919 loss: 0.2415 2024/07/24 15:08:36 - mmengine - INFO - Iter(train) [ 1040/19224] lr: 1.9970e-05 eta: 12:22:26 time: 2.7994 data_time: 0.0106 memory: 11513 loss: 0.2494 2024/07/24 15:09:03 - mmengine - INFO - Iter(train) [ 1050/19224] lr: 1.9968e-05 eta: 12:22:50 time: 2.7334 data_time: 0.0105 memory: 11408 loss: 0.2556 2024/07/24 15:09:31 - mmengine - INFO - Iter(train) [ 1060/19224] lr: 1.9967e-05 eta: 12:23:14 time: 2.7315 data_time: 0.0113 memory: 11281 loss: 0.2385 2024/07/24 15:09:57 - mmengine - INFO - Iter(train) [ 1070/19224] lr: 1.9966e-05 eta: 12:23:13 time: 2.5952 data_time: 0.0103 memory: 11171 loss: 0.2540 2024/07/24 15:10:21 - mmengine - INFO - Iter(train) [ 1080/19224] lr: 1.9964e-05 eta: 12:22:43 time: 2.4241 data_time: 0.0104 memory: 11021 loss: 0.3347 2024/07/24 15:10:43 - mmengine - INFO - Iter(train) [ 1090/19224] lr: 1.9963e-05 eta: 12:21:45 time: 2.2538 data_time: 0.0103 memory: 10808 loss: 0.3045 2024/07/24 15:11:00 - mmengine - INFO - Iter(train) [ 1100/19224] lr: 1.9961e-05 eta: 12:19:12 time: 1.6773 data_time: 0.0092 memory: 10216 loss: 0.4068 2024/07/24 15:11:32 - mmengine - INFO - Iter(train) [ 1110/19224] lr: 1.9960e-05 eta: 12:20:41 time: 3.1429 data_time: 0.0110 memory: 12843 loss: 0.2506 2024/07/24 15:12:02 - mmengine - INFO - Iter(train) [ 1120/19224] lr: 1.9958e-05 eta: 12:21:44 time: 2.9967 data_time: 0.0107 memory: 11905 loss: 0.2685 2024/07/24 15:12:31 - mmengine - INFO - Iter(train) [ 1130/19224] lr: 1.9957e-05 eta: 12:22:30 time: 2.8990 data_time: 0.0111 memory: 11691 loss: 0.2153 2024/07/24 15:12:59 - mmengine - INFO - Iter(train) [ 1140/19224] lr: 1.9955e-05 eta: 12:23:12 time: 2.8773 data_time: 0.0104 memory: 11528 loss: 0.2087 2024/07/24 15:13:27 - mmengine - INFO - Iter(train) [ 1150/19224] lr: 1.9953e-05 eta: 12:23:29 time: 2.7317 data_time: 0.0108 memory: 11319 loss: 0.2960 2024/07/24 15:13:52 - mmengine - INFO - Iter(train) [ 1160/19224] lr: 1.9952e-05 eta: 12:23:22 time: 2.5833 data_time: 0.0105 memory: 11202 loss: 0.2331 2024/07/24 15:14:17 - mmengine - INFO - Iter(train) [ 1170/19224] lr: 1.9950e-05 eta: 12:23:01 time: 2.4927 data_time: 0.0101 memory: 11089 loss: 0.2994 2024/07/24 15:14:40 - mmengine - INFO - Iter(train) [ 1180/19224] lr: 1.9948e-05 eta: 12:22:00 time: 2.2345 data_time: 0.0110 memory: 10887 loss: 0.4146 2024/07/24 15:14:59 - mmengine - INFO - Iter(train) [ 1190/19224] lr: 1.9947e-05 eta: 12:20:12 time: 1.9159 data_time: 0.0099 memory: 10313 loss: 0.2615 2024/07/24 15:15:14 - mmengine - INFO - Iter(train) [ 1200/19224] lr: 1.9945e-05 eta: 12:17:31 time: 1.5517 data_time: 0.0092 memory: 9972 loss: 0.2674 2024/07/24 15:15:50 - mmengine - INFO - Iter(train) [ 1210/19224] lr: 1.9943e-05 eta: 12:19:45 time: 3.5245 data_time: 0.0102 memory: 15393 loss: 0.2433 2024/07/24 15:16:20 - mmengine - INFO - Iter(train) [ 1220/19224] lr: 1.9941e-05 eta: 12:20:46 time: 3.0432 data_time: 0.0110 memory: 12071 loss: 0.2234 2024/07/24 15:16:49 - mmengine - INFO - Iter(train) [ 1230/19224] lr: 1.9940e-05 eta: 12:21:27 time: 2.9196 data_time: 0.0104 memory: 11758 loss: 0.2235 2024/07/24 15:17:17 - mmengine - INFO - Iter(train) [ 1240/19224] lr: 1.9938e-05 eta: 12:21:53 time: 2.8209 data_time: 0.0108 memory: 11683 loss: 0.2342 2024/07/24 15:17:45 - mmengine - INFO - Iter(train) [ 1250/19224] lr: 1.9936e-05 eta: 12:22:15 time: 2.7981 data_time: 0.0110 memory: 11361 loss: 0.2328 2024/07/24 15:18:12 - mmengine - INFO - Iter(train) [ 1260/19224] lr: 1.9934e-05 eta: 12:22:11 time: 2.6257 data_time: 0.0103 memory: 11281 loss: 0.2386 2024/07/24 15:18:38 - mmengine - INFO - Iter(train) [ 1270/19224] lr: 1.9932e-05 eta: 12:22:09 time: 2.6367 data_time: 0.0107 memory: 11123 loss: 0.2453 2024/07/24 15:19:02 - mmengine - INFO - Iter(train) [ 1280/19224] lr: 1.9930e-05 eta: 12:21:31 time: 2.3856 data_time: 0.0110 memory: 10964 loss: 0.2659 2024/07/24 15:19:22 - mmengine - INFO - Iter(train) [ 1290/19224] lr: 1.9928e-05 eta: 12:20:03 time: 2.0303 data_time: 0.0092 memory: 10402 loss: 0.2831 2024/07/24 15:19:39 - mmengine - INFO - Iter(train) [ 1300/19224] lr: 1.9926e-05 eta: 12:17:54 time: 1.7143 data_time: 0.0091 memory: 10044 loss: 0.3535 2024/07/24 15:20:12 - mmengine - INFO - Iter(train) [ 1310/19224] lr: 1.9924e-05 eta: 12:19:24 time: 3.3095 data_time: 0.0105 memory: 13146 loss: 0.2512 2024/07/24 15:20:44 - mmengine - INFO - Iter(train) [ 1320/19224] lr: 1.9922e-05 eta: 12:20:25 time: 3.1118 data_time: 0.0108 memory: 12097 loss: 0.2065 2024/07/24 15:21:12 - mmengine - INFO - Iter(train) [ 1330/19224] lr: 1.9920e-05 eta: 12:20:55 time: 2.8861 data_time: 0.0105 memory: 11654 loss: 0.2423 2024/07/24 15:21:42 - mmengine - INFO - Iter(train) [ 1340/19224] lr: 1.9917e-05 eta: 12:21:27 time: 2.9094 data_time: 0.0104 memory: 11593 loss: 0.2701 2024/07/24 15:22:10 - mmengine - INFO - Iter(train) [ 1350/19224] lr: 1.9915e-05 eta: 12:21:49 time: 2.8432 data_time: 0.0105 memory: 11389 loss: 0.2532 2024/07/24 15:22:37 - mmengine - INFO - Iter(train) [ 1360/19224] lr: 1.9913e-05 eta: 12:21:48 time: 2.6749 data_time: 0.0107 memory: 11212 loss: 0.2501 2024/07/24 15:23:03 - mmengine - INFO - Iter(train) [ 1370/19224] lr: 1.9911e-05 eta: 12:21:36 time: 2.5903 data_time: 0.0116 memory: 11070 loss: 0.2629 2024/07/24 15:23:27 - mmengine - INFO - Iter(train) [ 1380/19224] lr: 1.9909e-05 eta: 12:21:01 time: 2.4105 data_time: 0.0102 memory: 10862 loss: 0.3174 2024/07/24 15:23:48 - mmengine - INFO - Iter(train) [ 1390/19224] lr: 1.9906e-05 eta: 12:19:54 time: 2.1686 data_time: 0.0107 memory: 10481 loss: 0.2262 2024/07/24 15:24:03 - mmengine - INFO - Iter(train) [ 1400/19224] lr: 1.9904e-05 eta: 12:17:24 time: 1.5072 data_time: 0.0098 memory: 9913 loss: 0.3289 2024/07/24 15:24:39 - mmengine - INFO - Iter(train) [ 1410/19224] lr: 1.9902e-05 eta: 12:19:21 time: 3.6005 data_time: 0.0108 memory: 14660 loss: 0.2499 2024/07/24 15:25:12 - mmengine - INFO - Iter(train) [ 1420/19224] lr: 1.9899e-05 eta: 12:20:36 time: 3.2912 data_time: 0.0112 memory: 12187 loss: 0.2375 2024/07/24 15:25:42 - mmengine - INFO - Iter(train) [ 1430/19224] lr: 1.9897e-05 eta: 12:21:04 time: 2.9173 data_time: 0.0106 memory: 11686 loss: 0.2472 2024/07/24 15:26:10 - mmengine - INFO - Iter(train) [ 1440/19224] lr: 1.9894e-05 eta: 12:21:16 time: 2.7961 data_time: 0.0111 memory: 11360 loss: 0.2190 2024/07/24 15:26:35 - mmengine - INFO - Iter(train) [ 1450/19224] lr: 1.9892e-05 eta: 12:20:59 time: 2.5721 data_time: 0.0112 memory: 11195 loss: 0.3648 2024/07/24 15:27:00 - mmengine - INFO - Iter(train) [ 1460/19224] lr: 1.9890e-05 eta: 12:20:33 time: 2.4883 data_time: 0.0117 memory: 11021 loss: 0.3325 2024/07/24 15:27:23 - mmengine - INFO - Iter(train) [ 1470/19224] lr: 1.9887e-05 eta: 12:19:38 time: 2.2572 data_time: 0.0102 memory: 10825 loss: 0.3725 2024/07/24 15:27:42 - mmengine - INFO - Iter(train) [ 1480/19224] lr: 1.9885e-05 eta: 12:18:07 time: 1.9491 data_time: 0.0092 memory: 10358 loss: 0.2424 2024/07/24 15:28:00 - mmengine - INFO - Iter(train) [ 1490/19224] lr: 1.9882e-05 eta: 12:16:14 time: 1.7516 data_time: 0.0097 memory: 10089 loss: 0.2746 2024/07/24 15:28:14 - mmengine - INFO - Iter(train) [ 1500/19224] lr: 1.9879e-05 eta: 12:13:40 time: 1.3983 data_time: 0.0090 memory: 9794 loss: 0.3198 2024/07/24 15:28:47 - mmengine - INFO - Iter(train) [ 1510/19224] lr: 1.9877e-05 eta: 12:14:56 time: 3.3445 data_time: 0.0102 memory: 14087 loss: 0.2312 2024/07/24 15:29:18 - mmengine - INFO - Iter(train) [ 1520/19224] lr: 1.9874e-05 eta: 12:15:36 time: 3.0494 data_time: 0.0111 memory: 11820 loss: 0.2241 2024/07/24 15:29:47 - mmengine - INFO - Iter(train) [ 1530/19224] lr: 1.9871e-05 eta: 12:16:05 time: 2.9574 data_time: 0.0105 memory: 11652 loss: 0.2440 2024/07/24 15:30:19 - mmengine - INFO - Iter(train) [ 1540/19224] lr: 1.9869e-05 eta: 12:16:59 time: 3.1801 data_time: 0.0103 memory: 11408 loss: 0.2571 2024/07/24 15:30:47 - mmengine - INFO - Iter(train) [ 1550/19224] lr: 1.9866e-05 eta: 12:17:12 time: 2.8356 data_time: 0.0116 memory: 11287 loss: 0.2659 2024/07/24 15:31:13 - mmengine - INFO - Iter(train) [ 1560/19224] lr: 1.9863e-05 eta: 12:16:54 time: 2.5631 data_time: 0.0110 memory: 11157 loss: 0.2635 2024/07/24 15:31:37 - mmengine - INFO - Iter(train) [ 1570/19224] lr: 1.9860e-05 eta: 12:16:19 time: 2.4221 data_time: 0.0107 memory: 10880 loss: 0.2554 2024/07/24 15:31:57 - mmengine - INFO - Iter(train) [ 1580/19224] lr: 1.9858e-05 eta: 12:14:59 time: 2.0106 data_time: 0.0093 memory: 10502 loss: 0.2603 2024/07/24 15:32:16 - mmengine - INFO - Iter(train) [ 1590/19224] lr: 1.9855e-05 eta: 12:13:21 time: 1.8412 data_time: 0.0095 memory: 10261 loss: 0.2777 2024/07/24 15:32:28 - mmengine - INFO - Iter(train) [ 1600/19224] lr: 1.9852e-05 eta: 12:10:38 time: 1.2410 data_time: 0.0085 memory: 9683 loss: 0.3124 2024/07/24 15:33:04 - mmengine - INFO - Iter(train) [ 1610/19224] lr: 1.9849e-05 eta: 12:12:18 time: 3.6239 data_time: 0.0106 memory: 18895 loss: 0.2118 2024/07/24 15:33:35 - mmengine - INFO - Iter(train) [ 1620/19224] lr: 1.9846e-05 eta: 12:12:54 time: 3.0564 data_time: 0.0106 memory: 12282 loss: 0.2007 2024/07/24 15:34:03 - mmengine - INFO - Iter(train) [ 1630/19224] lr: 1.9843e-05 eta: 12:13:07 time: 2.8506 data_time: 0.0105 memory: 11613 loss: 0.2025 2024/07/24 15:34:31 - mmengine - INFO - Iter(train) [ 1640/19224] lr: 1.9840e-05 eta: 12:13:05 time: 2.7114 data_time: 0.0107 memory: 11444 loss: 0.2264 2024/07/24 15:34:57 - mmengine - INFO - Iter(train) [ 1650/19224] lr: 1.9837e-05 eta: 12:12:59 time: 2.6792 data_time: 0.0120 memory: 11201 loss: 0.2410 2024/07/24 15:35:24 - mmengine - INFO - Iter(train) [ 1660/19224] lr: 1.9834e-05 eta: 12:12:45 time: 2.6153 data_time: 0.0114 memory: 11128 loss: 0.3300 2024/07/24 15:35:47 - mmengine - INFO - Iter(train) [ 1670/19224] lr: 1.9831e-05 eta: 12:12:07 time: 2.3802 data_time: 0.0104 memory: 10931 loss: 0.2789 2024/07/24 15:36:08 - mmengine - INFO - Iter(train) [ 1680/19224] lr: 1.9828e-05 eta: 12:11:02 time: 2.1105 data_time: 0.0107 memory: 10621 loss: 0.2771 2024/07/24 15:36:26 - mmengine - INFO - Iter(train) [ 1690/19224] lr: 1.9825e-05 eta: 12:09:20 time: 1.7593 data_time: 0.0092 memory: 10166 loss: 0.2618 2024/07/24 15:36:39 - mmengine - INFO - Iter(train) [ 1700/19224] lr: 1.9822e-05 eta: 12:06:49 time: 1.2770 data_time: 0.0087 memory: 9413 loss: 0.3348 2024/07/24 15:37:18 - mmengine - INFO - Iter(train) [ 1710/19224] lr: 1.9818e-05 eta: 12:08:47 time: 3.8809 data_time: 0.0103 memory: 18895 loss: 0.2425 2024/07/24 15:37:49 - mmengine - INFO - Iter(train) [ 1720/19224] lr: 1.9815e-05 eta: 12:09:28 time: 3.1426 data_time: 0.0111 memory: 12215 loss: 0.2097 2024/07/24 15:38:19 - mmengine - INFO - Iter(train) [ 1730/19224] lr: 1.9812e-05 eta: 12:09:49 time: 2.9554 data_time: 0.0112 memory: 11733 loss: 0.2198 2024/07/24 15:38:47 - mmengine - INFO - Iter(train) [ 1740/19224] lr: 1.9809e-05 eta: 12:09:59 time: 2.8553 data_time: 0.0107 memory: 11663 loss: 0.2144 2024/07/24 15:39:15 - mmengine - INFO - Iter(train) [ 1750/19224] lr: 1.9805e-05 eta: 12:09:58 time: 2.7466 data_time: 0.0112 memory: 11286 loss: 0.2451 2024/07/24 15:39:41 - mmengine - INFO - Iter(train) [ 1760/19224] lr: 1.9802e-05 eta: 12:09:45 time: 2.6303 data_time: 0.0113 memory: 11176 loss: 0.2722 2024/07/24 15:40:05 - mmengine - INFO - Iter(train) [ 1770/19224] lr: 1.9799e-05 eta: 12:09:10 time: 2.4009 data_time: 0.0107 memory: 11011 loss: 0.3020 2024/07/24 15:40:26 - mmengine - INFO - Iter(train) [ 1780/19224] lr: 1.9795e-05 eta: 12:08:06 time: 2.1119 data_time: 0.0106 memory: 10502 loss: 0.3238 2024/07/24 15:40:45 - mmengine - INFO - Iter(train) [ 1790/19224] lr: 1.9792e-05 eta: 12:06:41 time: 1.8855 data_time: 0.0096 memory: 10174 loss: 0.2580 2024/07/24 15:40:57 - mmengine - INFO - Iter(train) [ 1800/19224] lr: 1.9788e-05 eta: 12:04:11 time: 1.2158 data_time: 0.0087 memory: 9411 loss: 0.2657 2024/07/24 15:41:33 - mmengine - INFO - Iter(train) [ 1810/19224] lr: 1.9785e-05 eta: 12:05:29 time: 3.5648 data_time: 0.0104 memory: 14300 loss: 0.2186 2024/07/24 15:42:03 - mmengine - INFO - Iter(train) [ 1820/19224] lr: 1.9782e-05 eta: 12:05:54 time: 3.0201 data_time: 0.0105 memory: 12210 loss: 0.2294 2024/07/24 15:42:32 - mmengine - INFO - Iter(train) [ 1830/19224] lr: 1.9778e-05 eta: 12:06:10 time: 2.9311 data_time: 0.0112 memory: 11751 loss: 0.2102 2024/07/24 15:43:01 - mmengine - INFO - Iter(train) [ 1840/19224] lr: 1.9774e-05 eta: 12:06:19 time: 2.8687 data_time: 0.0112 memory: 11528 loss: 0.2516 2024/07/24 15:43:28 - mmengine - INFO - Iter(train) [ 1850/19224] lr: 1.9771e-05 eta: 12:06:10 time: 2.6804 data_time: 0.0113 memory: 11360 loss: 0.2551 2024/07/24 15:43:53 - mmengine - INFO - Iter(train) [ 1860/19224] lr: 1.9767e-05 eta: 12:05:50 time: 2.5557 data_time: 0.0107 memory: 11141 loss: 0.2817 2024/07/24 15:44:18 - mmengine - INFO - Iter(train) [ 1870/19224] lr: 1.9764e-05 eta: 12:05:20 time: 2.4621 data_time: 0.0110 memory: 11040 loss: 0.3127 2024/07/24 15:44:40 - mmengine - INFO - Iter(train) [ 1880/19224] lr: 1.9760e-05 eta: 12:04:24 time: 2.1702 data_time: 0.0104 memory: 10646 loss: 0.2771 2024/07/24 15:44:58 - mmengine - INFO - Iter(train) [ 1890/19224] lr: 1.9756e-05 eta: 12:03:02 time: 1.8791 data_time: 0.0104 memory: 10261 loss: 0.2685 2024/07/24 15:45:12 - mmengine - INFO - Iter(train) [ 1900/19224] lr: 1.9753e-05 eta: 12:00:57 time: 1.4116 data_time: 0.0098 memory: 9935 loss: 0.3806 2024/07/24 15:45:46 - mmengine - INFO - Iter(train) [ 1910/19224] lr: 1.9749e-05 eta: 12:01:49 time: 3.3466 data_time: 0.0110 memory: 13416 loss: 0.2199 2024/07/24 15:46:17 - mmengine - INFO - Iter(train) [ 1920/19224] lr: 1.9745e-05 eta: 12:02:15 time: 3.0618 data_time: 0.0114 memory: 12061 loss: 0.1902 2024/07/24 15:46:46 - mmengine - INFO - Iter(train) [ 1930/19224] lr: 1.9741e-05 eta: 12:02:27 time: 2.9241 data_time: 0.0116 memory: 11710 loss: 0.2861 2024/07/24 15:47:14 - mmengine - INFO - Iter(train) [ 1940/19224] lr: 1.9738e-05 eta: 12:02:32 time: 2.8393 data_time: 0.0111 memory: 11373 loss: 0.2528 2024/07/24 15:47:42 - mmengine - INFO - Iter(train) [ 1950/19224] lr: 1.9734e-05 eta: 12:02:29 time: 2.7650 data_time: 0.0106 memory: 11557 loss: 0.2599 2024/07/24 15:48:08 - mmengine - INFO - Iter(train) [ 1960/19224] lr: 1.9730e-05 eta: 12:02:12 time: 2.5967 data_time: 0.0105 memory: 11165 loss: 0.2833 2024/07/24 15:48:33 - mmengine - INFO - Iter(train) [ 1970/19224] lr: 1.9726e-05 eta: 12:01:49 time: 2.5299 data_time: 0.0108 memory: 11041 loss: 0.3467 2024/07/24 15:48:57 - mmengine - INFO - Iter(train) [ 1980/19224] lr: 1.9722e-05 eta: 12:01:11 time: 2.3671 data_time: 0.0109 memory: 10913 loss: 0.3001 2024/07/24 15:49:17 - mmengine - INFO - Iter(train) [ 1990/19224] lr: 1.9718e-05 eta: 12:00:00 time: 1.9843 data_time: 0.0098 memory: 10523 loss: 0.2503 2024/07/24 15:49:31 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 15:49:31 - mmengine - INFO - Iter(train) [ 2000/19224] lr: 1.9714e-05 eta: 11:58:07 time: 1.4834 data_time: 0.0092 memory: 9998 loss: 0.2819 2024/07/24 15:49:31 - mmengine - INFO - Saving checkpoint at 2000 iterations 2024/07/24 15:50:08 - mmengine - INFO - Iter(train) [ 2010/19224] lr: 1.9710e-05 eta: 11:59:22 time: 3.6616 data_time: 0.1963 memory: 14435 loss: 0.2201 2024/07/24 15:50:39 - mmengine - INFO - Iter(train) [ 2020/19224] lr: 1.9706e-05 eta: 11:59:42 time: 3.0436 data_time: 0.0115 memory: 12050 loss: 0.2108 2024/07/24 15:51:08 - mmengine - INFO - Iter(train) [ 2030/19224] lr: 1.9702e-05 eta: 11:59:55 time: 2.9609 data_time: 0.0105 memory: 11600 loss: 0.2188 2024/07/24 15:51:36 - mmengine - INFO - Iter(train) [ 2040/19224] lr: 1.9698e-05 eta: 11:59:56 time: 2.8211 data_time: 0.0104 memory: 11464 loss: 0.2282 2024/07/24 15:52:04 - mmengine - INFO - Iter(train) [ 2050/19224] lr: 1.9694e-05 eta: 11:59:52 time: 2.7645 data_time: 0.0102 memory: 11365 loss: 0.2566 2024/07/24 15:52:30 - mmengine - INFO - Iter(train) [ 2060/19224] lr: 1.9690e-05 eta: 11:59:37 time: 2.6348 data_time: 0.0102 memory: 11218 loss: 0.3007 2024/07/24 15:52:55 - mmengine - INFO - Iter(train) [ 2070/19224] lr: 1.9685e-05 eta: 11:59:12 time: 2.5127 data_time: 0.0107 memory: 11060 loss: 0.2856 2024/07/24 15:53:19 - mmengine - INFO - Iter(train) [ 2080/19224] lr: 1.9681e-05 eta: 11:58:34 time: 2.3702 data_time: 0.0104 memory: 10769 loss: 0.3286 2024/07/24 15:53:39 - mmengine - INFO - Iter(train) [ 2090/19224] lr: 1.9677e-05 eta: 11:57:27 time: 1.9961 data_time: 0.0104 memory: 10602 loss: 0.3508 2024/07/24 15:53:52 - mmengine - INFO - Iter(train) [ 2100/19224] lr: 1.9673e-05 eta: 11:55:25 time: 1.3269 data_time: 0.0087 memory: 9886 loss: 0.3117 2024/07/24 15:54:27 - mmengine - INFO - Iter(train) [ 2110/19224] lr: 1.9668e-05 eta: 11:56:18 time: 3.4755 data_time: 0.0104 memory: 16069 loss: 0.2483 2024/07/24 15:54:58 - mmengine - INFO - Iter(train) [ 2120/19224] lr: 1.9664e-05 eta: 11:56:36 time: 3.0381 data_time: 0.0115 memory: 11989 loss: 0.2158 2024/07/24 15:55:28 - mmengine - INFO - Iter(train) [ 2130/19224] lr: 1.9660e-05 eta: 11:56:50 time: 3.0078 data_time: 0.0107 memory: 11767 loss: 0.2354 2024/07/24 15:55:56 - mmengine - INFO - Iter(train) [ 2140/19224] lr: 1.9655e-05 eta: 11:56:50 time: 2.8216 data_time: 0.0107 memory: 11542 loss: 0.2426 2024/07/24 15:56:23 - mmengine - INFO - Iter(train) [ 2150/19224] lr: 1.9651e-05 eta: 11:56:40 time: 2.7139 data_time: 0.0109 memory: 11319 loss: 0.2816 2024/07/24 15:56:48 - mmengine - INFO - Iter(train) [ 2160/19224] lr: 1.9646e-05 eta: 11:56:17 time: 2.5528 data_time: 0.0111 memory: 11080 loss: 0.2223 2024/07/24 15:57:13 - mmengine - INFO - Iter(train) [ 2170/19224] lr: 1.9642e-05 eta: 11:55:46 time: 2.4414 data_time: 0.0105 memory: 10926 loss: 0.2506 2024/07/24 15:57:34 - mmengine - INFO - Iter(train) [ 2180/19224] lr: 1.9638e-05 eta: 11:54:51 time: 2.1280 data_time: 0.0097 memory: 10615 loss: 0.2328 2024/07/24 15:57:53 - mmengine - INFO - Iter(train) [ 2190/19224] lr: 1.9633e-05 eta: 11:53:39 time: 1.9153 data_time: 0.0093 memory: 10228 loss: 0.2374 2024/07/24 15:58:09 - mmengine - INFO - Iter(train) [ 2200/19224] lr: 1.9629e-05 eta: 11:51:58 time: 1.5375 data_time: 0.0095 memory: 9942 loss: 0.2825 2024/07/24 15:58:44 - mmengine - INFO - Iter(train) [ 2210/19224] lr: 1.9624e-05 eta: 11:52:49 time: 3.4997 data_time: 0.0105 memory: 14744 loss: 0.2160 2024/07/24 15:59:15 - mmengine - INFO - Iter(train) [ 2220/19224] lr: 1.9619e-05 eta: 11:53:12 time: 3.1411 data_time: 0.0105 memory: 12255 loss: 0.2164 2024/07/24 15:59:45 - mmengine - INFO - Iter(train) [ 2230/19224] lr: 1.9615e-05 eta: 11:53:20 time: 2.9583 data_time: 0.0107 memory: 11891 loss: 0.2078 2024/07/24 16:00:15 - mmengine - INFO - Iter(train) [ 2240/19224] lr: 1.9610e-05 eta: 11:53:36 time: 3.0571 data_time: 0.0111 memory: 11613 loss: 0.2558 2024/07/24 16:00:42 - mmengine - INFO - Iter(train) [ 2250/19224] lr: 1.9605e-05 eta: 11:53:23 time: 2.6789 data_time: 0.0107 memory: 11384 loss: 0.2203 2024/07/24 16:01:08 - mmengine - INFO - Iter(train) [ 2260/19224] lr: 1.9601e-05 eta: 11:53:05 time: 2.6209 data_time: 0.0109 memory: 11281 loss: 0.2539 2024/07/24 16:01:33 - mmengine - INFO - Iter(train) [ 2270/19224] lr: 1.9596e-05 eta: 11:52:34 time: 2.4407 data_time: 0.0105 memory: 11037 loss: 0.2606 2024/07/24 16:01:56 - mmengine - INFO - Iter(train) [ 2280/19224] lr: 1.9591e-05 eta: 11:51:51 time: 2.2898 data_time: 0.0108 memory: 10801 loss: 0.3265 2024/07/24 16:02:15 - mmengine - INFO - Iter(train) [ 2290/19224] lr: 1.9586e-05 eta: 11:50:43 time: 1.9387 data_time: 0.0100 memory: 10388 loss: 0.2396 2024/07/24 16:02:29 - mmengine - INFO - Iter(train) [ 2300/19224] lr: 1.9582e-05 eta: 11:48:57 time: 1.4258 data_time: 0.0091 memory: 9955 loss: 0.3153 2024/07/24 16:03:02 - mmengine - INFO - Iter(train) [ 2310/19224] lr: 1.9577e-05 eta: 11:49:32 time: 3.3231 data_time: 0.0103 memory: 14445 loss: 0.2321 2024/07/24 16:03:33 - mmengine - INFO - Iter(train) [ 2320/19224] lr: 1.9572e-05 eta: 11:49:43 time: 3.0156 data_time: 0.0111 memory: 12064 loss: 0.2164 2024/07/24 16:04:02 - mmengine - INFO - Iter(train) [ 2330/19224] lr: 1.9567e-05 eta: 11:49:50 time: 2.9701 data_time: 0.0108 memory: 11733 loss: 0.2023 2024/07/24 16:04:30 - mmengine - INFO - Iter(train) [ 2340/19224] lr: 1.9562e-05 eta: 11:49:45 time: 2.7919 data_time: 0.0106 memory: 11533 loss: 0.2406 2024/07/24 16:04:57 - mmengine - INFO - Iter(train) [ 2350/19224] lr: 1.9557e-05 eta: 11:49:34 time: 2.7231 data_time: 0.0108 memory: 11332 loss: 0.2206 2024/07/24 16:05:24 - mmengine - INFO - Iter(train) [ 2360/19224] lr: 1.9552e-05 eta: 11:49:16 time: 2.6276 data_time: 0.0114 memory: 11195 loss: 0.2375 2024/07/24 16:05:49 - mmengine - INFO - Iter(train) [ 2370/19224] lr: 1.9547e-05 eta: 11:48:52 time: 2.5393 data_time: 0.0102 memory: 11031 loss: 0.2559 2024/07/24 16:06:12 - mmengine - INFO - Iter(train) [ 2380/19224] lr: 1.9542e-05 eta: 11:48:10 time: 2.2947 data_time: 0.0100 memory: 10871 loss: 0.3083 2024/07/24 16:06:33 - mmengine - INFO - Iter(train) [ 2390/19224] lr: 1.9537e-05 eta: 11:47:11 time: 2.0421 data_time: 0.0094 memory: 10353 loss: 0.2392 2024/07/24 16:06:46 - mmengine - INFO - Iter(train) [ 2400/19224] lr: 1.9532e-05 eta: 11:45:22 time: 1.3172 data_time: 0.0089 memory: 9892 loss: 0.2348 2024/07/24 16:07:20 - mmengine - INFO - Iter(train) [ 2410/19224] lr: 1.9527e-05 eta: 11:46:01 time: 3.4399 data_time: 0.0106 memory: 14425 loss: 0.2243 2024/07/24 16:07:50 - mmengine - INFO - Iter(train) [ 2420/19224] lr: 1.9522e-05 eta: 11:46:09 time: 2.9909 data_time: 0.0110 memory: 11959 loss: 0.2386 2024/07/24 16:08:19 - mmengine - INFO - Iter(train) [ 2430/19224] lr: 1.9517e-05 eta: 11:46:09 time: 2.8934 data_time: 0.0113 memory: 11729 loss: 0.2107 2024/07/24 16:08:47 - mmengine - INFO - Iter(train) [ 2440/19224] lr: 1.9512e-05 eta: 11:46:02 time: 2.7908 data_time: 0.0107 memory: 11513 loss: 0.2263 2024/07/24 16:09:14 - mmengine - INFO - Iter(train) [ 2450/19224] lr: 1.9506e-05 eta: 11:45:49 time: 2.6981 data_time: 0.0107 memory: 11278 loss: 0.2720 2024/07/24 16:09:40 - mmengine - INFO - Iter(train) [ 2460/19224] lr: 1.9501e-05 eta: 11:45:28 time: 2.5830 data_time: 0.0104 memory: 11112 loss: 0.2466 2024/07/24 16:10:04 - mmengine - INFO - Iter(train) [ 2470/19224] lr: 1.9496e-05 eta: 11:44:59 time: 2.4736 data_time: 0.0105 memory: 11016 loss: 0.2878 2024/07/24 16:10:27 - mmengine - INFO - Iter(train) [ 2480/19224] lr: 1.9490e-05 eta: 11:44:19 time: 2.3024 data_time: 0.0103 memory: 10830 loss: 0.2739 2024/07/24 16:10:46 - mmengine - INFO - Iter(train) [ 2490/19224] lr: 1.9485e-05 eta: 11:43:11 time: 1.8895 data_time: 0.0094 memory: 10251 loss: 0.2565 2024/07/24 16:11:00 - mmengine - INFO - Iter(train) [ 2500/19224] lr: 1.9480e-05 eta: 11:41:32 time: 1.4147 data_time: 0.0088 memory: 9660 loss: 0.2647 2024/07/24 16:11:36 - mmengine - INFO - Iter(train) [ 2510/19224] lr: 1.9474e-05 eta: 11:42:19 time: 3.6002 data_time: 0.0110 memory: 16632 loss: 0.2400 2024/07/24 16:12:07 - mmengine - INFO - Iter(train) [ 2520/19224] lr: 1.9469e-05 eta: 11:42:31 time: 3.0870 data_time: 0.0105 memory: 12194 loss: 0.2010 2024/07/24 16:12:37 - mmengine - INFO - Iter(train) [ 2530/19224] lr: 1.9464e-05 eta: 11:42:33 time: 2.9393 data_time: 0.0109 memory: 11825 loss: 0.2105 2024/07/24 16:13:06 - mmengine - INFO - Iter(train) [ 2540/19224] lr: 1.9458e-05 eta: 11:42:32 time: 2.8863 data_time: 0.0112 memory: 11598 loss: 0.2441 2024/07/24 16:13:34 - mmengine - INFO - Iter(train) [ 2550/19224] lr: 1.9453e-05 eta: 11:42:24 time: 2.8025 data_time: 0.0109 memory: 11479 loss: 0.2271 2024/07/24 16:14:01 - mmengine - INFO - Iter(train) [ 2560/19224] lr: 1.9447e-05 eta: 11:42:10 time: 2.6925 data_time: 0.0107 memory: 11353 loss: 0.2195 2024/07/24 16:14:26 - mmengine - INFO - Iter(train) [ 2570/19224] lr: 1.9442e-05 eta: 11:41:47 time: 2.5635 data_time: 0.0110 memory: 11188 loss: 0.2482 2024/07/24 16:14:49 - mmengine - INFO - Iter(train) [ 2580/19224] lr: 1.9436e-05 eta: 11:41:06 time: 2.2901 data_time: 0.0108 memory: 10771 loss: 0.2764 2024/07/24 16:15:09 - mmengine - INFO - Iter(train) [ 2590/19224] lr: 1.9430e-05 eta: 11:40:08 time: 2.0114 data_time: 0.0104 memory: 10333 loss: 0.2154 2024/07/24 16:15:25 - mmengine - INFO - Iter(train) [ 2600/19224] lr: 1.9425e-05 eta: 11:38:40 time: 1.5464 data_time: 0.0095 memory: 9930 loss: 0.2490 2024/07/24 16:16:00 - mmengine - INFO - Iter(train) [ 2610/19224] lr: 1.9419e-05 eta: 11:39:20 time: 3.5430 data_time: 0.0107 memory: 13956 loss: 0.1984 2024/07/24 16:16:31 - mmengine - INFO - Iter(train) [ 2620/19224] lr: 1.9414e-05 eta: 11:39:31 time: 3.0936 data_time: 0.0107 memory: 12078 loss: 0.2075 2024/07/24 16:17:00 - mmengine - INFO - Iter(train) [ 2630/19224] lr: 1.9408e-05 eta: 11:39:29 time: 2.9079 data_time: 0.0109 memory: 11654 loss: 0.2055 2024/07/24 16:17:28 - mmengine - INFO - Iter(train) [ 2640/19224] lr: 1.9402e-05 eta: 11:39:19 time: 2.7766 data_time: 0.0111 memory: 11510 loss: 0.2448 2024/07/24 16:17:54 - mmengine - INFO - Iter(train) [ 2650/19224] lr: 1.9396e-05 eta: 11:38:57 time: 2.5809 data_time: 0.0105 memory: 11255 loss: 0.2425 2024/07/24 16:18:19 - mmengine - INFO - Iter(train) [ 2660/19224] lr: 1.9391e-05 eta: 11:38:32 time: 2.5215 data_time: 0.0110 memory: 11139 loss: 0.2489 2024/07/24 16:18:44 - mmengine - INFO - Iter(train) [ 2670/19224] lr: 1.9385e-05 eta: 11:38:03 time: 2.4855 data_time: 0.0106 memory: 10988 loss: 0.3065 2024/07/24 16:19:08 - mmengine - INFO - Iter(train) [ 2680/19224] lr: 1.9379e-05 eta: 11:37:29 time: 2.3797 data_time: 0.0103 memory: 10848 loss: 0.4877 2024/07/24 16:19:27 - mmengine - INFO - Iter(train) [ 2690/19224] lr: 1.9373e-05 eta: 11:36:25 time: 1.9083 data_time: 0.0093 memory: 10492 loss: 0.3210 2024/07/24 16:19:42 - mmengine - INFO - Iter(train) [ 2700/19224] lr: 1.9367e-05 eta: 11:34:59 time: 1.5330 data_time: 0.0088 memory: 9873 loss: 0.2983 2024/07/24 16:20:28 - mmengine - INFO - Iter(train) [ 2710/19224] lr: 1.9361e-05 eta: 11:36:44 time: 4.6513 data_time: 1.5379 memory: 13504 loss: 0.2290 2024/07/24 16:20:58 - mmengine - INFO - Iter(train) [ 2720/19224] lr: 1.9355e-05 eta: 11:36:41 time: 2.9115 data_time: 0.0110 memory: 11898 loss: 0.2163 2024/07/24 16:21:26 - mmengine - INFO - Iter(train) [ 2730/19224] lr: 1.9349e-05 eta: 11:36:37 time: 2.8756 data_time: 0.0115 memory: 12460 loss: 0.2138 2024/07/24 16:21:55 - mmengine - INFO - Iter(train) [ 2740/19224] lr: 1.9343e-05 eta: 11:36:29 time: 2.8260 data_time: 0.0104 memory: 11609 loss: 0.2390 2024/07/24 16:22:22 - mmengine - INFO - Iter(train) [ 2750/19224] lr: 1.9337e-05 eta: 11:36:15 time: 2.7189 data_time: 0.0110 memory: 11293 loss: 0.2168 2024/07/24 16:22:48 - mmengine - INFO - Iter(train) [ 2760/19224] lr: 1.9331e-05 eta: 11:35:56 time: 2.6519 data_time: 0.0108 memory: 11198 loss: 0.2751 2024/07/24 16:23:13 - mmengine - INFO - Iter(train) [ 2770/19224] lr: 1.9325e-05 eta: 11:35:28 time: 2.4840 data_time: 0.0103 memory: 11017 loss: 0.2726 2024/07/24 16:23:36 - mmengine - INFO - Iter(train) [ 2780/19224] lr: 1.9319e-05 eta: 11:34:45 time: 2.2385 data_time: 0.0098 memory: 10614 loss: 0.2976 2024/07/24 16:23:56 - mmengine - INFO - Iter(train) [ 2790/19224] lr: 1.9313e-05 eta: 11:33:49 time: 2.0163 data_time: 0.0103 memory: 10354 loss: 0.2242 2024/07/24 16:24:09 - mmengine - INFO - Iter(train) [ 2800/19224] lr: 1.9307e-05 eta: 11:32:14 time: 1.3466 data_time: 0.0099 memory: 9875 loss: 0.2730 2024/07/24 16:24:41 - mmengine - INFO - Iter(train) [ 2810/19224] lr: 1.9301e-05 eta: 11:32:29 time: 3.2112 data_time: 0.0110 memory: 13731 loss: 0.2247 2024/07/24 16:25:11 - mmengine - INFO - Iter(train) [ 2820/19224] lr: 1.9295e-05 eta: 11:32:31 time: 3.0112 data_time: 0.0106 memory: 11868 loss: 0.2167 2024/07/24 16:25:40 - mmengine - INFO - Iter(train) [ 2830/19224] lr: 1.9288e-05 eta: 11:32:27 time: 2.9065 data_time: 0.0106 memory: 11575 loss: 0.1991 2024/07/24 16:26:07 - mmengine - INFO - Iter(train) [ 2840/19224] lr: 1.9282e-05 eta: 11:32:12 time: 2.7051 data_time: 0.0104 memory: 11265 loss: 0.2343 2024/07/24 16:26:34 - mmengine - INFO - Iter(train) [ 2850/19224] lr: 1.9276e-05 eta: 11:31:53 time: 2.6470 data_time: 0.0107 memory: 11176 loss: 0.2801 2024/07/24 16:27:00 - mmengine - INFO - Iter(train) [ 2860/19224] lr: 1.9269e-05 eta: 11:31:31 time: 2.5980 data_time: 0.0106 memory: 11057 loss: 0.3083 2024/07/24 16:27:23 - mmengine - INFO - Iter(train) [ 2870/19224] lr: 1.9263e-05 eta: 11:30:50 time: 2.2606 data_time: 0.0105 memory: 10816 loss: 0.2833 2024/07/24 16:27:43 - mmengine - INFO - Iter(train) [ 2880/19224] lr: 1.9257e-05 eta: 11:29:55 time: 2.0113 data_time: 0.0095 memory: 10415 loss: 0.2595 2024/07/24 16:28:01 - mmengine - INFO - Iter(train) [ 2890/19224] lr: 1.9250e-05 eta: 11:28:50 time: 1.8249 data_time: 0.0091 memory: 10069 loss: 0.3042 2024/07/24 16:28:15 - mmengine - INFO - Iter(train) [ 2900/19224] lr: 1.9244e-05 eta: 11:27:19 time: 1.3603 data_time: 0.0088 memory: 9830 loss: 0.3855 2024/07/24 16:28:51 - mmengine - INFO - Iter(train) [ 2910/19224] lr: 1.9238e-05 eta: 11:27:58 time: 3.6738 data_time: 0.0107 memory: 17245 loss: 0.2102 2024/07/24 16:29:22 - mmengine - INFO - Iter(train) [ 2920/19224] lr: 1.9231e-05 eta: 11:28:01 time: 3.0419 data_time: 0.0105 memory: 12075 loss: 0.2124 2024/07/24 16:29:52 - mmengine - INFO - Iter(train) [ 2930/19224] lr: 1.9225e-05 eta: 11:28:01 time: 2.9942 data_time: 0.0104 memory: 12057 loss: 0.2111 2024/07/24 16:30:23 - mmengine - INFO - Iter(train) [ 2940/19224] lr: 1.9218e-05 eta: 11:28:10 time: 3.1462 data_time: 0.0104 memory: 11542 loss: 0.2387 2024/07/24 16:30:50 - mmengine - INFO - Iter(train) [ 2950/19224] lr: 1.9211e-05 eta: 11:27:56 time: 2.7424 data_time: 0.0110 memory: 11333 loss: 0.2077 2024/07/24 16:31:17 - mmengine - INFO - Iter(train) [ 2960/19224] lr: 1.9205e-05 eta: 11:27:37 time: 2.6598 data_time: 0.0104 memory: 11198 loss: 0.3072 2024/07/24 16:31:43 - mmengine - INFO - Iter(train) [ 2970/19224] lr: 1.9198e-05 eta: 11:27:16 time: 2.6132 data_time: 0.0105 memory: 11054 loss: 0.2599 2024/07/24 16:32:07 - mmengine - INFO - Iter(train) [ 2980/19224] lr: 1.9192e-05 eta: 11:26:41 time: 2.3622 data_time: 0.0105 memory: 10887 loss: 0.3034 2024/07/24 16:32:27 - mmengine - INFO - Iter(train) [ 2990/19224] lr: 1.9185e-05 eta: 11:25:47 time: 2.0047 data_time: 0.0094 memory: 10455 loss: 0.3263 2024/07/24 16:32:40 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 16:32:40 - mmengine - INFO - Iter(train) [ 3000/19224] lr: 1.9178e-05 eta: 11:24:15 time: 1.3075 data_time: 0.0085 memory: 9706 loss: 0.3133 2024/07/24 16:32:40 - mmengine - INFO - Saving checkpoint at 3000 iterations 2024/07/24 16:33:16 - mmengine - INFO - Iter(train) [ 3010/19224] lr: 1.9172e-05 eta: 11:24:48 time: 3.6063 data_time: 0.2050 memory: 13718 loss: 0.2533 2024/07/24 16:33:47 - mmengine - INFO - Iter(train) [ 3020/19224] lr: 1.9165e-05 eta: 11:24:53 time: 3.0974 data_time: 0.0108 memory: 11814 loss: 0.1966 2024/07/24 16:34:17 - mmengine - INFO - Iter(train) [ 3030/19224] lr: 1.9158e-05 eta: 11:24:52 time: 2.9844 data_time: 0.0106 memory: 11627 loss: 0.2124 2024/07/24 16:34:45 - mmengine - INFO - Iter(train) [ 3040/19224] lr: 1.9151e-05 eta: 11:24:41 time: 2.8152 data_time: 0.0106 memory: 11521 loss: 0.2092 2024/07/24 16:35:12 - mmengine - INFO - Iter(train) [ 3050/19224] lr: 1.9145e-05 eta: 11:24:24 time: 2.6941 data_time: 0.0110 memory: 11365 loss: 0.2421 2024/07/24 16:35:38 - mmengine - INFO - Iter(train) [ 3060/19224] lr: 1.9138e-05 eta: 11:24:00 time: 2.5706 data_time: 0.0107 memory: 11185 loss: 0.2461 2024/07/24 16:36:02 - mmengine - INFO - Iter(train) [ 3070/19224] lr: 1.9131e-05 eta: 11:23:31 time: 2.4691 data_time: 0.0103 memory: 11013 loss: 0.3437 2024/07/24 16:36:25 - mmengine - INFO - Iter(train) [ 3080/19224] lr: 1.9124e-05 eta: 11:22:52 time: 2.2707 data_time: 0.0109 memory: 10821 loss: 0.3341 2024/07/24 16:36:45 - mmengine - INFO - Iter(train) [ 3090/19224] lr: 1.9117e-05 eta: 11:21:57 time: 1.9805 data_time: 0.0094 memory: 10423 loss: 0.2232 2024/07/24 16:36:59 - mmengine - INFO - Iter(train) [ 3100/19224] lr: 1.9110e-05 eta: 11:20:34 time: 1.4331 data_time: 0.0091 memory: 9961 loss: 0.2519 2024/07/24 16:37:35 - mmengine - INFO - Iter(train) [ 3110/19224] lr: 1.9103e-05 eta: 11:21:03 time: 3.5798 data_time: 0.0100 memory: 19084 loss: 0.1855 2024/07/24 16:38:05 - mmengine - INFO - Iter(train) [ 3120/19224] lr: 1.9096e-05 eta: 11:21:01 time: 2.9778 data_time: 0.0110 memory: 12038 loss: 0.2792 2024/07/24 16:38:34 - mmengine - INFO - Iter(train) [ 3130/19224] lr: 1.9089e-05 eta: 11:20:54 time: 2.9041 data_time: 0.0109 memory: 11722 loss: 0.2379 2024/07/24 16:39:02 - mmengine - INFO - Iter(train) [ 3140/19224] lr: 1.9082e-05 eta: 11:20:41 time: 2.7807 data_time: 0.0114 memory: 11790 loss: 0.2366 2024/07/24 16:39:28 - mmengine - INFO - Iter(train) [ 3150/19224] lr: 1.9075e-05 eta: 11:20:21 time: 2.6404 data_time: 0.0106 memory: 11284 loss: 0.2139 2024/07/24 16:39:55 - mmengine - INFO - Iter(train) [ 3160/19224] lr: 1.9068e-05 eta: 11:20:03 time: 2.6841 data_time: 0.0105 memory: 11254 loss: 0.2933 2024/07/24 16:40:21 - mmengine - INFO - Iter(train) [ 3170/19224] lr: 1.9061e-05 eta: 11:19:40 time: 2.5809 data_time: 0.0105 memory: 11076 loss: 0.2435 2024/07/24 16:40:43 - mmengine - INFO - Iter(train) [ 3180/19224] lr: 1.9054e-05 eta: 11:18:57 time: 2.1877 data_time: 0.0101 memory: 10755 loss: 0.3437 2024/07/24 16:41:01 - mmengine - INFO - Iter(train) [ 3190/19224] lr: 1.9047e-05 eta: 11:17:56 time: 1.8317 data_time: 0.0094 memory: 10254 loss: 0.2780 2024/07/24 16:41:13 - mmengine - INFO - Iter(train) [ 3200/19224] lr: 1.9039e-05 eta: 11:16:26 time: 1.2474 data_time: 0.0088 memory: 9532 loss: 0.2386 2024/07/24 16:41:48 - mmengine - INFO - Iter(train) [ 3210/19224] lr: 1.9032e-05 eta: 11:16:45 time: 3.4306 data_time: 0.0107 memory: 13632 loss: 0.2796 2024/07/24 16:42:17 - mmengine - INFO - Iter(train) [ 3220/19224] lr: 1.9025e-05 eta: 11:16:40 time: 2.9531 data_time: 0.0104 memory: 11963 loss: 0.2121 2024/07/24 16:42:45 - mmengine - INFO - Iter(train) [ 3230/19224] lr: 1.9018e-05 eta: 11:16:28 time: 2.8065 data_time: 0.0109 memory: 11666 loss: 0.2004 2024/07/24 16:43:12 - mmengine - INFO - Iter(train) [ 3240/19224] lr: 1.9010e-05 eta: 11:16:11 time: 2.7063 data_time: 0.0105 memory: 11324 loss: 0.2229 2024/07/24 16:43:38 - mmengine - INFO - Iter(train) [ 3250/19224] lr: 1.9003e-05 eta: 11:15:47 time: 2.5608 data_time: 0.0107 memory: 11198 loss: 0.2925 2024/07/24 16:44:03 - mmengine - INFO - Iter(train) [ 3260/19224] lr: 1.8996e-05 eta: 11:15:19 time: 2.4920 data_time: 0.0106 memory: 11074 loss: 0.2707 2024/07/24 16:44:25 - mmengine - INFO - Iter(train) [ 3270/19224] lr: 1.8988e-05 eta: 11:14:40 time: 2.2513 data_time: 0.0104 memory: 10833 loss: 0.2775 2024/07/24 16:44:46 - mmengine - INFO - Iter(train) [ 3280/19224] lr: 1.8981e-05 eta: 11:13:51 time: 2.0479 data_time: 0.0102 memory: 10429 loss: 0.2769 2024/07/24 16:45:04 - mmengine - INFO - Iter(train) [ 3290/19224] lr: 1.8974e-05 eta: 11:12:51 time: 1.8299 data_time: 0.0094 memory: 10158 loss: 0.2482 2024/07/24 16:45:18 - mmengine - INFO - Iter(train) [ 3300/19224] lr: 1.8966e-05 eta: 11:11:32 time: 1.4094 data_time: 0.0089 memory: 9867 loss: 0.2461 2024/07/24 16:45:56 - mmengine - INFO - Iter(train) [ 3310/19224] lr: 1.8959e-05 eta: 11:12:08 time: 3.8157 data_time: 0.0105 memory: 18250 loss: 0.2477 2024/07/24 16:46:27 - mmengine - INFO - Iter(train) [ 3320/19224] lr: 1.8951e-05 eta: 11:12:07 time: 3.0419 data_time: 0.0110 memory: 12253 loss: 0.2167 2024/07/24 16:46:57 - mmengine - INFO - Iter(train) [ 3330/19224] lr: 1.8944e-05 eta: 11:12:04 time: 3.0083 data_time: 0.0128 memory: 11926 loss: 0.1881 2024/07/24 16:47:25 - mmengine - INFO - Iter(train) [ 3340/19224] lr: 1.8936e-05 eta: 11:11:53 time: 2.8230 data_time: 0.0125 memory: 11649 loss: 0.2211 2024/07/24 16:47:52 - mmengine - INFO - Iter(train) [ 3350/19224] lr: 1.8929e-05 eta: 11:11:36 time: 2.7202 data_time: 0.0106 memory: 11393 loss: 0.2210 2024/07/24 16:48:19 - mmengine - INFO - Iter(train) [ 3360/19224] lr: 1.8921e-05 eta: 11:11:15 time: 2.6244 data_time: 0.0106 memory: 11295 loss: 0.2507 2024/07/24 16:48:44 - mmengine - INFO - Iter(train) [ 3370/19224] lr: 1.8913e-05 eta: 11:10:50 time: 2.5466 data_time: 0.0105 memory: 11117 loss: 0.2532 2024/07/24 16:49:07 - mmengine - INFO - Iter(train) [ 3380/19224] lr: 1.8906e-05 eta: 11:10:13 time: 2.2920 data_time: 0.0103 memory: 10871 loss: 0.2483 2024/07/24 16:49:26 - mmengine - INFO - Iter(train) [ 3390/19224] lr: 1.8898e-05 eta: 11:09:17 time: 1.8961 data_time: 0.0094 memory: 10380 loss: 0.2296 2024/07/24 16:49:40 - mmengine - INFO - Iter(train) [ 3400/19224] lr: 1.8890e-05 eta: 11:08:01 time: 1.4406 data_time: 0.0095 memory: 9879 loss: 0.2704 2024/07/24 16:50:16 - mmengine - INFO - Iter(train) [ 3410/19224] lr: 1.8883e-05 eta: 11:08:24 time: 3.5727 data_time: 0.0102 memory: 14347 loss: 0.2261 2024/07/24 16:50:47 - mmengine - INFO - Iter(train) [ 3420/19224] lr: 1.8875e-05 eta: 11:08:26 time: 3.1420 data_time: 0.0108 memory: 12241 loss: 0.2070 2024/07/24 16:51:19 - mmengine - INFO - Iter(train) [ 3430/19224] lr: 1.8867e-05 eta: 11:08:28 time: 3.1182 data_time: 0.0105 memory: 11686 loss: 0.2136 2024/07/24 16:51:49 - mmengine - INFO - Iter(train) [ 3440/19224] lr: 1.8859e-05 eta: 11:08:25 time: 3.0406 data_time: 0.0104 memory: 11486 loss: 0.2079 2024/07/24 16:52:17 - mmengine - INFO - Iter(train) [ 3450/19224] lr: 1.8851e-05 eta: 11:08:14 time: 2.8422 data_time: 0.0107 memory: 11297 loss: 0.2311 2024/07/24 16:52:45 - mmengine - INFO - Iter(train) [ 3460/19224] lr: 1.8844e-05 eta: 11:08:00 time: 2.7954 data_time: 0.0110 memory: 11107 loss: 0.2481 2024/07/24 16:53:12 - mmengine - INFO - Iter(train) [ 3470/19224] lr: 1.8836e-05 eta: 11:07:38 time: 2.6247 data_time: 0.0103 memory: 11053 loss: 0.2453 2024/07/24 16:53:35 - mmengine - INFO - Iter(train) [ 3480/19224] lr: 1.8828e-05 eta: 11:07:03 time: 2.3360 data_time: 0.0105 memory: 10799 loss: 0.2821 2024/07/24 16:53:56 - mmengine - INFO - Iter(train) [ 3490/19224] lr: 1.8820e-05 eta: 11:06:17 time: 2.0731 data_time: 0.0099 memory: 10241 loss: 0.2126 2024/07/24 16:54:11 - mmengine - INFO - Iter(train) [ 3500/19224] lr: 1.8812e-05 eta: 11:05:07 time: 1.5421 data_time: 0.0090 memory: 9804 loss: 0.3073 2024/07/24 16:54:45 - mmengine - INFO - Iter(train) [ 3510/19224] lr: 1.8804e-05 eta: 11:05:18 time: 3.3652 data_time: 0.0103 memory: 12887 loss: 0.2190 2024/07/24 16:55:16 - mmengine - INFO - Iter(train) [ 3520/19224] lr: 1.8796e-05 eta: 11:05:17 time: 3.0839 data_time: 0.0105 memory: 11959 loss: 0.2379 2024/07/24 16:55:46 - mmengine - INFO - Iter(train) [ 3530/19224] lr: 1.8788e-05 eta: 11:05:12 time: 2.9936 data_time: 0.0110 memory: 11731 loss: 0.1929 2024/07/24 16:56:16 - mmengine - INFO - Iter(train) [ 3540/19224] lr: 1.8780e-05 eta: 11:05:08 time: 3.0271 data_time: 0.0111 memory: 11535 loss: 0.2262 2024/07/24 16:56:44 - mmengine - INFO - Iter(train) [ 3550/19224] lr: 1.8772e-05 eta: 11:04:54 time: 2.8012 data_time: 0.0105 memory: 11295 loss: 0.2197 2024/07/24 16:57:11 - mmengine - INFO - Iter(train) [ 3560/19224] lr: 1.8764e-05 eta: 11:04:36 time: 2.7315 data_time: 0.0108 memory: 11291 loss: 0.2162 2024/07/24 16:57:37 - mmengine - INFO - Iter(train) [ 3570/19224] lr: 1.8755e-05 eta: 11:04:14 time: 2.6195 data_time: 0.0103 memory: 11163 loss: 0.2508 2024/07/24 16:58:00 - mmengine - INFO - Iter(train) [ 3580/19224] lr: 1.8747e-05 eta: 11:03:35 time: 2.2254 data_time: 0.0093 memory: 10815 loss: 0.5139 2024/07/24 16:58:19 - mmengine - INFO - Iter(train) [ 3590/19224] lr: 1.8739e-05 eta: 11:02:41 time: 1.8939 data_time: 0.0095 memory: 10077 loss: 0.2377 2024/07/24 16:58:33 - mmengine - INFO - Iter(train) [ 3600/19224] lr: 1.8731e-05 eta: 11:01:29 time: 1.4764 data_time: 0.0089 memory: 9825 loss: 0.2848 2024/07/24 16:59:13 - mmengine - INFO - Iter(train) [ 3610/19224] lr: 1.8723e-05 eta: 11:02:07 time: 3.9997 data_time: 0.0109 memory: 14884 loss: 0.2421 2024/07/24 16:59:45 - mmengine - INFO - Iter(train) [ 3620/19224] lr: 1.8714e-05 eta: 11:02:06 time: 3.1168 data_time: 0.0103 memory: 12236 loss: 0.2103 2024/07/24 17:00:18 - mmengine - INFO - Iter(train) [ 3630/19224] lr: 1.8706e-05 eta: 11:02:15 time: 3.3435 data_time: 0.0106 memory: 11861 loss: 0.1987 2024/07/24 17:00:48 - mmengine - INFO - Iter(train) [ 3640/19224] lr: 1.8698e-05 eta: 11:02:09 time: 3.0054 data_time: 0.0111 memory: 11533 loss: 0.1927 2024/07/24 17:01:17 - mmengine - INFO - Iter(train) [ 3650/19224] lr: 1.8690e-05 eta: 11:01:59 time: 2.9032 data_time: 0.0115 memory: 11417 loss: 0.2290 2024/07/24 17:01:45 - mmengine - INFO - Iter(train) [ 3660/19224] lr: 1.8681e-05 eta: 11:01:46 time: 2.8452 data_time: 0.0109 memory: 11177 loss: 0.2415 2024/07/24 17:02:10 - mmengine - INFO - Iter(train) [ 3670/19224] lr: 1.8673e-05 eta: 11:01:16 time: 2.4639 data_time: 0.0107 memory: 11027 loss: 0.2677 2024/07/24 17:02:33 - mmengine - INFO - Iter(train) [ 3680/19224] lr: 1.8664e-05 eta: 11:00:41 time: 2.3248 data_time: 0.0108 memory: 10816 loss: 0.2367 2024/07/24 17:02:53 - mmengine - INFO - Iter(train) [ 3690/19224] lr: 1.8656e-05 eta: 10:59:52 time: 1.9903 data_time: 0.0096 memory: 10331 loss: 0.2223 2024/07/24 17:03:09 - mmengine - INFO - Iter(train) [ 3700/19224] lr: 1.8648e-05 eta: 10:58:46 time: 1.5772 data_time: 0.0096 memory: 9763 loss: 0.2121 2024/07/24 17:03:44 - mmengine - INFO - Iter(train) [ 3710/19224] lr: 1.8639e-05 eta: 10:58:59 time: 3.4677 data_time: 0.0105 memory: 14405 loss: 0.2168 2024/07/24 17:04:16 - mmengine - INFO - Iter(train) [ 3720/19224] lr: 1.8631e-05 eta: 10:59:01 time: 3.2123 data_time: 0.0123 memory: 12217 loss: 0.2026 2024/07/24 17:04:47 - mmengine - INFO - Iter(train) [ 3730/19224] lr: 1.8622e-05 eta: 10:59:00 time: 3.1438 data_time: 0.0104 memory: 11733 loss: 0.2226 2024/07/24 17:05:18 - mmengine - INFO - Iter(train) [ 3740/19224] lr: 1.8614e-05 eta: 10:58:57 time: 3.0818 data_time: 0.0192 memory: 11620 loss: 0.1982 2024/07/24 17:05:46 - mmengine - INFO - Iter(train) [ 3750/19224] lr: 1.8605e-05 eta: 10:58:42 time: 2.8069 data_time: 0.0104 memory: 11378 loss: 0.1920 2024/07/24 17:06:14 - mmengine - INFO - Iter(train) [ 3760/19224] lr: 1.8596e-05 eta: 10:58:25 time: 2.7557 data_time: 0.0104 memory: 11227 loss: 0.2539 2024/07/24 17:06:40 - mmengine - INFO - Iter(train) [ 3770/19224] lr: 1.8588e-05 eta: 10:58:01 time: 2.6121 data_time: 0.0103 memory: 11068 loss: 0.2298 2024/07/24 17:07:02 - mmengine - INFO - Iter(train) [ 3780/19224] lr: 1.8579e-05 eta: 10:57:23 time: 2.2375 data_time: 0.0098 memory: 10852 loss: 0.2339 2024/07/24 17:07:23 - mmengine - INFO - Iter(train) [ 3790/19224] lr: 1.8570e-05 eta: 10:56:38 time: 2.0814 data_time: 0.0102 memory: 10338 loss: 0.2675 2024/07/24 17:07:38 - mmengine - INFO - Iter(train) [ 3800/19224] lr: 1.8562e-05 eta: 10:55:29 time: 1.4760 data_time: 0.0091 memory: 9900 loss: 0.2413 2024/07/24 17:08:10 - mmengine - INFO - Iter(train) [ 3810/19224] lr: 1.8553e-05 eta: 10:55:32 time: 3.2688 data_time: 0.0105 memory: 13323 loss: 0.1924 2024/07/24 17:08:41 - mmengine - INFO - Iter(train) [ 3820/19224] lr: 1.8544e-05 eta: 10:55:25 time: 3.0104 data_time: 0.0113 memory: 11949 loss: 0.1951 2024/07/24 17:09:10 - mmengine - INFO - Iter(train) [ 3830/19224] lr: 1.8536e-05 eta: 10:55:14 time: 2.9074 data_time: 0.0107 memory: 11680 loss: 0.2713 2024/07/24 17:09:38 - mmengine - INFO - Iter(train) [ 3840/19224] lr: 1.8527e-05 eta: 10:55:00 time: 2.8354 data_time: 0.0103 memory: 11484 loss: 0.2097 2024/07/24 17:10:06 - mmengine - INFO - Iter(train) [ 3850/19224] lr: 1.8518e-05 eta: 10:54:43 time: 2.7628 data_time: 0.0105 memory: 11234 loss: 0.2253 2024/07/24 17:10:33 - mmengine - INFO - Iter(train) [ 3860/19224] lr: 1.8509e-05 eta: 10:54:25 time: 2.7568 data_time: 0.0106 memory: 11128 loss: 0.3008 2024/07/24 17:10:59 - mmengine - INFO - Iter(train) [ 3870/19224] lr: 1.8500e-05 eta: 10:54:01 time: 2.5848 data_time: 0.0104 memory: 11043 loss: 0.2661 2024/07/24 17:11:22 - mmengine - INFO - Iter(train) [ 3880/19224] lr: 1.8491e-05 eta: 10:53:24 time: 2.2857 data_time: 0.0103 memory: 10836 loss: 0.4406 2024/07/24 17:11:41 - mmengine - INFO - Iter(train) [ 3890/19224] lr: 1.8482e-05 eta: 10:52:32 time: 1.8742 data_time: 0.0096 memory: 10197 loss: 0.2128 2024/07/24 17:11:55 - mmengine - INFO - Iter(train) [ 3900/19224] lr: 1.8474e-05 eta: 10:51:22 time: 1.4285 data_time: 0.0088 memory: 9945 loss: 0.3109 2024/07/24 17:12:38 - mmengine - INFO - Iter(train) [ 3910/19224] lr: 1.8465e-05 eta: 10:52:04 time: 4.2620 data_time: 0.0101 memory: 18474 loss: 0.2088 2024/07/24 17:13:13 - mmengine - INFO - Iter(train) [ 3920/19224] lr: 1.8456e-05 eta: 10:52:17 time: 3.5411 data_time: 0.0107 memory: 12080 loss: 0.2039 2024/07/24 17:13:45 - mmengine - INFO - Iter(train) [ 3930/19224] lr: 1.8447e-05 eta: 10:52:15 time: 3.1575 data_time: 0.0107 memory: 11898 loss: 0.2118 2024/07/24 17:14:15 - mmengine - INFO - Iter(train) [ 3940/19224] lr: 1.8438e-05 eta: 10:52:08 time: 3.0389 data_time: 0.0114 memory: 11704 loss: 0.1954 2024/07/24 17:14:45 - mmengine - INFO - Iter(train) [ 3950/19224] lr: 1.8428e-05 eta: 10:51:58 time: 2.9828 data_time: 0.0110 memory: 11533 loss: 0.2144 2024/07/24 17:15:13 - mmengine - INFO - Iter(train) [ 3960/19224] lr: 1.8419e-05 eta: 10:51:41 time: 2.7859 data_time: 0.0107 memory: 11352 loss: 0.2507 2024/07/24 17:15:41 - mmengine - INFO - Iter(train) [ 3970/19224] lr: 1.8410e-05 eta: 10:51:24 time: 2.7871 data_time: 0.0107 memory: 11093 loss: 0.2664 2024/07/24 17:16:05 - mmengine - INFO - Iter(train) [ 3980/19224] lr: 1.8401e-05 eta: 10:50:54 time: 2.4238 data_time: 0.0107 memory: 10831 loss: 0.3057 2024/07/24 17:16:26 - mmengine - INFO - Iter(train) [ 3990/19224] lr: 1.8392e-05 eta: 10:50:09 time: 2.0781 data_time: 0.0090 memory: 10238 loss: 0.2453 2024/07/24 17:16:41 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 17:16:41 - mmengine - INFO - Iter(train) [ 4000/19224] lr: 1.8383e-05 eta: 10:49:05 time: 1.5487 data_time: 0.0098 memory: 9857 loss: 0.2947 2024/07/24 17:16:41 - mmengine - INFO - Saving checkpoint at 4000 iterations 2024/07/24 17:17:19 - mmengine - INFO - Iter(train) [ 4010/19224] lr: 1.8374e-05 eta: 10:49:25 time: 3.7564 data_time: 0.2071 memory: 12698 loss: 0.2523 2024/07/24 17:17:51 - mmengine - INFO - Iter(train) [ 4020/19224] lr: 1.8364e-05 eta: 10:49:24 time: 3.2098 data_time: 0.0111 memory: 11903 loss: 0.2423 2024/07/24 17:18:21 - mmengine - INFO - Iter(train) [ 4030/19224] lr: 1.8355e-05 eta: 10:49:16 time: 3.0380 data_time: 0.0112 memory: 11729 loss: 0.2789 2024/07/24 17:18:49 - mmengine - INFO - Iter(train) [ 4040/19224] lr: 1.8346e-05 eta: 10:49:01 time: 2.8438 data_time: 0.0111 memory: 11490 loss: 0.2281 2024/07/24 17:19:19 - mmengine - INFO - Iter(train) [ 4050/19224] lr: 1.8337e-05 eta: 10:48:49 time: 2.9232 data_time: 0.0110 memory: 11261 loss: 0.2423 2024/07/24 17:19:46 - mmengine - INFO - Iter(train) [ 4060/19224] lr: 1.8327e-05 eta: 10:48:29 time: 2.7142 data_time: 0.0113 memory: 11210 loss: 0.2572 2024/07/24 17:20:12 - mmengine - INFO - Iter(train) [ 4070/19224] lr: 1.8318e-05 eta: 10:48:04 time: 2.5742 data_time: 0.0103 memory: 10926 loss: 0.2602 2024/07/24 17:20:35 - mmengine - INFO - Iter(train) [ 4080/19224] lr: 1.8309e-05 eta: 10:47:28 time: 2.2959 data_time: 0.0097 memory: 10632 loss: 0.2247 2024/07/24 17:20:55 - mmengine - INFO - Iter(train) [ 4090/19224] lr: 1.8299e-05 eta: 10:46:44 time: 2.0629 data_time: 0.0094 memory: 10401 loss: 0.2649 2024/07/24 17:21:09 - mmengine - INFO - Iter(train) [ 4100/19224] lr: 1.8290e-05 eta: 10:45:33 time: 1.3402 data_time: 0.0086 memory: 9723 loss: 0.2303 2024/07/24 17:21:45 - mmengine - INFO - Iter(train) [ 4110/19224] lr: 1.8280e-05 eta: 10:45:45 time: 3.5980 data_time: 0.0104 memory: 13741 loss: 0.2222 2024/07/24 17:22:16 - mmengine - INFO - Iter(train) [ 4120/19224] lr: 1.8271e-05 eta: 10:45:41 time: 3.1364 data_time: 0.0107 memory: 12148 loss: 0.2132 2024/07/24 17:22:50 - mmengine - INFO - Iter(train) [ 4130/19224] lr: 1.8261e-05 eta: 10:45:45 time: 3.3926 data_time: 0.0108 memory: 11756 loss: 0.1897 2024/07/24 17:23:18 - mmengine - INFO - Iter(train) [ 4140/19224] lr: 1.8252e-05 eta: 10:45:30 time: 2.8522 data_time: 0.0103 memory: 11444 loss: 0.2612 2024/07/24 17:23:46 - mmengine - INFO - Iter(train) [ 4150/19224] lr: 1.8242e-05 eta: 10:45:11 time: 2.7354 data_time: 0.0103 memory: 11211 loss: 0.2094 2024/07/24 17:24:11 - mmengine - INFO - Iter(train) [ 4160/19224] lr: 1.8233e-05 eta: 10:44:43 time: 2.5198 data_time: 0.0101 memory: 11047 loss: 0.2233 2024/07/24 17:24:35 - mmengine - INFO - Iter(train) [ 4170/19224] lr: 1.8223e-05 eta: 10:44:11 time: 2.3839 data_time: 0.0104 memory: 10865 loss: 0.2184 2024/07/24 17:24:56 - mmengine - INFO - Iter(train) [ 4180/19224] lr: 1.8214e-05 eta: 10:43:29 time: 2.1328 data_time: 0.0095 memory: 10492 loss: 0.2512 2024/07/24 17:25:14 - mmengine - INFO - Iter(train) [ 4190/19224] lr: 1.8204e-05 eta: 10:42:37 time: 1.8339 data_time: 0.0091 memory: 10140 loss: 0.2957 2024/07/24 17:25:30 - mmengine - INFO - Iter(train) [ 4200/19224] lr: 1.8194e-05 eta: 10:41:35 time: 1.5338 data_time: 0.0091 memory: 9792 loss: 0.2684 2024/07/24 17:26:07 - mmengine - INFO - Iter(train) [ 4210/19224] lr: 1.8185e-05 eta: 10:41:52 time: 3.7625 data_time: 0.0113 memory: 13805 loss: 0.1993 2024/07/24 17:26:40 - mmengine - INFO - Iter(train) [ 4220/19224] lr: 1.8175e-05 eta: 10:41:52 time: 3.2813 data_time: 0.0105 memory: 12101 loss: 0.2549 2024/07/24 17:27:12 - mmengine - INFO - Iter(train) [ 4230/19224] lr: 1.8165e-05 eta: 10:41:48 time: 3.1843 data_time: 0.0106 memory: 12085 loss: 0.1858 2024/07/24 17:27:42 - mmengine - INFO - Iter(train) [ 4240/19224] lr: 1.8156e-05 eta: 10:41:38 time: 3.0030 data_time: 0.0104 memory: 11555 loss: 0.2073 2024/07/24 17:28:11 - mmengine - INFO - Iter(train) [ 4250/19224] lr: 1.8146e-05 eta: 10:41:22 time: 2.8440 data_time: 0.0109 memory: 11324 loss: 0.2048 2024/07/24 17:28:37 - mmengine - INFO - Iter(train) [ 4260/19224] lr: 1.8136e-05 eta: 10:40:58 time: 2.6296 data_time: 0.0103 memory: 11278 loss: 0.2372 2024/07/24 17:29:01 - mmengine - INFO - Iter(train) [ 4270/19224] lr: 1.8126e-05 eta: 10:40:29 time: 2.4595 data_time: 0.0104 memory: 10989 loss: 0.2690 2024/07/24 17:29:25 - mmengine - INFO - Iter(train) [ 4280/19224] lr: 1.8116e-05 eta: 10:39:56 time: 2.3716 data_time: 0.0106 memory: 10861 loss: 0.2337 2024/07/24 17:29:45 - mmengine - INFO - Iter(train) [ 4290/19224] lr: 1.8107e-05 eta: 10:39:11 time: 2.0099 data_time: 0.0093 memory: 10454 loss: 0.2639 2024/07/24 17:30:02 - mmengine - INFO - Iter(train) [ 4300/19224] lr: 1.8097e-05 eta: 10:38:16 time: 1.7141 data_time: 0.0094 memory: 10034 loss: 0.2924 2024/07/24 17:30:40 - mmengine - INFO - Iter(train) [ 4310/19224] lr: 1.8087e-05 eta: 10:38:32 time: 3.7885 data_time: 0.0105 memory: 14368 loss: 0.2258 2024/07/24 17:31:12 - mmengine - INFO - Iter(train) [ 4320/19224] lr: 1.8077e-05 eta: 10:38:28 time: 3.1946 data_time: 0.0117 memory: 12383 loss: 0.3252 2024/07/24 17:31:42 - mmengine - INFO - Iter(train) [ 4330/19224] lr: 1.8067e-05 eta: 10:38:18 time: 3.0183 data_time: 0.0103 memory: 11751 loss: 0.1975 2024/07/24 17:32:11 - mmengine - INFO - Iter(train) [ 4340/19224] lr: 1.8057e-05 eta: 10:38:02 time: 2.8525 data_time: 0.0099 memory: 11429 loss: 0.2110 2024/07/24 17:32:39 - mmengine - INFO - Iter(train) [ 4350/19224] lr: 1.8047e-05 eta: 10:37:46 time: 2.8541 data_time: 0.0102 memory: 11346 loss: 0.2298 2024/07/24 17:33:07 - mmengine - INFO - Iter(train) [ 4360/19224] lr: 1.8037e-05 eta: 10:37:26 time: 2.7457 data_time: 0.0116 memory: 11209 loss: 0.2582 2024/07/24 17:33:34 - mmengine - INFO - Iter(train) [ 4370/19224] lr: 1.8027e-05 eta: 10:37:05 time: 2.7108 data_time: 0.0118 memory: 11104 loss: 0.2219 2024/07/24 17:33:57 - mmengine - INFO - Iter(train) [ 4380/19224] lr: 1.8017e-05 eta: 10:36:30 time: 2.3062 data_time: 0.0104 memory: 10903 loss: 0.2428 2024/07/24 17:34:16 - mmengine - INFO - Iter(train) [ 4390/19224] lr: 1.8007e-05 eta: 10:35:42 time: 1.9227 data_time: 0.0094 memory: 10254 loss: 0.2137 2024/07/24 17:34:29 - mmengine - INFO - Iter(train) [ 4400/19224] lr: 1.7997e-05 eta: 10:34:33 time: 1.2840 data_time: 0.0086 memory: 9748 loss: 0.2246 2024/07/24 17:35:05 - mmengine - INFO - Iter(train) [ 4410/19224] lr: 1.7987e-05 eta: 10:34:40 time: 3.5385 data_time: 0.0104 memory: 13408 loss: 0.2243 2024/07/24 17:35:36 - mmengine - INFO - Iter(train) [ 4420/19224] lr: 1.7976e-05 eta: 10:34:33 time: 3.1287 data_time: 0.0112 memory: 11984 loss: 0.2111 2024/07/24 17:36:06 - mmengine - INFO - Iter(train) [ 4430/19224] lr: 1.7966e-05 eta: 10:34:22 time: 3.0132 data_time: 0.0108 memory: 11830 loss: 0.2128 2024/07/24 17:36:36 - mmengine - INFO - Iter(train) [ 4440/19224] lr: 1.7956e-05 eta: 10:34:10 time: 2.9736 data_time: 0.0113 memory: 11430 loss: 0.1996 2024/07/24 17:37:05 - mmengine - INFO - Iter(train) [ 4450/19224] lr: 1.7946e-05 eta: 10:33:57 time: 2.9550 data_time: 0.0129 memory: 11271 loss: 0.2348 2024/07/24 17:37:31 - mmengine - INFO - Iter(train) [ 4460/19224] lr: 1.7936e-05 eta: 10:33:32 time: 2.5936 data_time: 0.0112 memory: 11128 loss: 0.2348 2024/07/24 17:37:56 - mmengine - INFO - Iter(train) [ 4470/19224] lr: 1.7925e-05 eta: 10:33:03 time: 2.4919 data_time: 0.0103 memory: 10948 loss: 0.2530 2024/07/24 17:38:18 - mmengine - INFO - Iter(train) [ 4480/19224] lr: 1.7915e-05 eta: 10:32:26 time: 2.2157 data_time: 0.0096 memory: 10564 loss: 0.3309 2024/07/24 17:38:36 - mmengine - INFO - Iter(train) [ 4490/19224] lr: 1.7905e-05 eta: 10:31:35 time: 1.8077 data_time: 0.0094 memory: 10074 loss: 0.2563 2024/07/24 17:38:49 - mmengine - INFO - Iter(train) [ 4500/19224] lr: 1.7894e-05 eta: 10:30:25 time: 1.2225 data_time: 0.0081 memory: 9524 loss: 0.2209 2024/07/24 17:39:29 - mmengine - INFO - Iter(train) [ 4510/19224] lr: 1.7884e-05 eta: 10:30:47 time: 4.0405 data_time: 0.0108 memory: 16719 loss: 0.2103 2024/07/24 17:40:01 - mmengine - INFO - Iter(train) [ 4520/19224] lr: 1.7874e-05 eta: 10:30:43 time: 3.2279 data_time: 0.0106 memory: 12313 loss: 0.2190 2024/07/24 17:40:34 - mmengine - INFO - Iter(train) [ 4530/19224] lr: 1.7863e-05 eta: 10:30:40 time: 3.2739 data_time: 0.0118 memory: 12132 loss: 0.1962 2024/07/24 17:41:05 - mmengine - INFO - Iter(train) [ 4540/19224] lr: 1.7853e-05 eta: 10:30:29 time: 3.0516 data_time: 0.0107 memory: 11668 loss: 0.2223 2024/07/24 17:41:33 - mmengine - INFO - Iter(train) [ 4550/19224] lr: 1.7842e-05 eta: 10:30:13 time: 2.8757 data_time: 0.0108 memory: 11299 loss: 0.2316 2024/07/24 17:42:00 - mmengine - INFO - Iter(train) [ 4560/19224] lr: 1.7832e-05 eta: 10:29:52 time: 2.7168 data_time: 0.0112 memory: 11177 loss: 0.2587 2024/07/24 17:42:27 - mmengine - INFO - Iter(train) [ 4570/19224] lr: 1.7821e-05 eta: 10:29:28 time: 2.6352 data_time: 0.0104 memory: 11114 loss: 0.2511 2024/07/24 17:42:52 - mmengine - INFO - Iter(train) [ 4580/19224] lr: 1.7811e-05 eta: 10:28:59 time: 2.4871 data_time: 0.0101 memory: 10890 loss: 0.2385 2024/07/24 17:43:12 - mmengine - INFO - Iter(train) [ 4590/19224] lr: 1.7800e-05 eta: 10:28:15 time: 1.9985 data_time: 0.0096 memory: 10455 loss: 0.2610 2024/07/24 17:43:28 - mmengine - INFO - Iter(train) [ 4600/19224] lr: 1.7790e-05 eta: 10:27:18 time: 1.5890 data_time: 0.0095 memory: 10041 loss: 0.2363 2024/07/24 17:44:05 - mmengine - INFO - Iter(train) [ 4610/19224] lr: 1.7779e-05 eta: 10:27:29 time: 3.7223 data_time: 0.0101 memory: 15485 loss: 0.2278 2024/07/24 17:44:35 - mmengine - INFO - Iter(train) [ 4620/19224] lr: 1.7769e-05 eta: 10:27:17 time: 3.0372 data_time: 0.0116 memory: 11805 loss: 0.2214 2024/07/24 17:45:06 - mmengine - INFO - Iter(train) [ 4630/19224] lr: 1.7758e-05 eta: 10:27:09 time: 3.1142 data_time: 0.0116 memory: 11600 loss: 0.2235 2024/07/24 17:45:36 - mmengine - INFO - Iter(train) [ 4640/19224] lr: 1.7747e-05 eta: 10:26:56 time: 2.9805 data_time: 0.0107 memory: 11364 loss: 0.2240 2024/07/24 17:46:03 - mmengine - INFO - Iter(train) [ 4650/19224] lr: 1.7737e-05 eta: 10:26:33 time: 2.6873 data_time: 0.0110 memory: 11204 loss: 0.2549 2024/07/24 17:46:30 - mmengine - INFO - Iter(train) [ 4660/19224] lr: 1.7726e-05 eta: 10:26:10 time: 2.6609 data_time: 0.0101 memory: 11079 loss: 0.2058 2024/07/24 17:46:54 - mmengine - INFO - Iter(train) [ 4670/19224] lr: 1.7715e-05 eta: 10:25:40 time: 2.4638 data_time: 0.0104 memory: 10947 loss: 0.2301 2024/07/24 17:47:17 - mmengine - INFO - Iter(train) [ 4680/19224] lr: 1.7705e-05 eta: 10:25:05 time: 2.2846 data_time: 0.0096 memory: 10496 loss: 0.3916 2024/07/24 17:47:37 - mmengine - INFO - Iter(train) [ 4690/19224] lr: 1.7694e-05 eta: 10:24:21 time: 1.9710 data_time: 0.0092 memory: 10228 loss: 0.2292 2024/07/24 17:47:51 - mmengine - INFO - Iter(train) [ 4700/19224] lr: 1.7683e-05 eta: 10:23:20 time: 1.4571 data_time: 0.0104 memory: 9892 loss: 0.2824 2024/07/24 17:48:30 - mmengine - INFO - Iter(train) [ 4710/19224] lr: 1.7672e-05 eta: 10:23:35 time: 3.8788 data_time: 0.0115 memory: 15899 loss: 0.2275 2024/07/24 17:49:03 - mmengine - INFO - Iter(train) [ 4720/19224] lr: 1.7662e-05 eta: 10:23:31 time: 3.2848 data_time: 0.0108 memory: 12153 loss: 0.1817 2024/07/24 17:49:34 - mmengine - INFO - Iter(train) [ 4730/19224] lr: 1.7651e-05 eta: 10:23:21 time: 3.0919 data_time: 0.0110 memory: 11745 loss: 0.2193 2024/07/24 17:50:04 - mmengine - INFO - Iter(train) [ 4740/19224] lr: 1.7640e-05 eta: 10:23:08 time: 2.9959 data_time: 0.0110 memory: 11638 loss: 0.1990 2024/07/24 17:50:32 - mmengine - INFO - Iter(train) [ 4750/19224] lr: 1.7629e-05 eta: 10:22:50 time: 2.8650 data_time: 0.0104 memory: 11365 loss: 0.2261 2024/07/24 17:51:02 - mmengine - INFO - Iter(train) [ 4760/19224] lr: 1.7618e-05 eta: 10:22:36 time: 2.9480 data_time: 0.0111 memory: 11236 loss: 0.2021 2024/07/24 17:51:33 - mmengine - INFO - Iter(train) [ 4770/19224] lr: 1.7607e-05 eta: 10:22:25 time: 3.0671 data_time: 0.0110 memory: 11143 loss: 0.2277 2024/07/24 17:51:59 - mmengine - INFO - Iter(train) [ 4780/19224] lr: 1.7596e-05 eta: 10:22:01 time: 2.6540 data_time: 0.0106 memory: 11005 loss: 0.2413 2024/07/24 17:52:21 - mmengine - INFO - Iter(train) [ 4790/19224] lr: 1.7585e-05 eta: 10:21:24 time: 2.2140 data_time: 0.0096 memory: 10440 loss: 0.2222 2024/07/24 17:52:37 - mmengine - INFO - Iter(train) [ 4800/19224] lr: 1.7574e-05 eta: 10:20:29 time: 1.6070 data_time: 0.0092 memory: 10187 loss: 0.2926 2024/07/24 17:52:53 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 17:52:53 - mmengine - WARNING - Reach the end of the dataloader, it will be restarted and continue to iterate. It is recommended to use `mmengine.dataset.InfiniteSampler` to enable the dataloader to iterate infinitely. 2024/07/24 17:53:15 - mmengine - INFO - Iter(train) [ 4810/19224] lr: 1.7563e-05 eta: 10:20:39 time: 3.7830 data_time: 0.3188 memory: 19416 loss: 0.2450 2024/07/24 17:53:48 - mmengine - INFO - Iter(train) [ 4820/19224] lr: 1.7552e-05 eta: 10:20:33 time: 3.2378 data_time: 0.0106 memory: 12433 loss: 0.1836 2024/07/24 17:54:18 - mmengine - INFO - Iter(train) [ 4830/19224] lr: 1.7541e-05 eta: 10:20:20 time: 3.0304 data_time: 0.0106 memory: 11912 loss: 0.1939 2024/07/24 17:54:49 - mmengine - INFO - Iter(train) [ 4840/19224] lr: 1.7530e-05 eta: 10:20:10 time: 3.1019 data_time: 0.0109 memory: 11707 loss: 0.1887 2024/07/24 17:55:18 - mmengine - INFO - Iter(train) [ 4850/19224] lr: 1.7519e-05 eta: 10:19:52 time: 2.8743 data_time: 0.0108 memory: 11956 loss: 0.2052 2024/07/24 17:55:47 - mmengine - INFO - Iter(train) [ 4860/19224] lr: 1.7508e-05 eta: 10:19:38 time: 2.9654 data_time: 0.0103 memory: 11227 loss: 0.2089 2024/07/24 17:56:16 - mmengine - INFO - Iter(train) [ 4870/19224] lr: 1.7497e-05 eta: 10:19:21 time: 2.9010 data_time: 0.0109 memory: 11063 loss: 0.2463 2024/07/24 17:56:41 - mmengine - INFO - Iter(train) [ 4880/19224] lr: 1.7486e-05 eta: 10:18:51 time: 2.4408 data_time: 0.0115 memory: 10899 loss: 0.3010 2024/07/24 17:57:00 - mmengine - INFO - Iter(train) [ 4890/19224] lr: 1.7474e-05 eta: 10:18:05 time: 1.9295 data_time: 0.0106 memory: 10447 loss: 0.2361 2024/07/24 17:57:17 - mmengine - INFO - Iter(train) [ 4900/19224] lr: 1.7463e-05 eta: 10:17:15 time: 1.7343 data_time: 0.0099 memory: 10082 loss: 0.2162 2024/07/24 17:57:42 - mmengine - INFO - Iter(train) [ 4910/19224] lr: 1.7452e-05 eta: 10:16:45 time: 2.4498 data_time: 0.0097 memory: 17914 loss: 0.2435 2024/07/24 17:58:14 - mmengine - INFO - Iter(train) [ 4920/19224] lr: 1.7441e-05 eta: 10:16:37 time: 3.2059 data_time: 0.0110 memory: 12286 loss: 0.1665 2024/07/24 17:58:48 - mmengine - INFO - Iter(train) [ 4930/19224] lr: 1.7429e-05 eta: 10:16:34 time: 3.3848 data_time: 0.0105 memory: 11901 loss: 0.1795 2024/07/24 17:59:18 - mmengine - INFO - Iter(train) [ 4940/19224] lr: 1.7418e-05 eta: 10:16:21 time: 3.0169 data_time: 0.0105 memory: 11613 loss: 0.1881 2024/07/24 17:59:47 - mmengine - INFO - Iter(train) [ 4950/19224] lr: 1.7407e-05 eta: 10:16:04 time: 2.9044 data_time: 0.0108 memory: 11422 loss: 0.2164 2024/07/24 18:00:18 - mmengine - INFO - Iter(train) [ 4960/19224] lr: 1.7395e-05 eta: 10:15:52 time: 3.0865 data_time: 0.0108 memory: 11204 loss: 0.1984 2024/07/24 18:00:48 - mmengine - INFO - Iter(train) [ 4970/19224] lr: 1.7384e-05 eta: 10:15:38 time: 3.0018 data_time: 0.0107 memory: 11097 loss: 0.2107 2024/07/24 18:01:14 - mmengine - INFO - Iter(train) [ 4980/19224] lr: 1.7373e-05 eta: 10:15:12 time: 2.5779 data_time: 0.0106 memory: 10942 loss: 0.2513 2024/07/24 18:01:34 - mmengine - INFO - Iter(train) [ 4990/19224] lr: 1.7361e-05 eta: 10:14:29 time: 2.0091 data_time: 0.0096 memory: 10656 loss: 0.2255 2024/07/24 18:01:51 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 18:01:51 - mmengine - INFO - Iter(train) [ 5000/19224] lr: 1.7350e-05 eta: 10:13:40 time: 1.7629 data_time: 0.0093 memory: 10032 loss: 0.2246 2024/07/24 18:01:51 - mmengine - INFO - Saving checkpoint at 5000 iterations 2024/07/24 18:02:19 - mmengine - INFO - Iter(train) [ 5010/19224] lr: 1.7339e-05 eta: 10:13:18 time: 2.7338 data_time: 0.2045 memory: 18895 loss: 0.3128 2024/07/24 18:02:51 - mmengine - INFO - Iter(train) [ 5020/19224] lr: 1.7327e-05 eta: 10:13:12 time: 3.2756 data_time: 0.0108 memory: 12519 loss: 0.2478 2024/07/24 18:03:24 - mmengine - INFO - Iter(train) [ 5030/19224] lr: 1.7316e-05 eta: 10:13:04 time: 3.2154 data_time: 0.0107 memory: 11998 loss: 0.1916 2024/07/24 18:03:57 - mmengine - INFO - Iter(train) [ 5040/19224] lr: 1.7304e-05 eta: 10:13:00 time: 3.3832 data_time: 0.0104 memory: 12017 loss: 0.2295 2024/07/24 18:04:28 - mmengine - INFO - Iter(train) [ 5050/19224] lr: 1.7293e-05 eta: 10:12:46 time: 3.0084 data_time: 0.0106 memory: 11638 loss: 0.2203 2024/07/24 18:04:56 - mmengine - INFO - Iter(train) [ 5060/19224] lr: 1.7281e-05 eta: 10:12:26 time: 2.8103 data_time: 0.0106 memory: 11293 loss: 0.2111 2024/07/24 18:05:23 - mmengine - INFO - Iter(train) [ 5070/19224] lr: 1.7269e-05 eta: 10:12:03 time: 2.7095 data_time: 0.0106 memory: 11205 loss: 0.2073 2024/07/24 18:05:48 - mmengine - INFO - Iter(train) [ 5080/19224] lr: 1.7258e-05 eta: 10:11:36 time: 2.5727 data_time: 0.0113 memory: 11001 loss: 0.2111 2024/07/24 18:06:12 - mmengine - INFO - Iter(train) [ 5090/19224] lr: 1.7246e-05 eta: 10:11:04 time: 2.3465 data_time: 0.0108 memory: 10813 loss: 0.2238 2024/07/24 18:06:30 - mmengine - INFO - Iter(train) [ 5100/19224] lr: 1.7235e-05 eta: 10:10:16 time: 1.8241 data_time: 0.0097 memory: 10242 loss: 0.2301 2024/07/24 18:06:58 - mmengine - INFO - Iter(train) [ 5110/19224] lr: 1.7223e-05 eta: 10:09:57 time: 2.8201 data_time: 0.0092 memory: 15347 loss: 0.2581 2024/07/24 18:07:40 - mmengine - INFO - Iter(train) [ 5120/19224] lr: 1.7211e-05 eta: 10:10:15 time: 4.1895 data_time: 0.0107 memory: 12923 loss: 0.2048 2024/07/24 18:08:15 - mmengine - INFO - Iter(train) [ 5130/19224] lr: 1.7200e-05 eta: 10:10:12 time: 3.4569 data_time: 0.0114 memory: 12008 loss: 0.2255 2024/07/24 18:08:48 - mmengine - INFO - Iter(train) [ 5140/19224] lr: 1.7188e-05 eta: 10:10:06 time: 3.3104 data_time: 0.0106 memory: 11640 loss: 0.2021 2024/07/24 18:09:16 - mmengine - INFO - Iter(train) [ 5150/19224] lr: 1.7176e-05 eta: 10:09:45 time: 2.7981 data_time: 0.0105 memory: 11521 loss: 0.1788 2024/07/24 18:09:42 - mmengine - INFO - Iter(train) [ 5160/19224] lr: 1.7165e-05 eta: 10:09:21 time: 2.6425 data_time: 0.0112 memory: 11269 loss: 0.2194 2024/07/24 18:10:08 - mmengine - INFO - Iter(train) [ 5170/19224] lr: 1.7153e-05 eta: 10:08:55 time: 2.6070 data_time: 0.0105 memory: 11170 loss: 0.2009 2024/07/24 18:10:33 - mmengine - INFO - Iter(train) [ 5180/19224] lr: 1.7141e-05 eta: 10:08:26 time: 2.5005 data_time: 0.0104 memory: 10983 loss: 0.2409 2024/07/24 18:10:55 - mmengine - INFO - Iter(train) [ 5190/19224] lr: 1.7129e-05 eta: 10:07:49 time: 2.1880 data_time: 0.0096 memory: 10468 loss: 0.1732 2024/07/24 18:11:15 - mmengine - INFO - Iter(train) [ 5200/19224] lr: 1.7117e-05 eta: 10:07:05 time: 1.9463 data_time: 0.0095 memory: 10250 loss: 0.2340 2024/07/24 18:11:39 - mmengine - INFO - Iter(train) [ 5210/19224] lr: 1.7106e-05 eta: 10:06:34 time: 2.3960 data_time: 0.0098 memory: 13434 loss: 0.2651 2024/07/24 18:12:12 - mmengine - INFO - Iter(train) [ 5220/19224] lr: 1.7094e-05 eta: 10:06:27 time: 3.2948 data_time: 0.0109 memory: 12661 loss: 0.1767 2024/07/24 18:12:42 - mmengine - INFO - Iter(train) [ 5230/19224] lr: 1.7082e-05 eta: 10:06:12 time: 3.0101 data_time: 0.0116 memory: 12080 loss: 0.1866 2024/07/24 18:13:13 - mmengine - INFO - Iter(train) [ 5240/19224] lr: 1.7070e-05 eta: 10:06:01 time: 3.1689 data_time: 0.0113 memory: 11707 loss: 0.1903 2024/07/24 18:13:43 - mmengine - INFO - Iter(train) [ 5250/19224] lr: 1.7058e-05 eta: 10:05:44 time: 2.9326 data_time: 0.0106 memory: 11446 loss: 0.1659 2024/07/24 18:14:12 - mmengine - INFO - Iter(train) [ 5260/19224] lr: 1.7046e-05 eta: 10:05:26 time: 2.8969 data_time: 0.0105 memory: 11278 loss: 0.2073 2024/07/24 18:14:37 - mmengine - INFO - Iter(train) [ 5270/19224] lr: 1.7034e-05 eta: 10:04:58 time: 2.5411 data_time: 0.0108 memory: 11121 loss: 0.2119 2024/07/24 18:15:02 - mmengine - INFO - Iter(train) [ 5280/19224] lr: 1.7022e-05 eta: 10:04:28 time: 2.4369 data_time: 0.0101 memory: 10912 loss: 0.2380 2024/07/24 18:15:23 - mmengine - INFO - Iter(train) [ 5290/19224] lr: 1.7010e-05 eta: 10:03:50 time: 2.1657 data_time: 0.0092 memory: 10424 loss: 0.1876 2024/07/24 18:15:41 - mmengine - INFO - Iter(train) [ 5300/19224] lr: 1.6998e-05 eta: 10:03:02 time: 1.7363 data_time: 0.0093 memory: 10018 loss: 0.2011 2024/07/24 18:16:02 - mmengine - INFO - Iter(train) [ 5310/19224] lr: 1.6986e-05 eta: 10:02:24 time: 2.1634 data_time: 0.0091 memory: 12656 loss: 0.2167 2024/07/24 18:16:33 - mmengine - INFO - Iter(train) [ 5320/19224] lr: 1.6974e-05 eta: 10:02:11 time: 3.0761 data_time: 0.0106 memory: 11955 loss: 0.1824 2024/07/24 18:17:03 - mmengine - INFO - Iter(train) [ 5330/19224] lr: 1.6962e-05 eta: 10:01:56 time: 3.0208 data_time: 0.0112 memory: 12005 loss: 0.2033 2024/07/24 18:17:33 - mmengine - INFO - Iter(train) [ 5340/19224] lr: 1.6950e-05 eta: 10:01:38 time: 2.9325 data_time: 0.0106 memory: 11519 loss: 0.1966 2024/07/24 18:18:02 - mmengine - INFO - Iter(train) [ 5350/19224] lr: 1.6938e-05 eta: 10:01:21 time: 2.9121 data_time: 0.0107 memory: 11342 loss: 0.1941 2024/07/24 18:18:30 - mmengine - INFO - Iter(train) [ 5360/19224] lr: 1.6925e-05 eta: 10:01:00 time: 2.8112 data_time: 0.0105 memory: 11293 loss: 0.1885 2024/07/24 18:18:56 - mmengine - INFO - Iter(train) [ 5370/19224] lr: 1.6913e-05 eta: 10:00:36 time: 2.6733 data_time: 0.0106 memory: 11074 loss: 0.2109 2024/07/24 18:19:21 - mmengine - INFO - Iter(train) [ 5380/19224] lr: 1.6901e-05 eta: 10:00:06 time: 2.4599 data_time: 0.0104 memory: 10956 loss: 0.2224 2024/07/24 18:19:44 - mmengine - INFO - Iter(train) [ 5390/19224] lr: 1.6889e-05 eta: 9:59:32 time: 2.2955 data_time: 0.0099 memory: 10610 loss: 0.2698 2024/07/24 18:20:03 - mmengine - INFO - Iter(train) [ 5400/19224] lr: 1.6877e-05 eta: 9:58:48 time: 1.8856 data_time: 0.0102 memory: 10238 loss: 0.2385 2024/07/24 18:20:28 - mmengine - INFO - Iter(train) [ 5410/19224] lr: 1.6864e-05 eta: 9:58:19 time: 2.4706 data_time: 0.0099 memory: 13808 loss: 0.2085 2024/07/24 18:21:02 - mmengine - INFO - Iter(train) [ 5420/19224] lr: 1.6852e-05 eta: 9:58:13 time: 3.4074 data_time: 0.0112 memory: 12416 loss: 0.1663 2024/07/24 18:21:34 - mmengine - INFO - Iter(train) [ 5430/19224] lr: 1.6840e-05 eta: 9:58:02 time: 3.1884 data_time: 0.0104 memory: 12282 loss: 0.2044 2024/07/24 18:22:04 - mmengine - INFO - Iter(train) [ 5440/19224] lr: 1.6828e-05 eta: 9:57:48 time: 3.0762 data_time: 0.0106 memory: 12132 loss: 0.1841 2024/07/24 18:22:33 - mmengine - INFO - Iter(train) [ 5450/19224] lr: 1.6815e-05 eta: 9:57:29 time: 2.8812 data_time: 0.0104 memory: 11441 loss: 0.1912 2024/07/24 18:23:01 - mmengine - INFO - Iter(train) [ 5460/19224] lr: 1.6803e-05 eta: 9:57:08 time: 2.7912 data_time: 0.0104 memory: 11332 loss: 0.2033 2024/07/24 18:23:27 - mmengine - INFO - Iter(train) [ 5470/19224] lr: 1.6791e-05 eta: 9:56:43 time: 2.6376 data_time: 0.0104 memory: 11185 loss: 0.2084 2024/07/24 18:23:54 - mmengine - INFO - Iter(train) [ 5480/19224] lr: 1.6778e-05 eta: 9:56:17 time: 2.6123 data_time: 0.0109 memory: 11023 loss: 0.2477 2024/07/24 18:24:15 - mmengine - INFO - Iter(train) [ 5490/19224] lr: 1.6766e-05 eta: 9:55:41 time: 2.1917 data_time: 0.0104 memory: 10716 loss: 0.4277 2024/07/24 18:24:36 - mmengine - INFO - Iter(train) [ 5500/19224] lr: 1.6753e-05 eta: 9:55:02 time: 2.0845 data_time: 0.0093 memory: 9986 loss: 0.2145 2024/07/24 18:25:01 - mmengine - INFO - Iter(train) [ 5510/19224] lr: 1.6741e-05 eta: 9:54:33 time: 2.4700 data_time: 0.0097 memory: 13696 loss: 0.2069 2024/07/24 18:25:34 - mmengine - INFO - Iter(train) [ 5520/19224] lr: 1.6729e-05 eta: 9:54:24 time: 3.2923 data_time: 0.0106 memory: 12315 loss: 0.1733 2024/07/24 18:26:06 - mmengine - INFO - Iter(train) [ 5530/19224] lr: 1.6716e-05 eta: 9:54:13 time: 3.2082 data_time: 0.0118 memory: 11857 loss: 0.1941 2024/07/24 18:26:37 - mmengine - INFO - Iter(train) [ 5540/19224] lr: 1.6704e-05 eta: 9:53:58 time: 3.0706 data_time: 0.0108 memory: 12375 loss: 0.1816 2024/07/24 18:27:07 - mmengine - INFO - Iter(train) [ 5550/19224] lr: 1.6691e-05 eta: 9:53:42 time: 2.9942 data_time: 0.0106 memory: 11477 loss: 0.1883 2024/07/24 18:27:36 - mmengine - INFO - Iter(train) [ 5560/19224] lr: 1.6679e-05 eta: 9:53:25 time: 2.9776 data_time: 0.0124 memory: 11311 loss: 0.2125 2024/07/24 18:28:04 - mmengine - INFO - Iter(train) [ 5570/19224] lr: 1.6666e-05 eta: 9:53:01 time: 2.7124 data_time: 0.0108 memory: 11175 loss: 0.1793 2024/07/24 18:28:29 - mmengine - INFO - Iter(train) [ 5580/19224] lr: 1.6653e-05 eta: 9:52:34 time: 2.5560 data_time: 0.0112 memory: 10965 loss: 0.2672 2024/07/24 18:28:51 - mmengine - INFO - Iter(train) [ 5590/19224] lr: 1.6641e-05 eta: 9:51:59 time: 2.2260 data_time: 0.0108 memory: 10781 loss: 0.2209 2024/07/24 18:29:11 - mmengine - INFO - Iter(train) [ 5600/19224] lr: 1.6628e-05 eta: 9:51:16 time: 1.9215 data_time: 0.0100 memory: 10229 loss: 0.2315 2024/07/24 18:29:37 - mmengine - INFO - Iter(train) [ 5610/19224] lr: 1.6616e-05 eta: 9:50:51 time: 2.6425 data_time: 0.0094 memory: 18895 loss: 0.2146 2024/07/24 18:30:14 - mmengine - INFO - Iter(train) [ 5620/19224] lr: 1.6603e-05 eta: 9:50:51 time: 3.6633 data_time: 0.0107 memory: 12363 loss: 0.1838 2024/07/24 18:30:44 - mmengine - INFO - Iter(train) [ 5630/19224] lr: 1.6590e-05 eta: 9:50:35 time: 3.0363 data_time: 0.0105 memory: 11786 loss: 0.1790 2024/07/24 18:31:14 - mmengine - INFO - Iter(train) [ 5640/19224] lr: 1.6578e-05 eta: 9:50:18 time: 2.9673 data_time: 0.0108 memory: 11751 loss: 0.2127 2024/07/24 18:31:42 - mmengine - INFO - Iter(train) [ 5650/19224] lr: 1.6565e-05 eta: 9:49:58 time: 2.8728 data_time: 0.0106 memory: 11479 loss: 0.1859 2024/07/24 18:32:11 - mmengine - INFO - Iter(train) [ 5660/19224] lr: 1.6552e-05 eta: 9:49:38 time: 2.8727 data_time: 0.0100 memory: 11321 loss: 0.1879 2024/07/24 18:32:38 - mmengine - INFO - Iter(train) [ 5670/19224] lr: 1.6539e-05 eta: 9:49:13 time: 2.6591 data_time: 0.0102 memory: 11163 loss: 0.2233 2024/07/24 18:33:07 - mmengine - INFO - Iter(train) [ 5680/19224] lr: 1.6527e-05 eta: 9:48:55 time: 2.9275 data_time: 0.0115 memory: 11066 loss: 0.1830 2024/07/24 18:33:32 - mmengine - INFO - Iter(train) [ 5690/19224] lr: 1.6514e-05 eta: 9:48:26 time: 2.5053 data_time: 0.0099 memory: 10766 loss: 0.2130 2024/07/24 18:33:53 - mmengine - INFO - Iter(train) [ 5700/19224] lr: 1.6501e-05 eta: 9:47:49 time: 2.1191 data_time: 0.0096 memory: 10074 loss: 0.2008 2024/07/24 18:34:19 - mmengine - INFO - Iter(train) [ 5710/19224] lr: 1.6488e-05 eta: 9:47:22 time: 2.5917 data_time: 0.0094 memory: 15899 loss: 0.2475 2024/07/24 18:34:52 - mmengine - INFO - Iter(train) [ 5720/19224] lr: 1.6476e-05 eta: 9:47:11 time: 3.2387 data_time: 0.0110 memory: 12296 loss: 0.1714 2024/07/24 18:35:23 - mmengine - INFO - Iter(train) [ 5730/19224] lr: 1.6463e-05 eta: 9:46:58 time: 3.1721 data_time: 0.0105 memory: 12118 loss: 0.1946 2024/07/24 18:35:53 - mmengine - INFO - Iter(train) [ 5740/19224] lr: 1.6450e-05 eta: 9:46:41 time: 3.0009 data_time: 0.0108 memory: 11513 loss: 0.1758 2024/07/24 18:36:21 - mmengine - INFO - Iter(train) [ 5750/19224] lr: 1.6437e-05 eta: 9:46:19 time: 2.7883 data_time: 0.0109 memory: 11299 loss: 0.2120 2024/07/24 18:36:49 - mmengine - INFO - Iter(train) [ 5760/19224] lr: 1.6424e-05 eta: 9:45:57 time: 2.7984 data_time: 0.0105 memory: 11169 loss: 0.2019 2024/07/24 18:37:15 - mmengine - INFO - Iter(train) [ 5770/19224] lr: 1.6411e-05 eta: 9:45:30 time: 2.5716 data_time: 0.0103 memory: 11052 loss: 0.2555 2024/07/24 18:37:38 - mmengine - INFO - Iter(train) [ 5780/19224] lr: 1.6398e-05 eta: 9:44:58 time: 2.3467 data_time: 0.0100 memory: 10878 loss: 0.2142 2024/07/24 18:38:01 - mmengine - INFO - Iter(train) [ 5790/19224] lr: 1.6385e-05 eta: 9:44:23 time: 2.2347 data_time: 0.0094 memory: 10483 loss: 0.2071 2024/07/24 18:38:22 - mmengine - INFO - Iter(train) [ 5800/19224] lr: 1.6372e-05 eta: 9:43:46 time: 2.1219 data_time: 0.0095 memory: 10289 loss: 0.2296 2024/07/24 18:38:44 - mmengine - INFO - Iter(train) [ 5810/19224] lr: 1.6359e-05 eta: 9:43:11 time: 2.2182 data_time: 0.0093 memory: 13055 loss: 0.2207 2024/07/24 18:39:16 - mmengine - INFO - Iter(train) [ 5820/19224] lr: 1.6346e-05 eta: 9:42:57 time: 3.1444 data_time: 0.0106 memory: 12217 loss: 0.1881 2024/07/24 18:39:47 - mmengine - INFO - Iter(train) [ 5830/19224] lr: 1.6333e-05 eta: 9:42:43 time: 3.1538 data_time: 0.0106 memory: 12187 loss: 0.1847 2024/07/24 18:40:18 - mmengine - INFO - Iter(train) [ 5840/19224] lr: 1.6320e-05 eta: 9:42:29 time: 3.0972 data_time: 0.0111 memory: 12066 loss: 0.2044 2024/07/24 18:40:47 - mmengine - INFO - Iter(train) [ 5850/19224] lr: 1.6307e-05 eta: 9:42:09 time: 2.8791 data_time: 0.0111 memory: 11513 loss: 0.2046 2024/07/24 18:41:15 - mmengine - INFO - Iter(train) [ 5860/19224] lr: 1.6294e-05 eta: 9:41:46 time: 2.7795 data_time: 0.0109 memory: 11334 loss: 0.1897 2024/07/24 18:41:42 - mmengine - INFO - Iter(train) [ 5870/19224] lr: 1.6281e-05 eta: 9:41:22 time: 2.6990 data_time: 0.0112 memory: 11169 loss: 0.1963 2024/07/24 18:42:06 - mmengine - INFO - Iter(train) [ 5880/19224] lr: 1.6268e-05 eta: 9:40:52 time: 2.4299 data_time: 0.0098 memory: 11024 loss: 0.2168 2024/07/24 18:42:28 - mmengine - INFO - Iter(train) [ 5890/19224] lr: 1.6255e-05 eta: 9:40:16 time: 2.1976 data_time: 0.0099 memory: 10741 loss: 0.2284 2024/07/24 18:42:46 - mmengine - INFO - Iter(train) [ 5900/19224] lr: 1.6241e-05 eta: 9:39:33 time: 1.8521 data_time: 0.0091 memory: 10178 loss: 0.2555 2024/07/24 18:43:09 - mmengine - INFO - Iter(train) [ 5910/19224] lr: 1.6228e-05 eta: 9:38:58 time: 2.2132 data_time: 0.0086 memory: 15201 loss: 0.2136 2024/07/24 18:43:41 - mmengine - INFO - Iter(train) [ 5920/19224] lr: 1.6215e-05 eta: 9:38:46 time: 3.2549 data_time: 0.0111 memory: 12489 loss: 0.2330 2024/07/24 18:44:11 - mmengine - INFO - Iter(train) [ 5930/19224] lr: 1.6202e-05 eta: 9:38:29 time: 2.9942 data_time: 0.0103 memory: 11910 loss: 0.2038 2024/07/24 18:44:40 - mmengine - INFO - Iter(train) [ 5940/19224] lr: 1.6189e-05 eta: 9:38:10 time: 2.9302 data_time: 0.0105 memory: 11528 loss: 0.2378 2024/07/24 18:45:11 - mmengine - INFO - Iter(train) [ 5950/19224] lr: 1.6175e-05 eta: 9:37:54 time: 3.0594 data_time: 0.0114 memory: 11499 loss: 0.2145 2024/07/24 18:45:40 - mmengine - INFO - Iter(train) [ 5960/19224] lr: 1.6162e-05 eta: 9:37:34 time: 2.9001 data_time: 0.0107 memory: 11287 loss: 0.2315 2024/07/24 18:46:07 - mmengine - INFO - Iter(train) [ 5970/19224] lr: 1.6149e-05 eta: 9:37:10 time: 2.6985 data_time: 0.0104 memory: 11557 loss: 0.1934 2024/07/24 18:46:32 - mmengine - INFO - Iter(train) [ 5980/19224] lr: 1.6136e-05 eta: 9:36:42 time: 2.5287 data_time: 0.0106 memory: 11028 loss: 0.2288 2024/07/24 18:46:56 - mmengine - INFO - Iter(train) [ 5990/19224] lr: 1.6122e-05 eta: 9:36:12 time: 2.4224 data_time: 0.0101 memory: 10779 loss: 0.2141 2024/07/24 18:47:16 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 18:47:16 - mmengine - INFO - Iter(train) [ 6000/19224] lr: 1.6109e-05 eta: 9:35:30 time: 1.9133 data_time: 0.0095 memory: 10494 loss: 0.2056 2024/07/24 18:47:16 - mmengine - INFO - Saving checkpoint at 6000 iterations 2024/07/24 18:47:42 - mmengine - INFO - Iter(train) [ 6010/19224] lr: 1.6096e-05 eta: 9:35:04 time: 2.6031 data_time: 0.2009 memory: 13967 loss: 0.2565 2024/07/24 18:48:16 - mmengine - INFO - Iter(train) [ 6020/19224] lr: 1.6082e-05 eta: 9:34:56 time: 3.4554 data_time: 0.0105 memory: 12857 loss: 0.1903 2024/07/24 18:48:50 - mmengine - INFO - Iter(train) [ 6030/19224] lr: 1.6069e-05 eta: 9:34:46 time: 3.3401 data_time: 0.0109 memory: 12012 loss: 0.1992 2024/07/24 18:49:28 - mmengine - INFO - Iter(train) [ 6040/19224] lr: 1.6055e-05 eta: 9:34:47 time: 3.8435 data_time: 0.0108 memory: 11636 loss: 0.1787 2024/07/24 18:50:00 - mmengine - INFO - Iter(train) [ 6050/19224] lr: 1.6042e-05 eta: 9:34:34 time: 3.2417 data_time: 0.0106 memory: 11311 loss: 0.2092 2024/07/24 18:50:29 - mmengine - INFO - Iter(train) [ 6060/19224] lr: 1.6029e-05 eta: 9:34:13 time: 2.8576 data_time: 0.0116 memory: 11195 loss: 0.2043 2024/07/24 18:50:55 - mmengine - INFO - Iter(train) [ 6070/19224] lr: 1.6015e-05 eta: 9:33:47 time: 2.6127 data_time: 0.0107 memory: 11020 loss: 0.2626 2024/07/24 18:51:19 - mmengine - INFO - Iter(train) [ 6080/19224] lr: 1.6002e-05 eta: 9:33:16 time: 2.3831 data_time: 0.0105 memory: 10727 loss: 0.2540 2024/07/24 18:51:39 - mmengine - INFO - Iter(train) [ 6090/19224] lr: 1.5988e-05 eta: 9:32:35 time: 1.9628 data_time: 0.0094 memory: 10245 loss: 0.2216 2024/07/24 18:51:57 - mmengine - INFO - Iter(train) [ 6100/19224] lr: 1.5975e-05 eta: 9:31:52 time: 1.8069 data_time: 0.0092 memory: 10042 loss: 0.2061 2024/07/24 18:52:21 - mmengine - INFO - Iter(train) [ 6110/19224] lr: 1.5961e-05 eta: 9:31:21 time: 2.4165 data_time: 0.0097 memory: 14669 loss: 0.2326 2024/07/24 18:52:54 - mmengine - INFO - Iter(train) [ 6120/19224] lr: 1.5948e-05 eta: 9:31:11 time: 3.3434 data_time: 0.0107 memory: 13120 loss: 0.1889 2024/07/24 18:53:27 - mmengine - INFO - Iter(train) [ 6130/19224] lr: 1.5934e-05 eta: 9:30:59 time: 3.2821 data_time: 0.0105 memory: 12158 loss: 0.2115 2024/07/24 18:53:57 - mmengine - INFO - Iter(train) [ 6140/19224] lr: 1.5921e-05 eta: 9:30:41 time: 2.9749 data_time: 0.0117 memory: 11652 loss: 0.2002 2024/07/24 18:54:25 - mmengine - INFO - Iter(train) [ 6150/19224] lr: 1.5907e-05 eta: 9:30:19 time: 2.8252 data_time: 0.0107 memory: 11450 loss: 0.2146 2024/07/24 18:54:54 - mmengine - INFO - Iter(train) [ 6160/19224] lr: 1.5893e-05 eta: 9:29:58 time: 2.8712 data_time: 0.0104 memory: 11224 loss: 0.2251 2024/07/24 18:55:24 - mmengine - INFO - Iter(train) [ 6170/19224] lr: 1.5880e-05 eta: 9:29:40 time: 3.0056 data_time: 0.0113 memory: 11081 loss: 0.1944 2024/07/24 18:55:50 - mmengine - INFO - Iter(train) [ 6180/19224] lr: 1.5866e-05 eta: 9:29:15 time: 2.6587 data_time: 0.0104 memory: 10892 loss: 0.3548 2024/07/24 18:56:14 - mmengine - INFO - Iter(train) [ 6190/19224] lr: 1.5852e-05 eta: 9:28:43 time: 2.3701 data_time: 0.0091 memory: 10504 loss: 0.2203 2024/07/24 18:56:34 - mmengine - INFO - Iter(train) [ 6200/19224] lr: 1.5839e-05 eta: 9:28:03 time: 1.9652 data_time: 0.0095 memory: 10102 loss: 0.2036 2024/07/24 18:56:58 - mmengine - INFO - Iter(train) [ 6210/19224] lr: 1.5825e-05 eta: 9:27:34 time: 2.4573 data_time: 0.0091 memory: 15275 loss: 0.2301 2024/07/24 18:57:31 - mmengine - INFO - Iter(train) [ 6220/19224] lr: 1.5811e-05 eta: 9:27:22 time: 3.2772 data_time: 0.0120 memory: 12222 loss: 0.1902 2024/07/24 18:58:02 - mmengine - INFO - Iter(train) [ 6230/19224] lr: 1.5798e-05 eta: 9:27:06 time: 3.1283 data_time: 0.0122 memory: 11818 loss: 0.1758 2024/07/24 18:58:33 - mmengine - INFO - Iter(train) [ 6240/19224] lr: 1.5784e-05 eta: 9:26:48 time: 3.0355 data_time: 0.0107 memory: 11600 loss: 0.1895 2024/07/24 18:59:02 - mmengine - INFO - Iter(train) [ 6250/19224] lr: 1.5770e-05 eta: 9:26:29 time: 2.9375 data_time: 0.0107 memory: 11446 loss: 0.2213 2024/07/24 18:59:29 - mmengine - INFO - Iter(train) [ 6260/19224] lr: 1.5756e-05 eta: 9:26:03 time: 2.6406 data_time: 0.0108 memory: 11297 loss: 0.2244 2024/07/24 18:59:55 - mmengine - INFO - Iter(train) [ 6270/19224] lr: 1.5743e-05 eta: 9:25:37 time: 2.6476 data_time: 0.0108 memory: 11210 loss: 0.1979 2024/07/24 19:00:23 - mmengine - INFO - Iter(train) [ 6280/19224] lr: 1.5729e-05 eta: 9:25:15 time: 2.8193 data_time: 0.0111 memory: 11050 loss: 0.2151 2024/07/24 19:00:42 - mmengine - INFO - Iter(train) [ 6290/19224] lr: 1.5715e-05 eta: 9:24:34 time: 1.9019 data_time: 0.0091 memory: 10472 loss: 0.2817 2024/07/24 19:00:59 - mmengine - INFO - Iter(train) [ 6300/19224] lr: 1.5701e-05 eta: 9:23:49 time: 1.6898 data_time: 0.0094 memory: 9856 loss: 0.2106 2024/07/24 19:01:25 - mmengine - INFO - Iter(train) [ 6310/19224] lr: 1.5687e-05 eta: 9:23:22 time: 2.5931 data_time: 0.0090 memory: 18443 loss: 0.2407 2024/07/24 19:01:58 - mmengine - INFO - Iter(train) [ 6320/19224] lr: 1.5673e-05 eta: 9:23:10 time: 3.3026 data_time: 0.0106 memory: 12764 loss: 0.2010 2024/07/24 19:02:27 - mmengine - INFO - Iter(train) [ 6330/19224] lr: 1.5660e-05 eta: 9:22:50 time: 2.9131 data_time: 0.0103 memory: 12203 loss: 0.1899 2024/07/24 19:02:56 - mmengine - INFO - Iter(train) [ 6340/19224] lr: 1.5646e-05 eta: 9:22:30 time: 2.9156 data_time: 0.0106 memory: 11477 loss: 0.1877 2024/07/24 19:03:24 - mmengine - INFO - Iter(train) [ 6350/19224] lr: 1.5632e-05 eta: 9:22:06 time: 2.7249 data_time: 0.0105 memory: 11368 loss: 0.2225 2024/07/24 19:03:51 - mmengine - INFO - Iter(train) [ 6360/19224] lr: 1.5618e-05 eta: 9:21:41 time: 2.6932 data_time: 0.0105 memory: 11196 loss: 0.2663 2024/07/24 19:04:16 - mmengine - INFO - Iter(train) [ 6370/19224] lr: 1.5604e-05 eta: 9:21:14 time: 2.5830 data_time: 0.0103 memory: 11039 loss: 0.2256 2024/07/24 19:04:39 - mmengine - INFO - Iter(train) [ 6380/19224] lr: 1.5590e-05 eta: 9:20:41 time: 2.2847 data_time: 0.0096 memory: 10808 loss: 0.2210 2024/07/24 19:04:59 - mmengine - INFO - Iter(train) [ 6390/19224] lr: 1.5576e-05 eta: 9:20:03 time: 2.0072 data_time: 0.0092 memory: 10197 loss: 0.2246 2024/07/24 19:05:18 - mmengine - INFO - Iter(train) [ 6400/19224] lr: 1.5562e-05 eta: 9:19:21 time: 1.8348 data_time: 0.0093 memory: 10204 loss: 0.2056 2024/07/24 19:05:40 - mmengine - INFO - Iter(train) [ 6410/19224] lr: 1.5548e-05 eta: 9:18:46 time: 2.1983 data_time: 0.0093 memory: 13218 loss: 0.2200 2024/07/24 19:06:12 - mmengine - INFO - Iter(train) [ 6420/19224] lr: 1.5534e-05 eta: 9:18:32 time: 3.1952 data_time: 0.0108 memory: 12178 loss: 0.1862 2024/07/24 19:06:44 - mmengine - INFO - Iter(train) [ 6430/19224] lr: 1.5520e-05 eta: 9:18:18 time: 3.2434 data_time: 0.0105 memory: 11926 loss: 0.2051 2024/07/24 19:07:13 - mmengine - INFO - Iter(train) [ 6440/19224] lr: 1.5506e-05 eta: 9:17:58 time: 2.9444 data_time: 0.0108 memory: 11622 loss: 0.2053 2024/07/24 19:07:44 - mmengine - INFO - Iter(train) [ 6450/19224] lr: 1.5492e-05 eta: 9:17:40 time: 3.0365 data_time: 0.0104 memory: 11395 loss: 0.2410 2024/07/24 19:08:12 - mmengine - INFO - Iter(train) [ 6460/19224] lr: 1.5478e-05 eta: 9:17:18 time: 2.8084 data_time: 0.0106 memory: 11291 loss: 0.1834 2024/07/24 19:08:39 - mmengine - INFO - Iter(train) [ 6470/19224] lr: 1.5464e-05 eta: 9:16:54 time: 2.7357 data_time: 0.0108 memory: 11149 loss: 0.2344 2024/07/24 19:09:04 - mmengine - INFO - Iter(train) [ 6480/19224] lr: 1.5449e-05 eta: 9:16:26 time: 2.5158 data_time: 0.0105 memory: 11003 loss: 0.2230 2024/07/24 19:09:26 - mmengine - INFO - Iter(train) [ 6490/19224] lr: 1.5435e-05 eta: 9:15:50 time: 2.1495 data_time: 0.0100 memory: 10595 loss: 0.2840 2024/07/24 19:09:46 - mmengine - INFO - Iter(train) [ 6500/19224] lr: 1.5421e-05 eta: 9:15:13 time: 2.0258 data_time: 0.0101 memory: 10136 loss: 0.1996 2024/07/24 19:10:10 - mmengine - INFO - Iter(train) [ 6510/19224] lr: 1.5407e-05 eta: 9:14:41 time: 2.3489 data_time: 0.0094 memory: 15311 loss: 0.1911 2024/07/24 19:10:41 - mmengine - INFO - Iter(train) [ 6520/19224] lr: 1.5393e-05 eta: 9:14:25 time: 3.1391 data_time: 0.0109 memory: 12239 loss: 0.1789 2024/07/24 19:11:11 - mmengine - INFO - Iter(train) [ 6530/19224] lr: 1.5379e-05 eta: 9:14:06 time: 2.9809 data_time: 0.0108 memory: 11798 loss: 0.1776 2024/07/24 19:11:39 - mmengine - INFO - Iter(train) [ 6540/19224] lr: 1.5364e-05 eta: 9:13:44 time: 2.8156 data_time: 0.0110 memory: 11580 loss: 0.2044 2024/07/24 19:12:07 - mmengine - INFO - Iter(train) [ 6550/19224] lr: 1.5350e-05 eta: 9:13:20 time: 2.7579 data_time: 0.0118 memory: 11288 loss: 0.1795 2024/07/24 19:12:34 - mmengine - INFO - Iter(train) [ 6560/19224] lr: 1.5336e-05 eta: 9:12:56 time: 2.7172 data_time: 0.0111 memory: 11299 loss: 0.2483 2024/07/24 19:12:59 - mmengine - INFO - Iter(train) [ 6570/19224] lr: 1.5322e-05 eta: 9:12:28 time: 2.5239 data_time: 0.0103 memory: 10945 loss: 0.2173 2024/07/24 19:13:22 - mmengine - INFO - Iter(train) [ 6580/19224] lr: 1.5307e-05 eta: 9:11:56 time: 2.3343 data_time: 0.0101 memory: 10852 loss: 0.2278 2024/07/24 19:13:43 - mmengine - INFO - Iter(train) [ 6590/19224] lr: 1.5293e-05 eta: 9:11:19 time: 2.0428 data_time: 0.0093 memory: 10379 loss: 0.2135 2024/07/24 19:14:01 - mmengine - INFO - Iter(train) [ 6600/19224] lr: 1.5279e-05 eta: 9:10:37 time: 1.7830 data_time: 0.0090 memory: 9942 loss: 0.2189 2024/07/24 19:14:30 - mmengine - INFO - Iter(train) [ 6610/19224] lr: 1.5265e-05 eta: 9:10:16 time: 2.8904 data_time: 0.0093 memory: 14523 loss: 0.2307 2024/07/24 19:15:04 - mmengine - INFO - Iter(train) [ 6620/19224] lr: 1.5250e-05 eta: 9:10:05 time: 3.4470 data_time: 0.0109 memory: 13453 loss: 0.2006 2024/07/24 19:15:36 - mmengine - INFO - Iter(train) [ 6630/19224] lr: 1.5236e-05 eta: 9:09:51 time: 3.2341 data_time: 0.0109 memory: 11919 loss: 0.1896 2024/07/24 19:16:07 - mmengine - INFO - Iter(train) [ 6640/19224] lr: 1.5222e-05 eta: 9:09:33 time: 3.0779 data_time: 0.0114 memory: 11702 loss: 0.2008 2024/07/24 19:16:36 - mmengine - INFO - Iter(train) [ 6650/19224] lr: 1.5207e-05 eta: 9:09:12 time: 2.8796 data_time: 0.0110 memory: 11475 loss: 0.1850 2024/07/24 19:17:02 - mmengine - INFO - Iter(train) [ 6660/19224] lr: 1.5193e-05 eta: 9:08:46 time: 2.6225 data_time: 0.0106 memory: 11260 loss: 0.2192 2024/07/24 19:17:28 - mmengine - INFO - Iter(train) [ 6670/19224] lr: 1.5178e-05 eta: 9:08:18 time: 2.5515 data_time: 0.0118 memory: 10970 loss: 0.2297 2024/07/24 19:17:52 - mmengine - INFO - Iter(train) [ 6680/19224] lr: 1.5164e-05 eta: 9:07:48 time: 2.4095 data_time: 0.0102 memory: 10856 loss: 0.2276 2024/07/24 19:18:14 - mmengine - INFO - Iter(train) [ 6690/19224] lr: 1.5150e-05 eta: 9:07:14 time: 2.1827 data_time: 0.0098 memory: 10720 loss: 0.2230 2024/07/24 19:18:31 - mmengine - INFO - Iter(train) [ 6700/19224] lr: 1.5135e-05 eta: 9:06:32 time: 1.7846 data_time: 0.0089 memory: 10149 loss: 0.2391 2024/07/24 19:18:53 - mmengine - INFO - Iter(train) [ 6710/19224] lr: 1.5121e-05 eta: 9:05:58 time: 2.2047 data_time: 0.0088 memory: 13517 loss: 0.2037 2024/07/24 19:19:27 - mmengine - INFO - Iter(train) [ 6720/19224] lr: 1.5106e-05 eta: 9:05:46 time: 3.3690 data_time: 0.0105 memory: 12366 loss: 0.1858 2024/07/24 19:19:58 - mmengine - INFO - Iter(train) [ 6730/19224] lr: 1.5092e-05 eta: 9:05:29 time: 3.1083 data_time: 0.0109 memory: 11987 loss: 0.1877 2024/07/24 19:20:28 - mmengine - INFO - Iter(train) [ 6740/19224] lr: 1.5077e-05 eta: 9:05:09 time: 2.9699 data_time: 0.0104 memory: 11729 loss: 0.1801 2024/07/24 19:20:57 - mmengine - INFO - Iter(train) [ 6750/19224] lr: 1.5063e-05 eta: 9:04:48 time: 2.9191 data_time: 0.0103 memory: 11415 loss: 0.1936 2024/07/24 19:21:25 - mmengine - INFO - Iter(train) [ 6760/19224] lr: 1.5048e-05 eta: 9:04:25 time: 2.7788 data_time: 0.0105 memory: 11182 loss: 0.2099 2024/07/24 19:21:51 - mmengine - INFO - Iter(train) [ 6770/19224] lr: 1.5034e-05 eta: 9:03:59 time: 2.6302 data_time: 0.0117 memory: 10991 loss: 0.2055 2024/07/24 19:22:16 - mmengine - INFO - Iter(train) [ 6780/19224] lr: 1.5019e-05 eta: 9:03:30 time: 2.4781 data_time: 0.0100 memory: 10867 loss: 0.2050 2024/07/24 19:22:38 - mmengine - INFO - Iter(train) [ 6790/19224] lr: 1.5004e-05 eta: 9:02:56 time: 2.1691 data_time: 0.0094 memory: 10450 loss: 0.2286 2024/07/24 19:22:56 - mmengine - INFO - Iter(train) [ 6800/19224] lr: 1.4990e-05 eta: 9:02:15 time: 1.8462 data_time: 0.0091 memory: 10134 loss: 0.2430 2024/07/24 19:23:20 - mmengine - INFO - Iter(train) [ 6810/19224] lr: 1.4975e-05 eta: 9:01:45 time: 2.4091 data_time: 0.0092 memory: 14428 loss: 0.2256 2024/07/24 19:23:53 - mmengine - INFO - Iter(train) [ 6820/19224] lr: 1.4961e-05 eta: 9:01:31 time: 3.2409 data_time: 0.0120 memory: 12564 loss: 0.1841 2024/07/24 19:24:23 - mmengine - INFO - Iter(train) [ 6830/19224] lr: 1.4946e-05 eta: 9:01:11 time: 3.0075 data_time: 0.0106 memory: 11691 loss: 0.1821 2024/07/24 19:24:52 - mmengine - INFO - Iter(train) [ 6840/19224] lr: 1.4931e-05 eta: 9:00:52 time: 2.9773 data_time: 0.0103 memory: 11499 loss: 0.2339 2024/07/24 19:25:20 - mmengine - INFO - Iter(train) [ 6850/19224] lr: 1.4917e-05 eta: 9:00:28 time: 2.7347 data_time: 0.0104 memory: 11349 loss: 0.2092 2024/07/24 19:25:47 - mmengine - INFO - Iter(train) [ 6860/19224] lr: 1.4902e-05 eta: 9:00:03 time: 2.7393 data_time: 0.0111 memory: 11230 loss: 0.1865 2024/07/24 19:26:14 - mmengine - INFO - Iter(train) [ 6870/19224] lr: 1.4887e-05 eta: 8:59:38 time: 2.6432 data_time: 0.0105 memory: 11056 loss: 0.2300 2024/07/24 19:26:38 - mmengine - INFO - Iter(train) [ 6880/19224] lr: 1.4873e-05 eta: 8:59:08 time: 2.4227 data_time: 0.0102 memory: 10846 loss: 0.2493 2024/07/24 19:27:00 - mmengine - INFO - Iter(train) [ 6890/19224] lr: 1.4858e-05 eta: 8:58:34 time: 2.1671 data_time: 0.0095 memory: 10552 loss: 0.2267 2024/07/24 19:27:19 - mmengine - INFO - Iter(train) [ 6900/19224] lr: 1.4843e-05 eta: 8:57:55 time: 1.9309 data_time: 0.0092 memory: 10199 loss: 0.2108 2024/07/24 19:27:39 - mmengine - INFO - Iter(train) [ 6910/19224] lr: 1.4828e-05 eta: 8:57:18 time: 2.0317 data_time: 0.0093 memory: 13309 loss: 0.2632 2024/07/24 19:28:13 - mmengine - INFO - Iter(train) [ 6920/19224] lr: 1.4814e-05 eta: 8:57:06 time: 3.3743 data_time: 0.0105 memory: 12265 loss: 0.1809 2024/07/24 19:28:44 - mmengine - INFO - Iter(train) [ 6930/19224] lr: 1.4799e-05 eta: 8:56:48 time: 3.0932 data_time: 0.0104 memory: 11984 loss: 0.1842 2024/07/24 19:29:13 - mmengine - INFO - Iter(train) [ 6940/19224] lr: 1.4784e-05 eta: 8:56:27 time: 2.9247 data_time: 0.0107 memory: 11593 loss: 0.2264 2024/07/24 19:29:42 - mmengine - INFO - Iter(train) [ 6950/19224] lr: 1.4769e-05 eta: 8:56:06 time: 2.9046 data_time: 0.0103 memory: 11306 loss: 0.2022 2024/07/24 19:30:12 - mmengine - INFO - Iter(train) [ 6960/19224] lr: 1.4754e-05 eta: 8:55:47 time: 3.0189 data_time: 0.0107 memory: 11233 loss: 0.1991 2024/07/24 19:30:42 - mmengine - INFO - Iter(train) [ 6970/19224] lr: 1.4740e-05 eta: 8:55:26 time: 2.9348 data_time: 0.0111 memory: 11160 loss: 0.2009 2024/07/24 19:31:09 - mmengine - INFO - Iter(train) [ 6980/19224] lr: 1.4725e-05 eta: 8:55:01 time: 2.7026 data_time: 0.0108 memory: 11060 loss: 0.2100 2024/07/24 19:31:31 - mmengine - INFO - Iter(train) [ 6990/19224] lr: 1.4710e-05 eta: 8:54:27 time: 2.1830 data_time: 0.0099 memory: 10628 loss: 0.3056 2024/07/24 19:31:48 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 19:31:48 - mmengine - INFO - Iter(train) [ 7000/19224] lr: 1.4695e-05 eta: 8:53:46 time: 1.7531 data_time: 0.0094 memory: 10189 loss: 0.2148 2024/07/24 19:31:48 - mmengine - INFO - Saving checkpoint at 7000 iterations 2024/07/24 19:32:14 - mmengine - INFO - Iter(train) [ 7010/19224] lr: 1.4680e-05 eta: 8:53:19 time: 2.5607 data_time: 0.1924 memory: 16069 loss: 0.1835 2024/07/24 19:32:46 - mmengine - INFO - Iter(train) [ 7020/19224] lr: 1.4665e-05 eta: 8:53:04 time: 3.2590 data_time: 0.0118 memory: 12390 loss: 0.1702 2024/07/24 19:33:17 - mmengine - INFO - Iter(train) [ 7030/19224] lr: 1.4650e-05 eta: 8:52:44 time: 3.0253 data_time: 0.0108 memory: 11970 loss: 0.2010 2024/07/24 19:33:45 - mmengine - INFO - Iter(train) [ 7040/19224] lr: 1.4635e-05 eta: 8:52:22 time: 2.8237 data_time: 0.0103 memory: 11698 loss: 0.1863 2024/07/24 19:34:18 - mmengine - INFO - Iter(train) [ 7050/19224] lr: 1.4620e-05 eta: 8:52:08 time: 3.3530 data_time: 0.0102 memory: 11470 loss: 0.1785 2024/07/24 19:34:51 - mmengine - INFO - Iter(train) [ 7060/19224] lr: 1.4606e-05 eta: 8:51:52 time: 3.2254 data_time: 0.0107 memory: 11353 loss: 0.1690 2024/07/24 19:35:22 - mmengine - INFO - Iter(train) [ 7070/19224] lr: 1.4591e-05 eta: 8:51:34 time: 3.1019 data_time: 0.0101 memory: 11157 loss: 0.2139 2024/07/24 19:35:49 - mmengine - INFO - Iter(train) [ 7080/19224] lr: 1.4576e-05 eta: 8:51:10 time: 2.7581 data_time: 0.0105 memory: 11050 loss: 0.2132 2024/07/24 19:36:15 - mmengine - INFO - Iter(train) [ 7090/19224] lr: 1.4561e-05 eta: 8:50:43 time: 2.5563 data_time: 0.0102 memory: 10963 loss: 0.2252 2024/07/24 19:36:36 - mmengine - INFO - Iter(train) [ 7100/19224] lr: 1.4546e-05 eta: 8:50:08 time: 2.0948 data_time: 0.0096 memory: 10437 loss: 0.2201 2024/07/24 19:36:58 - mmengine - INFO - Iter(train) [ 7110/19224] lr: 1.4531e-05 eta: 8:49:35 time: 2.2740 data_time: 0.0089 memory: 14357 loss: 0.2532 2024/07/24 19:37:31 - mmengine - INFO - Iter(train) [ 7120/19224] lr: 1.4516e-05 eta: 8:49:20 time: 3.2798 data_time: 0.0105 memory: 12540 loss: 0.1668 2024/07/24 19:38:03 - mmengine - INFO - Iter(train) [ 7130/19224] lr: 1.4501e-05 eta: 8:49:03 time: 3.1396 data_time: 0.0103 memory: 11839 loss: 0.1957 2024/07/24 19:38:32 - mmengine - INFO - Iter(train) [ 7140/19224] lr: 1.4485e-05 eta: 8:48:42 time: 2.9593 data_time: 0.0103 memory: 11550 loss: 0.1839 2024/07/24 19:39:00 - mmengine - INFO - Iter(train) [ 7150/19224] lr: 1.4470e-05 eta: 8:48:19 time: 2.8214 data_time: 0.0105 memory: 11333 loss: 0.2296 2024/07/24 19:39:28 - mmengine - INFO - Iter(train) [ 7160/19224] lr: 1.4455e-05 eta: 8:47:56 time: 2.7956 data_time: 0.0107 memory: 11318 loss: 0.1859 2024/07/24 19:39:55 - mmengine - INFO - Iter(train) [ 7170/19224] lr: 1.4440e-05 eta: 8:47:31 time: 2.6950 data_time: 0.0104 memory: 11022 loss: 0.1873 2024/07/24 19:40:20 - mmengine - INFO - Iter(train) [ 7180/19224] lr: 1.4425e-05 eta: 8:47:01 time: 2.4295 data_time: 0.0105 memory: 10806 loss: 0.2326 2024/07/24 19:40:40 - mmengine - INFO - Iter(train) [ 7190/19224] lr: 1.4410e-05 eta: 8:46:26 time: 2.0803 data_time: 0.0091 memory: 10409 loss: 0.2099 2024/07/24 19:40:59 - mmengine - INFO - Iter(train) [ 7200/19224] lr: 1.4395e-05 eta: 8:45:47 time: 1.8533 data_time: 0.0089 memory: 10129 loss: 0.2597 2024/07/24 19:41:21 - mmengine - INFO - Iter(train) [ 7210/19224] lr: 1.4380e-05 eta: 8:45:14 time: 2.2096 data_time: 0.0087 memory: 14347 loss: 0.1970 2024/07/24 19:41:54 - mmengine - INFO - Iter(train) [ 7220/19224] lr: 1.4365e-05 eta: 8:44:59 time: 3.3315 data_time: 0.0113 memory: 12504 loss: 0.3025 2024/07/24 19:42:26 - mmengine - INFO - Iter(train) [ 7230/19224] lr: 1.4349e-05 eta: 8:44:41 time: 3.1261 data_time: 0.0119 memory: 11924 loss: 0.1684 2024/07/24 19:42:56 - mmengine - INFO - Iter(train) [ 7240/19224] lr: 1.4334e-05 eta: 8:44:22 time: 3.0538 data_time: 0.0119 memory: 11600 loss: 0.1906 2024/07/24 19:43:24 - mmengine - INFO - Iter(train) [ 7250/19224] lr: 1.4319e-05 eta: 8:43:59 time: 2.7870 data_time: 0.0121 memory: 11550 loss: 0.1800 2024/07/24 19:43:52 - mmengine - INFO - Iter(train) [ 7260/19224] lr: 1.4304e-05 eta: 8:43:36 time: 2.8378 data_time: 0.0114 memory: 11301 loss: 0.2225 2024/07/24 19:44:19 - mmengine - INFO - Iter(train) [ 7270/19224] lr: 1.4289e-05 eta: 8:43:10 time: 2.6348 data_time: 0.0117 memory: 11106 loss: 0.2234 2024/07/24 19:44:43 - mmengine - INFO - Iter(train) [ 7280/19224] lr: 1.4273e-05 eta: 8:42:39 time: 2.3806 data_time: 0.0105 memory: 10835 loss: 0.2589 2024/07/24 19:45:06 - mmengine - INFO - Iter(train) [ 7290/19224] lr: 1.4258e-05 eta: 8:42:09 time: 2.3498 data_time: 0.0100 memory: 10603 loss: 0.2326 2024/07/24 19:45:26 - mmengine - INFO - Iter(train) [ 7300/19224] lr: 1.4243e-05 eta: 8:41:31 time: 1.9493 data_time: 0.0093 memory: 10348 loss: 0.2076 2024/07/24 19:45:52 - mmengine - INFO - Iter(train) [ 7310/19224] lr: 1.4228e-05 eta: 8:41:05 time: 2.6128 data_time: 0.0093 memory: 16719 loss: 0.2269 2024/07/24 19:46:25 - mmengine - INFO - Iter(train) [ 7320/19224] lr: 1.4212e-05 eta: 8:40:49 time: 3.2819 data_time: 0.0103 memory: 12363 loss: 0.1843 2024/07/24 19:46:54 - mmengine - INFO - Iter(train) [ 7330/19224] lr: 1.4197e-05 eta: 8:40:29 time: 2.9683 data_time: 0.0111 memory: 11917 loss: 0.1751 2024/07/24 19:47:25 - mmengine - INFO - Iter(train) [ 7340/19224] lr: 1.4182e-05 eta: 8:40:09 time: 3.0389 data_time: 0.0104 memory: 11709 loss: 0.1696 2024/07/24 19:47:53 - mmengine - INFO - Iter(train) [ 7350/19224] lr: 1.4167e-05 eta: 8:39:47 time: 2.8647 data_time: 0.0105 memory: 11499 loss: 0.2102 2024/07/24 19:48:20 - mmengine - INFO - Iter(train) [ 7360/19224] lr: 1.4151e-05 eta: 8:39:22 time: 2.7032 data_time: 0.0103 memory: 11273 loss: 0.2184 2024/07/24 19:48:47 - mmengine - INFO - Iter(train) [ 7370/19224] lr: 1.4136e-05 eta: 8:38:57 time: 2.7235 data_time: 0.0101 memory: 11189 loss: 0.2175 2024/07/24 19:49:13 - mmengine - INFO - Iter(train) [ 7380/19224] lr: 1.4121e-05 eta: 8:38:29 time: 2.5327 data_time: 0.0111 memory: 11042 loss: 0.2379 2024/07/24 19:49:36 - mmengine - INFO - Iter(train) [ 7390/19224] lr: 1.4105e-05 eta: 8:37:58 time: 2.2970 data_time: 0.0102 memory: 10681 loss: 0.2345 2024/07/24 19:49:56 - mmengine - INFO - Iter(train) [ 7400/19224] lr: 1.4090e-05 eta: 8:37:21 time: 1.9900 data_time: 0.0093 memory: 10447 loss: 0.1904 2024/07/24 19:50:20 - mmengine - INFO - Iter(train) [ 7410/19224] lr: 1.4074e-05 eta: 8:36:52 time: 2.4301 data_time: 0.0092 memory: 13497 loss: 0.2133 2024/07/24 19:50:54 - mmengine - INFO - Iter(train) [ 7420/19224] lr: 1.4059e-05 eta: 8:36:38 time: 3.3765 data_time: 0.0103 memory: 12714 loss: 0.1923 2024/07/24 19:51:25 - mmengine - INFO - Iter(train) [ 7430/19224] lr: 1.4044e-05 eta: 8:36:18 time: 3.0763 data_time: 0.0111 memory: 12460 loss: 0.1779 2024/07/24 19:51:56 - mmengine - INFO - Iter(train) [ 7440/19224] lr: 1.4028e-05 eta: 8:36:00 time: 3.1018 data_time: 0.0099 memory: 11607 loss: 0.1936 2024/07/24 19:52:25 - mmengine - INFO - Iter(train) [ 7450/19224] lr: 1.4013e-05 eta: 8:35:39 time: 2.9756 data_time: 0.0101 memory: 11325 loss: 0.2102 2024/07/24 19:52:52 - mmengine - INFO - Iter(train) [ 7460/19224] lr: 1.3997e-05 eta: 8:35:14 time: 2.6851 data_time: 0.0108 memory: 11163 loss: 0.2385 2024/07/24 19:53:17 - mmengine - INFO - Iter(train) [ 7470/19224] lr: 1.3982e-05 eta: 8:34:46 time: 2.5179 data_time: 0.0107 memory: 11027 loss: 0.2075 2024/07/24 19:53:42 - mmengine - INFO - Iter(train) [ 7480/19224] lr: 1.3966e-05 eta: 8:34:17 time: 2.4533 data_time: 0.0100 memory: 10602 loss: 0.2066 2024/07/24 19:54:03 - mmengine - INFO - Iter(train) [ 7490/19224] lr: 1.3951e-05 eta: 8:33:42 time: 2.1069 data_time: 0.0108 memory: 10252 loss: 0.2028 2024/07/24 19:54:19 - mmengine - INFO - Iter(train) [ 7500/19224] lr: 1.3936e-05 eta: 8:32:59 time: 1.5718 data_time: 0.0092 memory: 9955 loss: 0.2421 2024/07/24 19:54:43 - mmengine - INFO - Iter(train) [ 7510/19224] lr: 1.3920e-05 eta: 8:32:30 time: 2.4572 data_time: 0.0089 memory: 17245 loss: 0.2157 2024/07/24 19:55:18 - mmengine - INFO - Iter(train) [ 7520/19224] lr: 1.3905e-05 eta: 8:32:17 time: 3.4400 data_time: 0.0101 memory: 13328 loss: 0.1871 2024/07/24 19:55:50 - mmengine - INFO - Iter(train) [ 7530/19224] lr: 1.3889e-05 eta: 8:32:00 time: 3.2003 data_time: 0.0110 memory: 12113 loss: 0.1713 2024/07/24 19:56:21 - mmengine - INFO - Iter(train) [ 7540/19224] lr: 1.3873e-05 eta: 8:31:41 time: 3.1463 data_time: 0.0101 memory: 11873 loss: 0.1826 2024/07/24 19:56:52 - mmengine - INFO - Iter(train) [ 7550/19224] lr: 1.3858e-05 eta: 8:31:22 time: 3.0932 data_time: 0.0103 memory: 11649 loss: 0.1830 2024/07/24 19:57:21 - mmengine - INFO - Iter(train) [ 7560/19224] lr: 1.3842e-05 eta: 8:31:00 time: 2.8817 data_time: 0.0113 memory: 11375 loss: 0.2012 2024/07/24 19:57:47 - mmengine - INFO - Iter(train) [ 7570/19224] lr: 1.3827e-05 eta: 8:30:34 time: 2.6254 data_time: 0.0112 memory: 11185 loss: 0.1889 2024/07/24 19:58:11 - mmengine - INFO - Iter(train) [ 7580/19224] lr: 1.3811e-05 eta: 8:30:04 time: 2.4340 data_time: 0.0101 memory: 10932 loss: 0.2745 2024/07/24 19:58:32 - mmengine - INFO - Iter(train) [ 7590/19224] lr: 1.3796e-05 eta: 8:29:29 time: 2.0709 data_time: 0.0105 memory: 10690 loss: 0.2052 2024/07/24 19:58:50 - mmengine - INFO - Iter(train) [ 7600/19224] lr: 1.3780e-05 eta: 8:28:51 time: 1.8321 data_time: 0.0093 memory: 10067 loss: 0.2567 2024/07/24 19:59:14 - mmengine - INFO - Iter(train) [ 7610/19224] lr: 1.3764e-05 eta: 8:28:21 time: 2.3857 data_time: 0.0095 memory: 15692 loss: 0.2128 2024/07/24 19:59:44 - mmengine - INFO - Iter(train) [ 7620/19224] lr: 1.3749e-05 eta: 8:27:59 time: 2.9289 data_time: 0.0104 memory: 12050 loss: 0.1748 2024/07/24 20:00:13 - mmengine - INFO - Iter(train) [ 7630/19224] lr: 1.3733e-05 eta: 8:27:38 time: 2.9780 data_time: 0.0105 memory: 11878 loss: 0.1796 2024/07/24 20:00:40 - mmengine - INFO - Iter(train) [ 7640/19224] lr: 1.3718e-05 eta: 8:27:13 time: 2.6602 data_time: 0.0114 memory: 11683 loss: 0.1795 2024/07/24 20:01:07 - mmengine - INFO - Iter(train) [ 7650/19224] lr: 1.3702e-05 eta: 8:26:47 time: 2.6628 data_time: 0.0117 memory: 11459 loss: 0.1874 2024/07/24 20:01:30 - mmengine - INFO - Iter(train) [ 7660/19224] lr: 1.3686e-05 eta: 8:26:16 time: 2.2940 data_time: 0.0100 memory: 11177 loss: 0.1999 2024/07/24 20:01:51 - mmengine - INFO - Iter(train) [ 7670/19224] lr: 1.3671e-05 eta: 8:25:43 time: 2.1849 data_time: 0.0104 memory: 10959 loss: 0.1872 2024/07/24 20:02:11 - mmengine - INFO - Iter(train) [ 7680/19224] lr: 1.3655e-05 eta: 8:25:07 time: 1.9716 data_time: 0.0093 memory: 10502 loss: 0.1900 2024/07/24 20:02:28 - mmengine - INFO - Iter(train) [ 7690/19224] lr: 1.3639e-05 eta: 8:24:27 time: 1.7016 data_time: 0.0096 memory: 10334 loss: 0.2515 2024/07/24 20:02:43 - mmengine - INFO - Iter(train) [ 7700/19224] lr: 1.3624e-05 eta: 8:23:44 time: 1.5108 data_time: 0.0091 memory: 9873 loss: 0.2290 2024/07/24 20:03:07 - mmengine - INFO - Iter(train) [ 7710/19224] lr: 1.3608e-05 eta: 8:23:14 time: 2.4085 data_time: 0.0091 memory: 16768 loss: 0.2434 2024/07/24 20:03:40 - mmengine - INFO - Iter(train) [ 7720/19224] lr: 1.3592e-05 eta: 8:22:58 time: 3.2765 data_time: 0.0103 memory: 13270 loss: 0.2096 2024/07/24 20:04:08 - mmengine - INFO - Iter(train) [ 7730/19224] lr: 1.3576e-05 eta: 8:22:34 time: 2.7609 data_time: 0.0102 memory: 12210 loss: 0.1582 2024/07/24 20:04:34 - mmengine - INFO - Iter(train) [ 7740/19224] lr: 1.3561e-05 eta: 8:22:07 time: 2.5889 data_time: 0.0110 memory: 11820 loss: 0.1752 2024/07/24 20:04:58 - mmengine - INFO - Iter(train) [ 7750/19224] lr: 1.3545e-05 eta: 8:21:39 time: 2.4881 data_time: 0.0102 memory: 11832 loss: 0.1661 2024/07/24 20:05:24 - mmengine - INFO - Iter(train) [ 7760/19224] lr: 1.3529e-05 eta: 8:21:11 time: 2.5642 data_time: 0.0105 memory: 11336 loss: 0.2081 2024/07/24 20:05:50 - mmengine - INFO - Iter(train) [ 7770/19224] lr: 1.3513e-05 eta: 8:20:45 time: 2.5759 data_time: 0.0114 memory: 11203 loss: 0.2125 2024/07/24 20:06:14 - mmengine - INFO - Iter(train) [ 7780/19224] lr: 1.3498e-05 eta: 8:20:15 time: 2.4323 data_time: 0.0107 memory: 11102 loss: 0.2087 2024/07/24 20:06:33 - mmengine - INFO - Iter(train) [ 7790/19224] lr: 1.3482e-05 eta: 8:19:38 time: 1.8810 data_time: 0.0093 memory: 10727 loss: 0.2510 2024/07/24 20:06:48 - mmengine - INFO - Iter(train) [ 7800/19224] lr: 1.3466e-05 eta: 8:18:56 time: 1.5055 data_time: 0.0088 memory: 10077 loss: 0.1959 2024/07/24 20:07:11 - mmengine - INFO - Iter(train) [ 7810/19224] lr: 1.3450e-05 eta: 8:18:24 time: 2.2661 data_time: 0.0092 memory: 15543 loss: 0.2839 2024/07/24 20:07:40 - mmengine - INFO - Iter(train) [ 7820/19224] lr: 1.3434e-05 eta: 8:18:03 time: 2.9277 data_time: 0.0102 memory: 12431 loss: 0.1907 2024/07/24 20:08:07 - mmengine - INFO - Iter(train) [ 7830/19224] lr: 1.3419e-05 eta: 8:17:38 time: 2.7346 data_time: 0.0103 memory: 11956 loss: 0.1912 2024/07/24 20:08:35 - mmengine - INFO - Iter(train) [ 7840/19224] lr: 1.3403e-05 eta: 8:17:14 time: 2.7906 data_time: 0.0095 memory: 11654 loss: 0.2083 2024/07/24 20:09:01 - mmengine - INFO - Iter(train) [ 7850/19224] lr: 1.3387e-05 eta: 8:16:48 time: 2.5792 data_time: 0.0101 memory: 11426 loss: 0.2180 2024/07/24 20:09:29 - mmengine - INFO - Iter(train) [ 7860/19224] lr: 1.3371e-05 eta: 8:16:24 time: 2.8117 data_time: 0.0101 memory: 11301 loss: 0.2202 2024/07/24 20:09:57 - mmengine - INFO - Iter(train) [ 7870/19224] lr: 1.3355e-05 eta: 8:16:00 time: 2.7816 data_time: 0.0105 memory: 11176 loss: 0.2100 2024/07/24 20:10:26 - mmengine - INFO - Iter(train) [ 7880/19224] lr: 1.3339e-05 eta: 8:15:39 time: 2.9463 data_time: 0.0113 memory: 11037 loss: 0.2618 2024/07/24 20:10:49 - mmengine - INFO - Iter(train) [ 7890/19224] lr: 1.3323e-05 eta: 8:15:08 time: 2.2759 data_time: 0.0107 memory: 10852 loss: 0.2353 2024/07/24 20:11:08 - mmengine - INFO - Iter(train) [ 7900/19224] lr: 1.3308e-05 eta: 8:14:30 time: 1.8548 data_time: 0.0091 memory: 10358 loss: 0.1924 2024/07/24 20:11:39 - mmengine - INFO - Iter(train) [ 7910/19224] lr: 1.3292e-05 eta: 8:14:11 time: 3.0971 data_time: 0.0096 memory: 19084 loss: 0.1972 2024/07/24 20:12:11 - mmengine - INFO - Iter(train) [ 7920/19224] lr: 1.3276e-05 eta: 8:13:53 time: 3.2105 data_time: 0.0117 memory: 12733 loss: 0.1760 2024/07/24 20:12:42 - mmengine - INFO - Iter(train) [ 7930/19224] lr: 1.3260e-05 eta: 8:13:34 time: 3.1399 data_time: 0.0116 memory: 12085 loss: 0.1798 2024/07/24 20:13:08 - mmengine - INFO - Iter(train) [ 7940/19224] lr: 1.3244e-05 eta: 8:13:08 time: 2.6072 data_time: 0.0106 memory: 11604 loss: 0.2115 2024/07/24 20:13:33 - mmengine - INFO - Iter(train) [ 7950/19224] lr: 1.3228e-05 eta: 8:12:40 time: 2.4744 data_time: 0.0104 memory: 11361 loss: 0.2009 2024/07/24 20:13:57 - mmengine - INFO - Iter(train) [ 7960/19224] lr: 1.3212e-05 eta: 8:12:10 time: 2.3550 data_time: 0.0101 memory: 11304 loss: 0.1932 2024/07/24 20:14:21 - mmengine - INFO - Iter(train) [ 7970/19224] lr: 1.3196e-05 eta: 8:11:41 time: 2.4486 data_time: 0.0110 memory: 11177 loss: 0.2007 2024/07/24 20:14:41 - mmengine - INFO - Iter(train) [ 7980/19224] lr: 1.3180e-05 eta: 8:11:06 time: 2.0309 data_time: 0.0108 memory: 10915 loss: 0.3259 2024/07/24 20:14:59 - mmengine - INFO - Iter(train) [ 7990/19224] lr: 1.3164e-05 eta: 8:10:29 time: 1.7918 data_time: 0.0095 memory: 10421 loss: 0.2124 2024/07/24 20:15:16 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 20:15:16 - mmengine - INFO - Iter(train) [ 8000/19224] lr: 1.3148e-05 eta: 8:09:48 time: 1.6216 data_time: 0.0091 memory: 10137 loss: 0.2009 2024/07/24 20:15:16 - mmengine - INFO - Saving checkpoint at 8000 iterations 2024/07/24 20:15:40 - mmengine - INFO - Iter(train) [ 8010/19224] lr: 1.3132e-05 eta: 8:09:19 time: 2.4034 data_time: 0.1838 memory: 16632 loss: 0.1846 2024/07/24 20:16:09 - mmengine - INFO - Iter(train) [ 8020/19224] lr: 1.3116e-05 eta: 8:08:58 time: 2.9581 data_time: 0.0100 memory: 12465 loss: 0.1860 2024/07/24 20:16:38 - mmengine - INFO - Iter(train) [ 8030/19224] lr: 1.3100e-05 eta: 8:08:35 time: 2.8931 data_time: 0.0108 memory: 11952 loss: 0.1818 2024/07/24 20:17:05 - mmengine - INFO - Iter(train) [ 8040/19224] lr: 1.3084e-05 eta: 8:08:10 time: 2.6842 data_time: 0.0112 memory: 11875 loss: 0.1774 2024/07/24 20:17:33 - mmengine - INFO - Iter(train) [ 8050/19224] lr: 1.3068e-05 eta: 8:07:46 time: 2.8027 data_time: 0.0109 memory: 11508 loss: 0.2007 2024/07/24 20:17:58 - mmengine - INFO - Iter(train) [ 8060/19224] lr: 1.3052e-05 eta: 8:07:19 time: 2.4927 data_time: 0.0109 memory: 11308 loss: 0.2237 2024/07/24 20:18:23 - mmengine - INFO - Iter(train) [ 8070/19224] lr: 1.3036e-05 eta: 8:06:51 time: 2.5022 data_time: 0.0104 memory: 11241 loss: 0.1697 2024/07/24 20:18:48 - mmengine - INFO - Iter(train) [ 8080/19224] lr: 1.3020e-05 eta: 8:06:23 time: 2.4734 data_time: 0.0101 memory: 11072 loss: 0.2672 2024/07/24 20:19:10 - mmengine - INFO - Iter(train) [ 8090/19224] lr: 1.3004e-05 eta: 8:05:51 time: 2.2220 data_time: 0.0097 memory: 10798 loss: 0.2651 2024/07/24 20:19:29 - mmengine - INFO - Iter(train) [ 8100/19224] lr: 1.2988e-05 eta: 8:05:15 time: 1.8771 data_time: 0.0095 memory: 10337 loss: 0.2049 2024/07/24 20:19:54 - mmengine - INFO - Iter(train) [ 8110/19224] lr: 1.2972e-05 eta: 8:04:47 time: 2.5318 data_time: 0.0091 memory: 16719 loss: 0.1921 2024/07/24 20:20:26 - mmengine - INFO - Iter(train) [ 8120/19224] lr: 1.2956e-05 eta: 8:04:29 time: 3.2201 data_time: 0.0107 memory: 13403 loss: 0.1667 2024/07/24 20:20:57 - mmengine - INFO - Iter(train) [ 8130/19224] lr: 1.2940e-05 eta: 8:04:09 time: 3.0430 data_time: 0.0106 memory: 12022 loss: 0.2035 2024/07/24 20:21:25 - mmengine - INFO - Iter(train) [ 8140/19224] lr: 1.2923e-05 eta: 8:03:46 time: 2.8560 data_time: 0.0105 memory: 11692 loss: 0.1750 2024/07/24 20:21:52 - mmengine - INFO - Iter(train) [ 8150/19224] lr: 1.2907e-05 eta: 8:03:20 time: 2.6446 data_time: 0.0102 memory: 11382 loss: 0.1796 2024/07/24 20:22:18 - mmengine - INFO - Iter(train) [ 8160/19224] lr: 1.2891e-05 eta: 8:02:54 time: 2.6411 data_time: 0.0106 memory: 11176 loss: 0.2290 2024/07/24 20:22:44 - mmengine - INFO - Iter(train) [ 8170/19224] lr: 1.2875e-05 eta: 8:02:28 time: 2.5994 data_time: 0.0101 memory: 11013 loss: 0.2462 2024/07/24 20:23:06 - mmengine - INFO - Iter(train) [ 8180/19224] lr: 1.2859e-05 eta: 8:01:56 time: 2.2354 data_time: 0.0096 memory: 10722 loss: 0.3751 2024/07/24 20:23:26 - mmengine - INFO - Iter(train) [ 8190/19224] lr: 1.2843e-05 eta: 8:01:22 time: 1.9890 data_time: 0.0096 memory: 10403 loss: 0.1824 2024/07/24 20:23:42 - mmengine - INFO - Iter(train) [ 8200/19224] lr: 1.2827e-05 eta: 8:00:42 time: 1.5776 data_time: 0.0101 memory: 9942 loss: 0.2240 2024/07/24 20:24:05 - mmengine - INFO - Iter(train) [ 8210/19224] lr: 1.2810e-05 eta: 8:00:12 time: 2.3299 data_time: 0.0087 memory: 16358 loss: 0.3004 2024/07/24 20:24:36 - mmengine - INFO - Iter(train) [ 8220/19224] lr: 1.2794e-05 eta: 7:59:51 time: 3.0430 data_time: 0.0103 memory: 12562 loss: 0.1734 2024/07/24 20:25:04 - mmengine - INFO - Iter(train) [ 8230/19224] lr: 1.2778e-05 eta: 7:59:28 time: 2.8715 data_time: 0.0103 memory: 11954 loss: 0.1660 2024/07/24 20:25:33 - mmengine - INFO - Iter(train) [ 8240/19224] lr: 1.2762e-05 eta: 7:59:05 time: 2.8201 data_time: 0.0102 memory: 11502 loss: 0.2057 2024/07/24 20:26:00 - mmengine - INFO - Iter(train) [ 8250/19224] lr: 1.2746e-05 eta: 7:58:40 time: 2.7183 data_time: 0.0104 memory: 11295 loss: 0.1875 2024/07/24 20:26:26 - mmengine - INFO - Iter(train) [ 8260/19224] lr: 1.2729e-05 eta: 7:58:13 time: 2.5832 data_time: 0.0105 memory: 11336 loss: 0.1935 2024/07/24 20:26:51 - mmengine - INFO - Iter(train) [ 8270/19224] lr: 1.2713e-05 eta: 7:57:46 time: 2.5078 data_time: 0.0113 memory: 11154 loss: 0.1886 2024/07/24 20:27:14 - mmengine - INFO - Iter(train) [ 8280/19224] lr: 1.2697e-05 eta: 7:57:16 time: 2.3079 data_time: 0.0118 memory: 10906 loss: 0.2360 2024/07/24 20:27:36 - mmengine - INFO - Iter(train) [ 8290/19224] lr: 1.2681e-05 eta: 7:56:44 time: 2.1713 data_time: 0.0105 memory: 10541 loss: 0.2102 2024/07/24 20:27:56 - mmengine - INFO - Iter(train) [ 8300/19224] lr: 1.2665e-05 eta: 7:56:10 time: 2.0725 data_time: 0.0095 memory: 10259 loss: 0.1860 2024/07/24 20:28:22 - mmengine - INFO - Iter(train) [ 8310/19224] lr: 1.2648e-05 eta: 7:55:44 time: 2.6196 data_time: 0.0092 memory: 18474 loss: 0.2008 2024/07/24 20:28:54 - mmengine - INFO - Iter(train) [ 8320/19224] lr: 1.2632e-05 eta: 7:55:25 time: 3.1659 data_time: 0.0105 memory: 12467 loss: 0.1989 2024/07/24 20:29:25 - mmengine - INFO - Iter(train) [ 8330/19224] lr: 1.2616e-05 eta: 7:55:05 time: 3.0724 data_time: 0.0103 memory: 12132 loss: 0.1890 2024/07/24 20:29:53 - mmengine - INFO - Iter(train) [ 8340/19224] lr: 1.2600e-05 eta: 7:54:41 time: 2.8103 data_time: 0.0099 memory: 11889 loss: 0.1805 2024/07/24 20:30:25 - mmengine - INFO - Iter(train) [ 8350/19224] lr: 1.2583e-05 eta: 7:54:23 time: 3.1890 data_time: 0.0107 memory: 11550 loss: 0.1914 2024/07/24 20:30:55 - mmengine - INFO - Iter(train) [ 8360/19224] lr: 1.2567e-05 eta: 7:54:01 time: 2.9956 data_time: 0.0104 memory: 11426 loss: 0.2139 2024/07/24 20:31:21 - mmengine - INFO - Iter(train) [ 8370/19224] lr: 1.2551e-05 eta: 7:53:35 time: 2.6010 data_time: 0.0110 memory: 11297 loss: 0.2300 2024/07/24 20:31:48 - mmengine - INFO - Iter(train) [ 8380/19224] lr: 1.2534e-05 eta: 7:53:10 time: 2.7506 data_time: 0.0102 memory: 11078 loss: 0.2308 2024/07/24 20:32:15 - mmengine - INFO - Iter(train) [ 8390/19224] lr: 1.2518e-05 eta: 7:52:44 time: 2.6310 data_time: 0.0101 memory: 10827 loss: 0.2269 2024/07/24 20:32:36 - mmengine - INFO - Iter(train) [ 8400/19224] lr: 1.2502e-05 eta: 7:52:13 time: 2.1799 data_time: 0.0104 memory: 10449 loss: 0.2221 2024/07/24 20:33:05 - mmengine - INFO - Iter(train) [ 8410/19224] lr: 1.2486e-05 eta: 7:51:49 time: 2.8245 data_time: 0.0098 memory: 13765 loss: 0.2250 2024/07/24 20:33:38 - mmengine - INFO - Iter(train) [ 8420/19224] lr: 1.2469e-05 eta: 7:51:32 time: 3.3055 data_time: 0.0102 memory: 12698 loss: 0.1726 2024/07/24 20:34:14 - mmengine - INFO - Iter(train) [ 8430/19224] lr: 1.2453e-05 eta: 7:51:18 time: 3.6088 data_time: 0.0106 memory: 11825 loss: 0.1810 2024/07/24 20:34:49 - mmengine - INFO - Iter(train) [ 8440/19224] lr: 1.2437e-05 eta: 7:51:04 time: 3.5294 data_time: 0.0100 memory: 11591 loss: 0.1669 2024/07/24 20:35:17 - mmengine - INFO - Iter(train) [ 8450/19224] lr: 1.2420e-05 eta: 7:50:40 time: 2.7934 data_time: 0.0108 memory: 11386 loss: 0.2001 2024/07/24 20:35:44 - mmengine - INFO - Iter(train) [ 8460/19224] lr: 1.2404e-05 eta: 7:50:14 time: 2.6713 data_time: 0.0104 memory: 11285 loss: 0.2004 2024/07/24 20:36:10 - mmengine - INFO - Iter(train) [ 8470/19224] lr: 1.2387e-05 eta: 7:49:48 time: 2.6305 data_time: 0.0104 memory: 11212 loss: 0.2235 2024/07/24 20:36:36 - mmengine - INFO - Iter(train) [ 8480/19224] lr: 1.2371e-05 eta: 7:49:21 time: 2.5877 data_time: 0.0103 memory: 11091 loss: 0.1939 2024/07/24 20:37:00 - mmengine - INFO - Iter(train) [ 8490/19224] lr: 1.2355e-05 eta: 7:48:53 time: 2.4115 data_time: 0.0104 memory: 10908 loss: 0.2144 2024/07/24 20:37:19 - mmengine - INFO - Iter(train) [ 8500/19224] lr: 1.2338e-05 eta: 7:48:17 time: 1.8965 data_time: 0.0090 memory: 10274 loss: 0.2206 2024/07/24 20:37:43 - mmengine - INFO - Iter(train) [ 8510/19224] lr: 1.2322e-05 eta: 7:47:48 time: 2.3665 data_time: 0.0095 memory: 13560 loss: 0.2039 2024/07/24 20:38:17 - mmengine - INFO - Iter(train) [ 8520/19224] lr: 1.2306e-05 eta: 7:47:31 time: 3.4034 data_time: 0.0101 memory: 12636 loss: 0.1637 2024/07/24 20:38:48 - mmengine - INFO - Iter(train) [ 8530/19224] lr: 1.2289e-05 eta: 7:47:12 time: 3.1664 data_time: 0.0103 memory: 12127 loss: 0.1867 2024/07/24 20:39:17 - mmengine - INFO - Iter(train) [ 8540/19224] lr: 1.2273e-05 eta: 7:46:49 time: 2.8467 data_time: 0.0101 memory: 11627 loss: 0.1791 2024/07/24 20:39:46 - mmengine - INFO - Iter(train) [ 8550/19224] lr: 1.2256e-05 eta: 7:46:26 time: 2.8921 data_time: 0.0103 memory: 11353 loss: 0.1913 2024/07/24 20:40:13 - mmengine - INFO - Iter(train) [ 8560/19224] lr: 1.2240e-05 eta: 7:46:00 time: 2.6809 data_time: 0.0108 memory: 11173 loss: 0.2106 2024/07/24 20:40:37 - mmengine - INFO - Iter(train) [ 8570/19224] lr: 1.2224e-05 eta: 7:45:31 time: 2.4043 data_time: 0.0114 memory: 11005 loss: 0.2203 2024/07/24 20:40:58 - mmengine - INFO - Iter(train) [ 8580/19224] lr: 1.2207e-05 eta: 7:44:59 time: 2.1185 data_time: 0.0099 memory: 10729 loss: 0.2535 2024/07/24 20:41:17 - mmengine - INFO - Iter(train) [ 8590/19224] lr: 1.2191e-05 eta: 7:44:24 time: 1.9392 data_time: 0.0092 memory: 10519 loss: 0.2878 2024/07/24 20:41:34 - mmengine - INFO - Iter(train) [ 8600/19224] lr: 1.2174e-05 eta: 7:43:46 time: 1.6328 data_time: 0.0091 memory: 10049 loss: 0.2539 2024/07/24 20:41:57 - mmengine - INFO - Iter(train) [ 8610/19224] lr: 1.2158e-05 eta: 7:43:16 time: 2.3407 data_time: 0.0090 memory: 15923 loss: 0.1908 2024/07/24 20:42:26 - mmengine - INFO - Iter(train) [ 8620/19224] lr: 1.2141e-05 eta: 7:42:54 time: 2.9206 data_time: 0.0106 memory: 12621 loss: 0.1658 2024/07/24 20:42:54 - mmengine - INFO - Iter(train) [ 8630/19224] lr: 1.2125e-05 eta: 7:42:29 time: 2.7603 data_time: 0.0103 memory: 11959 loss: 0.1692 2024/07/24 20:43:22 - mmengine - INFO - Iter(train) [ 8640/19224] lr: 1.2108e-05 eta: 7:42:05 time: 2.8114 data_time: 0.0109 memory: 11550 loss: 0.2000 2024/07/24 20:43:47 - mmengine - INFO - Iter(train) [ 8650/19224] lr: 1.2092e-05 eta: 7:41:38 time: 2.5168 data_time: 0.0108 memory: 11513 loss: 0.1944 2024/07/24 20:44:11 - mmengine - INFO - Iter(train) [ 8660/19224] lr: 1.2075e-05 eta: 7:41:09 time: 2.4033 data_time: 0.0104 memory: 11249 loss: 0.2205 2024/07/24 20:44:35 - mmengine - INFO - Iter(train) [ 8670/19224] lr: 1.2059e-05 eta: 7:40:40 time: 2.3490 data_time: 0.0101 memory: 11168 loss: 0.1649 2024/07/24 20:44:57 - mmengine - INFO - Iter(train) [ 8680/19224] lr: 1.2043e-05 eta: 7:40:08 time: 2.1982 data_time: 0.0101 memory: 10890 loss: 0.2225 2024/07/24 20:45:14 - mmengine - INFO - Iter(train) [ 8690/19224] lr: 1.2026e-05 eta: 7:39:32 time: 1.7776 data_time: 0.0093 memory: 10255 loss: 0.2292 2024/07/24 20:45:30 - mmengine - INFO - Iter(train) [ 8700/19224] lr: 1.2010e-05 eta: 7:38:53 time: 1.5919 data_time: 0.0089 memory: 9936 loss: 0.2038 2024/07/24 20:45:52 - mmengine - INFO - Iter(train) [ 8710/19224] lr: 1.1993e-05 eta: 7:38:22 time: 2.1887 data_time: 0.0089 memory: 15840 loss: 0.2240 2024/07/24 20:46:20 - mmengine - INFO - Iter(train) [ 8720/19224] lr: 1.1977e-05 eta: 7:37:59 time: 2.8355 data_time: 0.0104 memory: 12243 loss: 0.1606 2024/07/24 20:46:49 - mmengine - INFO - Iter(train) [ 8730/19224] lr: 1.1960e-05 eta: 7:37:35 time: 2.8055 data_time: 0.0115 memory: 11749 loss: 0.1859 2024/07/24 20:47:15 - mmengine - INFO - Iter(train) [ 8740/19224] lr: 1.1943e-05 eta: 7:37:09 time: 2.6819 data_time: 0.0111 memory: 11593 loss: 0.1945 2024/07/24 20:47:43 - mmengine - INFO - Iter(train) [ 8750/19224] lr: 1.1927e-05 eta: 7:36:45 time: 2.7403 data_time: 0.0104 memory: 11299 loss: 0.1941 2024/07/24 20:48:08 - mmengine - INFO - Iter(train) [ 8760/19224] lr: 1.1910e-05 eta: 7:36:18 time: 2.5730 data_time: 0.0100 memory: 11147 loss: 0.2023 2024/07/24 20:48:33 - mmengine - INFO - Iter(train) [ 8770/19224] lr: 1.1894e-05 eta: 7:35:50 time: 2.4879 data_time: 0.0104 memory: 11128 loss: 0.2111 2024/07/24 20:48:56 - mmengine - INFO - Iter(train) [ 8780/19224] lr: 1.1877e-05 eta: 7:35:20 time: 2.2304 data_time: 0.0097 memory: 10677 loss: 0.2057 2024/07/24 20:49:15 - mmengine - INFO - Iter(train) [ 8790/19224] lr: 1.1861e-05 eta: 7:34:46 time: 1.9595 data_time: 0.0104 memory: 10291 loss: 0.2074 2024/07/24 20:49:33 - mmengine - INFO - Iter(train) [ 8800/19224] lr: 1.1844e-05 eta: 7:34:10 time: 1.8179 data_time: 0.0091 memory: 10037 loss: 0.2257 2024/07/24 20:49:55 - mmengine - INFO - Iter(train) [ 8810/19224] lr: 1.1828e-05 eta: 7:33:39 time: 2.1846 data_time: 0.0086 memory: 14435 loss: 0.2033 2024/07/24 20:50:28 - mmengine - INFO - Iter(train) [ 8820/19224] lr: 1.1811e-05 eta: 7:33:21 time: 3.3130 data_time: 0.0104 memory: 12356 loss: 0.1627 2024/07/24 20:50:58 - mmengine - INFO - Iter(train) [ 8830/19224] lr: 1.1795e-05 eta: 7:32:59 time: 2.9782 data_time: 0.0107 memory: 11905 loss: 0.1916 2024/07/24 20:51:28 - mmengine - INFO - Iter(train) [ 8840/19224] lr: 1.1778e-05 eta: 7:32:37 time: 2.9471 data_time: 0.0106 memory: 11796 loss: 0.1631 2024/07/24 20:51:56 - mmengine - INFO - Iter(train) [ 8850/19224] lr: 1.1761e-05 eta: 7:32:13 time: 2.8198 data_time: 0.0113 memory: 11634 loss: 0.1944 2024/07/24 20:52:22 - mmengine - INFO - Iter(train) [ 8860/19224] lr: 1.1745e-05 eta: 7:31:47 time: 2.6654 data_time: 0.0103 memory: 11357 loss: 0.2213 2024/07/24 20:52:48 - mmengine - INFO - Iter(train) [ 8870/19224] lr: 1.1728e-05 eta: 7:31:20 time: 2.5019 data_time: 0.0106 memory: 11195 loss: 0.2071 2024/07/24 20:53:12 - mmengine - INFO - Iter(train) [ 8880/19224] lr: 1.1712e-05 eta: 7:30:52 time: 2.4413 data_time: 0.0110 memory: 11024 loss: 0.1940 2024/07/24 20:53:35 - mmengine - INFO - Iter(train) [ 8890/19224] lr: 1.1695e-05 eta: 7:30:22 time: 2.2732 data_time: 0.0098 memory: 10848 loss: 0.2615 2024/07/24 20:53:53 - mmengine - INFO - Iter(train) [ 8900/19224] lr: 1.1678e-05 eta: 7:29:47 time: 1.8506 data_time: 0.0091 memory: 10401 loss: 0.2614 2024/07/24 20:54:15 - mmengine - INFO - Iter(train) [ 8910/19224] lr: 1.1662e-05 eta: 7:29:16 time: 2.1824 data_time: 0.0091 memory: 13663 loss: 0.1803 2024/07/24 20:54:46 - mmengine - INFO - Iter(train) [ 8920/19224] lr: 1.1645e-05 eta: 7:28:55 time: 3.0986 data_time: 0.0105 memory: 12395 loss: 0.1876 2024/07/24 20:55:15 - mmengine - INFO - Iter(train) [ 8930/19224] lr: 1.1629e-05 eta: 7:28:32 time: 2.9102 data_time: 0.0107 memory: 11786 loss: 0.1754 2024/07/24 20:55:44 - mmengine - INFO - Iter(train) [ 8940/19224] lr: 1.1612e-05 eta: 7:28:09 time: 2.8522 data_time: 0.0103 memory: 11533 loss: 0.1891 2024/07/24 20:56:11 - mmengine - INFO - Iter(train) [ 8950/19224] lr: 1.1595e-05 eta: 7:27:44 time: 2.7462 data_time: 0.0102 memory: 11323 loss: 0.1957 2024/07/24 20:56:38 - mmengine - INFO - Iter(train) [ 8960/19224] lr: 1.1579e-05 eta: 7:27:18 time: 2.6449 data_time: 0.0107 memory: 11271 loss: 0.1853 2024/07/24 20:57:02 - mmengine - INFO - Iter(train) [ 8970/19224] lr: 1.1562e-05 eta: 7:26:51 time: 2.4768 data_time: 0.0114 memory: 11096 loss: 0.2011 2024/07/24 20:57:26 - mmengine - INFO - Iter(train) [ 8980/19224] lr: 1.1545e-05 eta: 7:26:22 time: 2.4020 data_time: 0.0108 memory: 10966 loss: 0.2222 2024/07/24 20:57:47 - mmengine - INFO - Iter(train) [ 8990/19224] lr: 1.1529e-05 eta: 7:25:49 time: 2.0390 data_time: 0.0094 memory: 10610 loss: 0.1979 2024/07/24 20:58:04 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 20:58:04 - mmengine - INFO - Iter(train) [ 9000/19224] lr: 1.1512e-05 eta: 7:25:14 time: 1.7559 data_time: 0.0105 memory: 10143 loss: 0.2254 2024/07/24 20:58:04 - mmengine - INFO - Saving checkpoint at 9000 iterations 2024/07/24 20:58:28 - mmengine - INFO - Iter(train) [ 9010/19224] lr: 1.1495e-05 eta: 7:24:45 time: 2.3901 data_time: 0.1918 memory: 13504 loss: 0.1964 2024/07/24 20:59:00 - mmengine - INFO - Iter(train) [ 9020/19224] lr: 1.1479e-05 eta: 7:24:26 time: 3.2344 data_time: 0.0104 memory: 12424 loss: 0.1732 2024/07/24 20:59:31 - mmengine - INFO - Iter(train) [ 9030/19224] lr: 1.1462e-05 eta: 7:24:04 time: 3.0349 data_time: 0.0107 memory: 12078 loss: 0.1596 2024/07/24 21:00:01 - mmengine - INFO - Iter(train) [ 9040/19224] lr: 1.1445e-05 eta: 7:23:42 time: 2.9896 data_time: 0.0106 memory: 11836 loss: 0.1667 2024/07/24 21:00:32 - mmengine - INFO - Iter(train) [ 9050/19224] lr: 1.1429e-05 eta: 7:23:22 time: 3.1317 data_time: 0.0110 memory: 11591 loss: 0.1901 2024/07/24 21:01:00 - mmengine - INFO - Iter(train) [ 9060/19224] lr: 1.1412e-05 eta: 7:22:58 time: 2.7644 data_time: 0.0119 memory: 11408 loss: 0.1830 2024/07/24 21:01:25 - mmengine - INFO - Iter(train) [ 9070/19224] lr: 1.1395e-05 eta: 7:22:31 time: 2.5705 data_time: 0.0118 memory: 11127 loss: 0.1573 2024/07/24 21:01:50 - mmengine - INFO - Iter(train) [ 9080/19224] lr: 1.1379e-05 eta: 7:22:03 time: 2.4873 data_time: 0.0107 memory: 11013 loss: 0.2341 2024/07/24 21:02:12 - mmengine - INFO - Iter(train) [ 9090/19224] lr: 1.1362e-05 eta: 7:21:33 time: 2.1973 data_time: 0.0113 memory: 10733 loss: 0.2094 2024/07/24 21:02:30 - mmengine - INFO - Iter(train) [ 9100/19224] lr: 1.1345e-05 eta: 7:20:57 time: 1.7701 data_time: 0.0098 memory: 10113 loss: 0.2153 2024/07/24 21:02:52 - mmengine - INFO - Iter(train) [ 9110/19224] lr: 1.1329e-05 eta: 7:20:26 time: 2.1766 data_time: 0.0089 memory: 14744 loss: 0.2026 2024/07/24 21:03:24 - mmengine - INFO - Iter(train) [ 9120/19224] lr: 1.1312e-05 eta: 7:20:07 time: 3.2130 data_time: 0.0109 memory: 12879 loss: 0.1910 2024/07/24 21:03:53 - mmengine - INFO - Iter(train) [ 9130/19224] lr: 1.1295e-05 eta: 7:19:44 time: 2.9362 data_time: 0.0106 memory: 11935 loss: 0.1811 2024/07/24 21:04:22 - mmengine - INFO - Iter(train) [ 9140/19224] lr: 1.1279e-05 eta: 7:19:21 time: 2.8801 data_time: 0.0110 memory: 11640 loss: 0.1812 2024/07/24 21:04:50 - mmengine - INFO - Iter(train) [ 9150/19224] lr: 1.1262e-05 eta: 7:18:56 time: 2.7583 data_time: 0.0110 memory: 11625 loss: 0.1925 2024/07/24 21:05:17 - mmengine - INFO - Iter(train) [ 9160/19224] lr: 1.1245e-05 eta: 7:18:31 time: 2.7302 data_time: 0.0104 memory: 11310 loss: 0.1862 2024/07/24 21:05:43 - mmengine - INFO - Iter(train) [ 9170/19224] lr: 1.1228e-05 eta: 7:18:05 time: 2.5749 data_time: 0.0111 memory: 11130 loss: 0.1905 2024/07/24 21:06:07 - mmengine - INFO - Iter(train) [ 9180/19224] lr: 1.1212e-05 eta: 7:17:37 time: 2.4097 data_time: 0.0108 memory: 10954 loss: 0.2317 2024/07/24 21:06:28 - mmengine - INFO - Iter(train) [ 9190/19224] lr: 1.1195e-05 eta: 7:17:05 time: 2.1372 data_time: 0.0107 memory: 10632 loss: 0.1812 2024/07/24 21:06:47 - mmengine - INFO - Iter(train) [ 9200/19224] lr: 1.1178e-05 eta: 7:16:31 time: 1.8627 data_time: 0.0093 memory: 10201 loss: 0.1992 2024/07/24 21:07:08 - mmengine - INFO - Iter(train) [ 9210/19224] lr: 1.1161e-05 eta: 7:15:59 time: 2.1269 data_time: 0.0087 memory: 12865 loss: 0.2095 2024/07/24 21:07:39 - mmengine - INFO - Iter(train) [ 9220/19224] lr: 1.1145e-05 eta: 7:15:38 time: 3.0792 data_time: 0.0118 memory: 12199 loss: 0.1848 2024/07/24 21:08:07 - mmengine - INFO - Iter(train) [ 9230/19224] lr: 1.1128e-05 eta: 7:15:15 time: 2.8455 data_time: 0.0109 memory: 11823 loss: 0.2160 2024/07/24 21:08:35 - mmengine - INFO - Iter(train) [ 9240/19224] lr: 1.1111e-05 eta: 7:14:50 time: 2.7466 data_time: 0.0105 memory: 11439 loss: 0.1928 2024/07/24 21:09:01 - mmengine - INFO - Iter(train) [ 9250/19224] lr: 1.1095e-05 eta: 7:14:24 time: 2.6608 data_time: 0.0105 memory: 11293 loss: 0.2018 2024/07/24 21:09:28 - mmengine - INFO - Iter(train) [ 9260/19224] lr: 1.1078e-05 eta: 7:13:59 time: 2.6423 data_time: 0.0108 memory: 11231 loss: 0.2030 2024/07/24 21:09:53 - mmengine - INFO - Iter(train) [ 9270/19224] lr: 1.1061e-05 eta: 7:13:31 time: 2.4997 data_time: 0.0107 memory: 11072 loss: 0.2168 2024/07/24 21:10:17 - mmengine - INFO - Iter(train) [ 9280/19224] lr: 1.1044e-05 eta: 7:13:03 time: 2.4228 data_time: 0.0103 memory: 10994 loss: 0.2182 2024/07/24 21:10:38 - mmengine - INFO - Iter(train) [ 9290/19224] lr: 1.1028e-05 eta: 7:12:31 time: 2.0946 data_time: 0.0104 memory: 10679 loss: 0.2691 2024/07/24 21:10:56 - mmengine - INFO - Iter(train) [ 9300/19224] lr: 1.1011e-05 eta: 7:11:57 time: 1.8313 data_time: 0.0101 memory: 10203 loss: 0.1839 2024/07/24 21:11:19 - mmengine - INFO - Iter(train) [ 9310/19224] lr: 1.0994e-05 eta: 7:11:27 time: 2.2910 data_time: 0.0089 memory: 15389 loss: 0.2124 2024/07/24 21:11:51 - mmengine - INFO - Iter(train) [ 9320/19224] lr: 1.0977e-05 eta: 7:11:07 time: 3.1693 data_time: 0.0105 memory: 12387 loss: 0.1987 2024/07/24 21:12:20 - mmengine - INFO - Iter(train) [ 9330/19224] lr: 1.0960e-05 eta: 7:10:45 time: 2.9456 data_time: 0.0106 memory: 11861 loss: 0.1716 2024/07/24 21:12:49 - mmengine - INFO - Iter(train) [ 9340/19224] lr: 1.0944e-05 eta: 7:10:21 time: 2.8193 data_time: 0.0105 memory: 11880 loss: 0.2049 2024/07/24 21:13:17 - mmengine - INFO - Iter(train) [ 9350/19224] lr: 1.0927e-05 eta: 7:09:57 time: 2.8518 data_time: 0.0106 memory: 11519 loss: 0.1566 2024/07/24 21:13:45 - mmengine - INFO - Iter(train) [ 9360/19224] lr: 1.0910e-05 eta: 7:09:33 time: 2.7522 data_time: 0.0101 memory: 11324 loss: 0.1753 2024/07/24 21:14:11 - mmengine - INFO - Iter(train) [ 9370/19224] lr: 1.0893e-05 eta: 7:09:06 time: 2.6226 data_time: 0.0106 memory: 11408 loss: 0.2229 2024/07/24 21:14:35 - mmengine - INFO - Iter(train) [ 9380/19224] lr: 1.0877e-05 eta: 7:08:39 time: 2.4393 data_time: 0.0109 memory: 11010 loss: 0.2227 2024/07/24 21:14:58 - mmengine - INFO - Iter(train) [ 9390/19224] lr: 1.0860e-05 eta: 7:08:09 time: 2.2469 data_time: 0.0101 memory: 10740 loss: 0.4284 2024/07/24 21:15:17 - mmengine - INFO - Iter(train) [ 9400/19224] lr: 1.0843e-05 eta: 7:07:36 time: 1.9718 data_time: 0.0092 memory: 10261 loss: 0.1862 2024/07/24 21:15:41 - mmengine - INFO - Iter(train) [ 9410/19224] lr: 1.0826e-05 eta: 7:07:07 time: 2.3349 data_time: 0.0096 memory: 13948 loss: 0.1982 2024/07/24 21:16:14 - mmengine - INFO - Iter(train) [ 9420/19224] lr: 1.0809e-05 eta: 7:06:48 time: 3.3233 data_time: 0.0104 memory: 12648 loss: 0.1631 2024/07/24 21:16:45 - mmengine - INFO - Iter(train) [ 9430/19224] lr: 1.0793e-05 eta: 7:06:27 time: 3.1015 data_time: 0.0105 memory: 12083 loss: 0.1860 2024/07/24 21:17:15 - mmengine - INFO - Iter(train) [ 9440/19224] lr: 1.0776e-05 eta: 7:06:05 time: 2.9783 data_time: 0.0103 memory: 11825 loss: 0.2032 2024/07/24 21:17:44 - mmengine - INFO - Iter(train) [ 9450/19224] lr: 1.0759e-05 eta: 7:05:42 time: 2.9022 data_time: 0.0105 memory: 11736 loss: 0.1875 2024/07/24 21:18:11 - mmengine - INFO - Iter(train) [ 9460/19224] lr: 1.0742e-05 eta: 7:05:17 time: 2.7188 data_time: 0.0112 memory: 11334 loss: 0.2016 2024/07/24 21:18:37 - mmengine - INFO - Iter(train) [ 9470/19224] lr: 1.0725e-05 eta: 7:04:50 time: 2.5777 data_time: 0.0113 memory: 11139 loss: 0.2217 2024/07/24 21:19:01 - mmengine - INFO - Iter(train) [ 9480/19224] lr: 1.0709e-05 eta: 7:04:22 time: 2.4301 data_time: 0.0106 memory: 10995 loss: 0.2261 2024/07/24 21:19:22 - mmengine - INFO - Iter(train) [ 9490/19224] lr: 1.0692e-05 eta: 7:03:51 time: 2.1339 data_time: 0.0099 memory: 10771 loss: 0.1820 2024/07/24 21:19:41 - mmengine - INFO - Iter(train) [ 9500/19224] lr: 1.0675e-05 eta: 7:03:17 time: 1.8212 data_time: 0.0102 memory: 10174 loss: 0.1970 2024/07/24 21:20:05 - mmengine - INFO - Iter(train) [ 9510/19224] lr: 1.0658e-05 eta: 7:02:49 time: 2.4095 data_time: 0.0111 memory: 15416 loss: 0.1984 2024/07/24 21:20:37 - mmengine - INFO - Iter(train) [ 9520/19224] lr: 1.0641e-05 eta: 7:02:29 time: 3.2238 data_time: 0.0104 memory: 12708 loss: 0.1847 2024/07/24 21:21:07 - mmengine - INFO - Iter(train) [ 9530/19224] lr: 1.0625e-05 eta: 7:02:06 time: 2.9630 data_time: 0.0103 memory: 11825 loss: 0.1657 2024/07/24 21:21:36 - mmengine - INFO - Iter(train) [ 9540/19224] lr: 1.0608e-05 eta: 7:01:43 time: 2.9223 data_time: 0.0102 memory: 11654 loss: 0.1722 2024/07/24 21:22:03 - mmengine - INFO - Iter(train) [ 9550/19224] lr: 1.0591e-05 eta: 7:01:18 time: 2.7616 data_time: 0.0110 memory: 11417 loss: 0.1644 2024/07/24 21:22:30 - mmengine - INFO - Iter(train) [ 9560/19224] lr: 1.0574e-05 eta: 7:00:53 time: 2.6468 data_time: 0.0100 memory: 11226 loss: 0.1840 2024/07/24 21:22:54 - mmengine - INFO - Iter(train) [ 9570/19224] lr: 1.0557e-05 eta: 7:00:25 time: 2.4404 data_time: 0.0104 memory: 11133 loss: 0.2094 2024/07/24 21:23:18 - mmengine - INFO - Iter(train) [ 9580/19224] lr: 1.0541e-05 eta: 6:59:56 time: 2.3567 data_time: 0.0108 memory: 10886 loss: 0.2275 2024/07/24 21:23:38 - mmengine - INFO - Iter(train) [ 9590/19224] lr: 1.0524e-05 eta: 6:59:24 time: 2.0420 data_time: 0.0094 memory: 10598 loss: 0.2095 2024/07/24 21:23:55 - mmengine - INFO - Iter(train) [ 9600/19224] lr: 1.0507e-05 eta: 6:58:48 time: 1.6428 data_time: 0.0097 memory: 9962 loss: 0.2381 2024/07/24 21:24:13 - mmengine - INFO - Iter(train) [ 9610/19224] lr: 1.0490e-05 eta: 6:58:14 time: 1.8208 data_time: 0.0089 memory: 12045 loss: 0.2024 2024/07/24 21:24:53 - mmengine - INFO - Iter(train) [ 9620/19224] lr: 1.0473e-05 eta: 6:58:03 time: 4.0560 data_time: 0.2832 memory: 19416 loss: 0.2247 2024/07/24 21:25:24 - mmengine - INFO - Iter(train) [ 9630/19224] lr: 1.0456e-05 eta: 6:57:41 time: 3.0966 data_time: 0.0122 memory: 12562 loss: 0.1607 2024/07/24 21:25:53 - mmengine - INFO - Iter(train) [ 9640/19224] lr: 1.0440e-05 eta: 6:57:18 time: 2.8730 data_time: 0.0110 memory: 11825 loss: 0.1468 2024/07/24 21:26:21 - mmengine - INFO - Iter(train) [ 9650/19224] lr: 1.0423e-05 eta: 6:56:53 time: 2.7450 data_time: 0.0108 memory: 11729 loss: 0.1810 2024/07/24 21:26:47 - mmengine - INFO - Iter(train) [ 9660/19224] lr: 1.0406e-05 eta: 6:56:27 time: 2.5930 data_time: 0.0115 memory: 11285 loss: 0.1674 2024/07/24 21:27:12 - mmengine - INFO - Iter(train) [ 9670/19224] lr: 1.0389e-05 eta: 6:56:00 time: 2.5281 data_time: 0.0115 memory: 11044 loss: 0.1643 2024/07/24 21:27:36 - mmengine - INFO - Iter(train) [ 9680/19224] lr: 1.0372e-05 eta: 6:55:32 time: 2.4568 data_time: 0.0111 memory: 10972 loss: 0.1990 2024/07/24 21:27:59 - mmengine - INFO - Iter(train) [ 9690/19224] lr: 1.0355e-05 eta: 6:55:02 time: 2.2359 data_time: 0.0109 memory: 10727 loss: 0.1926 2024/07/24 21:28:19 - mmengine - INFO - Iter(train) [ 9700/19224] lr: 1.0339e-05 eta: 6:54:30 time: 1.9803 data_time: 0.0096 memory: 10450 loss: 0.1620 2024/07/24 21:28:33 - mmengine - INFO - Iter(train) [ 9710/19224] lr: 1.0322e-05 eta: 6:53:52 time: 1.4228 data_time: 0.0094 memory: 9917 loss: 0.1898 2024/07/24 21:29:03 - mmengine - INFO - Iter(train) [ 9720/19224] lr: 1.0305e-05 eta: 6:53:31 time: 3.0692 data_time: 0.0098 memory: 18895 loss: 0.1649 2024/07/24 21:29:34 - mmengine - INFO - Iter(train) [ 9730/19224] lr: 1.0288e-05 eta: 6:53:08 time: 3.0060 data_time: 0.0116 memory: 12165 loss: 0.1519 2024/07/24 21:30:02 - mmengine - INFO - Iter(train) [ 9740/19224] lr: 1.0271e-05 eta: 6:52:45 time: 2.8808 data_time: 0.0113 memory: 11762 loss: 0.1736 2024/07/24 21:30:33 - mmengine - INFO - Iter(train) [ 9750/19224] lr: 1.0254e-05 eta: 6:52:23 time: 3.0743 data_time: 0.0112 memory: 11450 loss: 0.1603 2024/07/24 21:30:59 - mmengine - INFO - Iter(train) [ 9760/19224] lr: 1.0238e-05 eta: 6:51:57 time: 2.5873 data_time: 0.0107 memory: 11249 loss: 0.1824 2024/07/24 21:31:25 - mmengine - INFO - Iter(train) [ 9770/19224] lr: 1.0221e-05 eta: 6:51:31 time: 2.5798 data_time: 0.0115 memory: 11186 loss: 0.1567 2024/07/24 21:31:50 - mmengine - INFO - Iter(train) [ 9780/19224] lr: 1.0204e-05 eta: 6:51:03 time: 2.4910 data_time: 0.0113 memory: 11008 loss: 0.1818 2024/07/24 21:32:12 - mmengine - INFO - Iter(train) [ 9790/19224] lr: 1.0187e-05 eta: 6:50:34 time: 2.2512 data_time: 0.0104 memory: 10806 loss: 0.1772 2024/07/24 21:32:33 - mmengine - INFO - Iter(train) [ 9800/19224] lr: 1.0170e-05 eta: 6:50:03 time: 2.0960 data_time: 0.0104 memory: 10494 loss: 0.1438 2024/07/24 21:32:48 - mmengine - INFO - Iter(train) [ 9810/19224] lr: 1.0153e-05 eta: 6:49:26 time: 1.5118 data_time: 0.0090 memory: 10038 loss: 0.1846 2024/07/24 21:33:16 - mmengine - INFO - Iter(train) [ 9820/19224] lr: 1.0136e-05 eta: 6:49:01 time: 2.7473 data_time: 0.0098 memory: 13517 loss: 0.1763 2024/07/24 21:33:45 - mmengine - INFO - Iter(train) [ 9830/19224] lr: 1.0120e-05 eta: 6:48:38 time: 2.9024 data_time: 0.0111 memory: 11991 loss: 0.1652 2024/07/24 21:34:13 - mmengine - INFO - Iter(train) [ 9840/19224] lr: 1.0103e-05 eta: 6:48:13 time: 2.7866 data_time: 0.0108 memory: 11702 loss: 0.1595 2024/07/24 21:34:40 - mmengine - INFO - Iter(train) [ 9850/19224] lr: 1.0086e-05 eta: 6:47:49 time: 2.7865 data_time: 0.0109 memory: 11439 loss: 0.1645 2024/07/24 21:35:07 - mmengine - INFO - Iter(train) [ 9860/19224] lr: 1.0069e-05 eta: 6:47:23 time: 2.6702 data_time: 0.0108 memory: 11314 loss: 0.1753 2024/07/24 21:35:33 - mmengine - INFO - Iter(train) [ 9870/19224] lr: 1.0052e-05 eta: 6:46:57 time: 2.5852 data_time: 0.0105 memory: 11175 loss: 0.1894 2024/07/24 21:35:58 - mmengine - INFO - Iter(train) [ 9880/19224] lr: 1.0035e-05 eta: 6:46:30 time: 2.4534 data_time: 0.0105 memory: 11022 loss: 0.2034 2024/07/24 21:36:19 - mmengine - INFO - Iter(train) [ 9890/19224] lr: 1.0019e-05 eta: 6:45:59 time: 2.1493 data_time: 0.0102 memory: 10598 loss: 0.1780 2024/07/24 21:36:39 - mmengine - INFO - Iter(train) [ 9900/19224] lr: 1.0002e-05 eta: 6:45:27 time: 2.0158 data_time: 0.0093 memory: 10235 loss: 0.1835 2024/07/24 21:36:55 - mmengine - INFO - Iter(train) [ 9910/19224] lr: 9.9848e-06 eta: 6:44:52 time: 1.6138 data_time: 0.0095 memory: 9949 loss: 0.1768 2024/07/24 21:37:25 - mmengine - INFO - Iter(train) [ 9920/19224] lr: 9.9680e-06 eta: 6:44:29 time: 2.9301 data_time: 0.0096 memory: 14660 loss: 0.1828 2024/07/24 21:37:56 - mmengine - INFO - Iter(train) [ 9930/19224] lr: 9.9511e-06 eta: 6:44:08 time: 3.1204 data_time: 0.0109 memory: 12236 loss: 0.1760 2024/07/24 21:38:25 - mmengine - INFO - Iter(train) [ 9940/19224] lr: 9.9343e-06 eta: 6:43:45 time: 2.9372 data_time: 0.0104 memory: 11844 loss: 0.1525 2024/07/24 21:38:54 - mmengine - INFO - Iter(train) [ 9950/19224] lr: 9.9175e-06 eta: 6:43:21 time: 2.8370 data_time: 0.0102 memory: 11573 loss: 0.1742 2024/07/24 21:39:21 - mmengine - INFO - Iter(train) [ 9960/19224] lr: 9.9006e-06 eta: 6:42:55 time: 2.7110 data_time: 0.0104 memory: 11580 loss: 0.1784 2024/07/24 21:39:47 - mmengine - INFO - Iter(train) [ 9970/19224] lr: 9.8838e-06 eta: 6:42:30 time: 2.6630 data_time: 0.0103 memory: 11327 loss: 0.2085 2024/07/24 21:40:13 - mmengine - INFO - Iter(train) [ 9980/19224] lr: 9.8669e-06 eta: 6:42:03 time: 2.5743 data_time: 0.0107 memory: 11177 loss: 0.1889 2024/07/24 21:40:38 - mmengine - INFO - Iter(train) [ 9990/19224] lr: 9.8501e-06 eta: 6:41:36 time: 2.4452 data_time: 0.0104 memory: 10935 loss: 0.2090 2024/07/24 21:40:58 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 21:40:58 - mmengine - INFO - Iter(train) [10000/19224] lr: 9.8332e-06 eta: 6:41:05 time: 2.0883 data_time: 0.0111 memory: 10621 loss: 0.1667 2024/07/24 21:40:58 - mmengine - INFO - Saving checkpoint at 10000 iterations 2024/07/24 21:41:19 - mmengine - INFO - Iter(train) [10010/19224] lr: 9.8164e-06 eta: 6:40:33 time: 2.0097 data_time: 0.1951 memory: 10153 loss: 0.1868 2024/07/24 21:41:52 - mmengine - INFO - Iter(train) [10020/19224] lr: 9.7995e-06 eta: 6:40:14 time: 3.3285 data_time: 0.0100 memory: 18443 loss: 0.1977 2024/07/24 21:42:22 - mmengine - INFO - Iter(train) [10030/19224] lr: 9.7827e-06 eta: 6:39:52 time: 3.0592 data_time: 0.0105 memory: 12043 loss: 0.1750 2024/07/24 21:42:52 - mmengine - INFO - Iter(train) [10040/19224] lr: 9.7659e-06 eta: 6:39:29 time: 2.9562 data_time: 0.0110 memory: 11767 loss: 0.1966 2024/07/24 21:43:20 - mmengine - INFO - Iter(train) [10050/19224] lr: 9.7490e-06 eta: 6:39:05 time: 2.8073 data_time: 0.0114 memory: 11781 loss: 0.1666 2024/07/24 21:43:47 - mmengine - INFO - Iter(train) [10060/19224] lr: 9.7322e-06 eta: 6:38:39 time: 2.6744 data_time: 0.0110 memory: 11264 loss: 0.2012 2024/07/24 21:44:13 - mmengine - INFO - Iter(train) [10070/19224] lr: 9.7153e-06 eta: 6:38:13 time: 2.5801 data_time: 0.0106 memory: 11278 loss: 0.1827 2024/07/24 21:44:37 - mmengine - INFO - Iter(train) [10080/19224] lr: 9.6985e-06 eta: 6:37:45 time: 2.4818 data_time: 0.0111 memory: 11108 loss: 0.1967 2024/07/24 21:45:01 - mmengine - INFO - Iter(train) [10090/19224] lr: 9.6816e-06 eta: 6:37:17 time: 2.3425 data_time: 0.0108 memory: 10910 loss: 0.1967 2024/07/24 21:45:21 - mmengine - INFO - Iter(train) [10100/19224] lr: 9.6648e-06 eta: 6:36:46 time: 2.0389 data_time: 0.0100 memory: 10481 loss: 0.2216 2024/07/24 21:45:38 - mmengine - INFO - Iter(train) [10110/19224] lr: 9.6480e-06 eta: 6:36:11 time: 1.6815 data_time: 0.0092 memory: 10085 loss: 0.2027 2024/07/24 21:46:07 - mmengine - INFO - Iter(train) [10120/19224] lr: 9.6311e-06 eta: 6:35:48 time: 2.9045 data_time: 0.0105 memory: 13408 loss: 0.1895 2024/07/24 21:46:38 - mmengine - INFO - Iter(train) [10130/19224] lr: 9.6143e-06 eta: 6:35:26 time: 3.1056 data_time: 0.0106 memory: 12134 loss: 0.1591 2024/07/24 21:47:07 - mmengine - INFO - Iter(train) [10140/19224] lr: 9.5975e-06 eta: 6:35:03 time: 2.8995 data_time: 0.0108 memory: 11832 loss: 0.1882 2024/07/24 21:47:35 - mmengine - INFO - Iter(train) [10150/19224] lr: 9.5806e-06 eta: 6:34:38 time: 2.7996 data_time: 0.0113 memory: 11539 loss: 0.2280 2024/07/24 21:48:02 - mmengine - INFO - Iter(train) [10160/19224] lr: 9.5638e-06 eta: 6:34:13 time: 2.6954 data_time: 0.0116 memory: 11347 loss: 0.1879 2024/07/24 21:48:27 - mmengine - INFO - Iter(train) [10170/19224] lr: 9.5470e-06 eta: 6:33:46 time: 2.5359 data_time: 0.0110 memory: 11177 loss: 0.1873 2024/07/24 21:48:51 - mmengine - INFO - Iter(train) [10180/19224] lr: 9.5301e-06 eta: 6:33:18 time: 2.3786 data_time: 0.0113 memory: 11019 loss: 0.2014 2024/07/24 21:49:13 - mmengine - INFO - Iter(train) [10190/19224] lr: 9.5133e-06 eta: 6:32:48 time: 2.1890 data_time: 0.0096 memory: 10771 loss: 0.1762 2024/07/24 21:49:33 - mmengine - INFO - Iter(train) [10200/19224] lr: 9.4965e-06 eta: 6:32:17 time: 1.9712 data_time: 0.0096 memory: 10349 loss: 0.1648 2024/07/24 21:49:50 - mmengine - INFO - Iter(train) [10210/19224] lr: 9.4797e-06 eta: 6:31:42 time: 1.7019 data_time: 0.0093 memory: 9980 loss: 0.1941 2024/07/24 21:50:21 - mmengine - INFO - Iter(train) [10220/19224] lr: 9.4628e-06 eta: 6:31:21 time: 3.1488 data_time: 0.0102 memory: 14405 loss: 0.1929 2024/07/24 21:50:52 - mmengine - INFO - Iter(train) [10230/19224] lr: 9.4460e-06 eta: 6:30:59 time: 3.0875 data_time: 0.0106 memory: 12157 loss: 0.2063 2024/07/24 21:51:23 - mmengine - INFO - Iter(train) [10240/19224] lr: 9.4292e-06 eta: 6:30:37 time: 3.0326 data_time: 0.0108 memory: 11910 loss: 0.1538 2024/07/24 21:51:52 - mmengine - INFO - Iter(train) [10250/19224] lr: 9.4124e-06 eta: 6:30:13 time: 2.8997 data_time: 0.0109 memory: 11666 loss: 0.1626 2024/07/24 21:52:20 - mmengine - INFO - Iter(train) [10260/19224] lr: 9.3956e-06 eta: 6:29:49 time: 2.8579 data_time: 0.0106 memory: 11956 loss: 0.1712 2024/07/24 21:52:47 - mmengine - INFO - Iter(train) [10270/19224] lr: 9.3788e-06 eta: 6:29:24 time: 2.7058 data_time: 0.0105 memory: 11845 loss: 0.1654 2024/07/24 21:53:13 - mmengine - INFO - Iter(train) [10280/19224] lr: 9.3619e-06 eta: 6:28:58 time: 2.6226 data_time: 0.0105 memory: 11154 loss: 0.1558 2024/07/24 21:53:38 - mmengine - INFO - Iter(train) [10290/19224] lr: 9.3451e-06 eta: 6:28:31 time: 2.4480 data_time: 0.0104 memory: 11005 loss: 0.1722 2024/07/24 21:53:58 - mmengine - INFO - Iter(train) [10300/19224] lr: 9.3283e-06 eta: 6:27:59 time: 2.0033 data_time: 0.0095 memory: 10442 loss: 0.1739 2024/07/24 21:54:14 - mmengine - INFO - Iter(train) [10310/19224] lr: 9.3115e-06 eta: 6:27:25 time: 1.5930 data_time: 0.0091 memory: 9974 loss: 0.1804 2024/07/24 21:54:44 - mmengine - INFO - Iter(train) [10320/19224] lr: 9.2947e-06 eta: 6:27:02 time: 3.0383 data_time: 0.0110 memory: 14445 loss: 0.1718 2024/07/24 21:55:15 - mmengine - INFO - Iter(train) [10330/19224] lr: 9.2779e-06 eta: 6:26:40 time: 3.0615 data_time: 0.0107 memory: 12059 loss: 0.1412 2024/07/24 21:55:44 - mmengine - INFO - Iter(train) [10340/19224] lr: 9.2611e-06 eta: 6:26:17 time: 2.9518 data_time: 0.0110 memory: 11852 loss: 0.1596 2024/07/24 21:56:14 - mmengine - INFO - Iter(train) [10350/19224] lr: 9.2443e-06 eta: 6:25:53 time: 2.9214 data_time: 0.0105 memory: 11618 loss: 0.1899 2024/07/24 21:56:42 - mmengine - INFO - Iter(train) [10360/19224] lr: 9.2275e-06 eta: 6:25:30 time: 2.8911 data_time: 0.0110 memory: 11609 loss: 0.1717 2024/07/24 21:57:10 - mmengine - INFO - Iter(train) [10370/19224] lr: 9.2107e-06 eta: 6:25:05 time: 2.7501 data_time: 0.0110 memory: 11286 loss: 0.1606 2024/07/24 21:57:36 - mmengine - INFO - Iter(train) [10380/19224] lr: 9.1939e-06 eta: 6:24:39 time: 2.6329 data_time: 0.0106 memory: 11195 loss: 0.1775 2024/07/24 21:58:00 - mmengine - INFO - Iter(train) [10390/19224] lr: 9.1771e-06 eta: 6:24:11 time: 2.4195 data_time: 0.0106 memory: 11022 loss: 0.2031 2024/07/24 21:58:21 - mmengine - INFO - Iter(train) [10400/19224] lr: 9.1603e-06 eta: 6:23:41 time: 2.0622 data_time: 0.0104 memory: 10441 loss: 0.1908 2024/07/24 21:58:40 - mmengine - INFO - Iter(train) [10410/19224] lr: 9.1435e-06 eta: 6:23:08 time: 1.8573 data_time: 0.0097 memory: 10146 loss: 0.1751 2024/07/24 21:59:10 - mmengine - INFO - Iter(train) [10420/19224] lr: 9.1268e-06 eta: 6:22:45 time: 3.0131 data_time: 0.0106 memory: 14235 loss: 0.3025 2024/07/24 21:59:41 - mmengine - INFO - Iter(train) [10430/19224] lr: 9.1100e-06 eta: 6:22:23 time: 3.0920 data_time: 0.0112 memory: 12220 loss: 0.1553 2024/07/24 22:00:12 - mmengine - INFO - Iter(train) [10440/19224] lr: 9.0932e-06 eta: 6:22:02 time: 3.1369 data_time: 0.0112 memory: 11860 loss: 0.1586 2024/07/24 22:00:41 - mmengine - INFO - Iter(train) [10450/19224] lr: 9.0764e-06 eta: 6:21:38 time: 2.8706 data_time: 0.0111 memory: 11691 loss: 0.1719 2024/07/24 22:01:09 - mmengine - INFO - Iter(train) [10460/19224] lr: 9.0597e-06 eta: 6:21:13 time: 2.8123 data_time: 0.0106 memory: 11492 loss: 0.1661 2024/07/24 22:01:35 - mmengine - INFO - Iter(train) [10470/19224] lr: 9.0429e-06 eta: 6:20:48 time: 2.6526 data_time: 0.0111 memory: 11268 loss: 0.1967 2024/07/24 22:02:02 - mmengine - INFO - Iter(train) [10480/19224] lr: 9.0261e-06 eta: 6:20:22 time: 2.6334 data_time: 0.0108 memory: 11111 loss: 0.2111 2024/07/24 22:02:26 - mmengine - INFO - Iter(train) [10490/19224] lr: 9.0094e-06 eta: 6:19:54 time: 2.3969 data_time: 0.0109 memory: 10918 loss: 0.2195 2024/07/24 22:02:47 - mmengine - INFO - Iter(train) [10500/19224] lr: 8.9926e-06 eta: 6:19:24 time: 2.1304 data_time: 0.0107 memory: 10702 loss: 0.1745 2024/07/24 22:03:05 - mmengine - INFO - Iter(train) [10510/19224] lr: 8.9758e-06 eta: 6:18:51 time: 1.7458 data_time: 0.0095 memory: 10187 loss: 0.1878 2024/07/24 22:03:33 - mmengine - INFO - Iter(train) [10520/19224] lr: 8.9591e-06 eta: 6:18:26 time: 2.8429 data_time: 0.0106 memory: 13376 loss: 0.1713 2024/07/24 22:04:04 - mmengine - INFO - Iter(train) [10530/19224] lr: 8.9423e-06 eta: 6:18:04 time: 3.0912 data_time: 0.0106 memory: 11911 loss: 0.1573 2024/07/24 22:04:33 - mmengine - INFO - Iter(train) [10540/19224] lr: 8.9256e-06 eta: 6:17:40 time: 2.8750 data_time: 0.0117 memory: 11501 loss: 0.1585 2024/07/24 22:05:02 - mmengine - INFO - Iter(train) [10550/19224] lr: 8.9088e-06 eta: 6:17:17 time: 2.9085 data_time: 0.0116 memory: 11310 loss: 0.1860 2024/07/24 22:05:28 - mmengine - INFO - Iter(train) [10560/19224] lr: 8.8921e-06 eta: 6:16:51 time: 2.6461 data_time: 0.0120 memory: 11268 loss: 0.2070 2024/07/24 22:05:54 - mmengine - INFO - Iter(train) [10570/19224] lr: 8.8753e-06 eta: 6:16:24 time: 2.5447 data_time: 0.0117 memory: 11102 loss: 0.1915 2024/07/24 22:06:18 - mmengine - INFO - Iter(train) [10580/19224] lr: 8.8586e-06 eta: 6:15:57 time: 2.4764 data_time: 0.0110 memory: 11128 loss: 0.2557 2024/07/24 22:06:42 - mmengine - INFO - Iter(train) [10590/19224] lr: 8.8419e-06 eta: 6:15:29 time: 2.3484 data_time: 0.0105 memory: 10813 loss: 0.1932 2024/07/24 22:07:02 - mmengine - INFO - Iter(train) [10600/19224] lr: 8.8251e-06 eta: 6:14:58 time: 2.0392 data_time: 0.0097 memory: 10368 loss: 0.1872 2024/07/24 22:07:18 - mmengine - INFO - Iter(train) [10610/19224] lr: 8.8084e-06 eta: 6:14:24 time: 1.5745 data_time: 0.0090 memory: 10035 loss: 0.1789 2024/07/24 22:07:50 - mmengine - INFO - Iter(train) [10620/19224] lr: 8.7917e-06 eta: 6:14:02 time: 3.1913 data_time: 0.0099 memory: 15840 loss: 0.2039 2024/07/24 22:08:21 - mmengine - INFO - Iter(train) [10630/19224] lr: 8.7750e-06 eta: 6:13:40 time: 3.0831 data_time: 0.0110 memory: 12040 loss: 0.1526 2024/07/24 22:08:50 - mmengine - INFO - Iter(train) [10640/19224] lr: 8.7582e-06 eta: 6:13:17 time: 2.9352 data_time: 0.0115 memory: 11945 loss: 0.1570 2024/07/24 22:09:19 - mmengine - INFO - Iter(train) [10650/19224] lr: 8.7415e-06 eta: 6:12:52 time: 2.8418 data_time: 0.0108 memory: 11593 loss: 0.1941 2024/07/24 22:09:46 - mmengine - INFO - Iter(train) [10660/19224] lr: 8.7248e-06 eta: 6:12:28 time: 2.7766 data_time: 0.0104 memory: 11332 loss: 0.1665 2024/07/24 22:10:13 - mmengine - INFO - Iter(train) [10670/19224] lr: 8.7081e-06 eta: 6:12:02 time: 2.6917 data_time: 0.0107 memory: 11244 loss: 0.1859 2024/07/24 22:10:39 - mmengine - INFO - Iter(train) [10680/19224] lr: 8.6914e-06 eta: 6:11:36 time: 2.5735 data_time: 0.0114 memory: 11072 loss: 0.1878 2024/07/24 22:11:04 - mmengine - INFO - Iter(train) [10690/19224] lr: 8.6747e-06 eta: 6:11:09 time: 2.4576 data_time: 0.0119 memory: 10881 loss: 0.1644 2024/07/24 22:11:25 - mmengine - INFO - Iter(train) [10700/19224] lr: 8.6580e-06 eta: 6:10:39 time: 2.1609 data_time: 0.0105 memory: 10687 loss: 0.2002 2024/07/24 22:11:42 - mmengine - INFO - Iter(train) [10710/19224] lr: 8.6413e-06 eta: 6:10:06 time: 1.7104 data_time: 0.0108 memory: 10128 loss: 0.2212 2024/07/24 22:12:13 - mmengine - INFO - Iter(train) [10720/19224] lr: 8.6246e-06 eta: 6:09:44 time: 3.1250 data_time: 0.0105 memory: 14347 loss: 0.1643 2024/07/24 22:12:45 - mmengine - INFO - Iter(train) [10730/19224] lr: 8.6079e-06 eta: 6:09:22 time: 3.1580 data_time: 0.0109 memory: 12187 loss: 0.1727 2024/07/24 22:13:15 - mmengine - INFO - Iter(train) [10740/19224] lr: 8.5913e-06 eta: 6:08:59 time: 3.0027 data_time: 0.0112 memory: 11825 loss: 0.1823 2024/07/24 22:13:44 - mmengine - INFO - Iter(train) [10750/19224] lr: 8.5746e-06 eta: 6:08:35 time: 2.8704 data_time: 0.0111 memory: 11611 loss: 0.1555 2024/07/24 22:14:12 - mmengine - INFO - Iter(train) [10760/19224] lr: 8.5579e-06 eta: 6:08:11 time: 2.8383 data_time: 0.0109 memory: 11388 loss: 0.1790 2024/07/24 22:14:40 - mmengine - INFO - Iter(train) [10770/19224] lr: 8.5412e-06 eta: 6:07:46 time: 2.8139 data_time: 0.0113 memory: 11285 loss: 0.1670 2024/07/24 22:15:05 - mmengine - INFO - Iter(train) [10780/19224] lr: 8.5246e-06 eta: 6:07:19 time: 2.4799 data_time: 0.0110 memory: 11075 loss: 0.1843 2024/07/24 22:15:26 - mmengine - INFO - Iter(train) [10790/19224] lr: 8.5079e-06 eta: 6:06:49 time: 2.1209 data_time: 0.0099 memory: 10759 loss: 0.1889 2024/07/24 22:15:45 - mmengine - INFO - Iter(train) [10800/19224] lr: 8.4913e-06 eta: 6:06:17 time: 1.8269 data_time: 0.0099 memory: 10250 loss: 0.1803 2024/07/24 22:15:58 - mmengine - INFO - Iter(train) [10810/19224] lr: 8.4746e-06 eta: 6:05:41 time: 1.3820 data_time: 0.0086 memory: 9628 loss: 0.1677 2024/07/24 22:16:26 - mmengine - INFO - Iter(train) [10820/19224] lr: 8.4580e-06 eta: 6:05:16 time: 2.7724 data_time: 0.0102 memory: 13663 loss: 0.1607 2024/07/24 22:16:56 - mmengine - INFO - Iter(train) [10830/19224] lr: 8.4413e-06 eta: 6:04:53 time: 3.0097 data_time: 0.0116 memory: 12236 loss: 0.1718 2024/07/24 22:17:26 - mmengine - INFO - Iter(train) [10840/19224] lr: 8.4247e-06 eta: 6:04:30 time: 2.9317 data_time: 0.0111 memory: 11654 loss: 0.1619 2024/07/24 22:17:54 - mmengine - INFO - Iter(train) [10850/19224] lr: 8.4080e-06 eta: 6:04:06 time: 2.8411 data_time: 0.0136 memory: 11571 loss: 0.1610 2024/07/24 22:18:21 - mmengine - INFO - Iter(train) [10860/19224] lr: 8.3914e-06 eta: 6:03:40 time: 2.7002 data_time: 0.0117 memory: 11365 loss: 0.1666 2024/07/24 22:18:47 - mmengine - INFO - Iter(train) [10870/19224] lr: 8.3748e-06 eta: 6:03:14 time: 2.6360 data_time: 0.0112 memory: 11179 loss: 0.1714 2024/07/24 22:19:13 - mmengine - INFO - Iter(train) [10880/19224] lr: 8.3582e-06 eta: 6:02:48 time: 2.6114 data_time: 0.0120 memory: 11154 loss: 0.1888 2024/07/24 22:19:36 - mmengine - INFO - Iter(train) [10890/19224] lr: 8.3415e-06 eta: 6:02:20 time: 2.3054 data_time: 0.0110 memory: 10823 loss: 0.1836 2024/07/24 22:19:56 - mmengine - INFO - Iter(train) [10900/19224] lr: 8.3249e-06 eta: 6:01:49 time: 1.9906 data_time: 0.0099 memory: 10316 loss: 0.1745 2024/07/24 22:20:13 - mmengine - INFO - Iter(train) [10910/19224] lr: 8.3083e-06 eta: 6:01:16 time: 1.6597 data_time: 0.0096 memory: 9942 loss: 0.2209 2024/07/24 22:20:43 - mmengine - INFO - Iter(train) [10920/19224] lr: 8.2917e-06 eta: 6:00:53 time: 3.0424 data_time: 0.0095 memory: 15692 loss: 0.1742 2024/07/24 22:21:14 - mmengine - INFO - Iter(train) [10930/19224] lr: 8.2751e-06 eta: 6:00:31 time: 3.0935 data_time: 0.0102 memory: 12215 loss: 0.1592 2024/07/24 22:21:44 - mmengine - INFO - Iter(train) [10940/19224] lr: 8.2585e-06 eta: 6:00:07 time: 2.9921 data_time: 0.0107 memory: 12031 loss: 0.1774 2024/07/24 22:22:13 - mmengine - INFO - Iter(train) [10950/19224] lr: 8.2420e-06 eta: 5:59:43 time: 2.8836 data_time: 0.0126 memory: 11501 loss: 0.1664 2024/07/24 22:22:40 - mmengine - INFO - Iter(train) [10960/19224] lr: 8.2254e-06 eta: 5:59:18 time: 2.7109 data_time: 0.0111 memory: 11278 loss: 0.2126 2024/07/24 22:23:07 - mmengine - INFO - Iter(train) [10970/19224] lr: 8.2088e-06 eta: 5:58:52 time: 2.6471 data_time: 0.0114 memory: 11210 loss: 0.1830 2024/07/24 22:23:31 - mmengine - INFO - Iter(train) [10980/19224] lr: 8.1922e-06 eta: 5:58:25 time: 2.4007 data_time: 0.0108 memory: 10914 loss: 0.1790 2024/07/24 22:23:52 - mmengine - INFO - Iter(train) [10990/19224] lr: 8.1757e-06 eta: 5:57:55 time: 2.0857 data_time: 0.0100 memory: 10554 loss: 0.1751 2024/07/24 22:24:10 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 22:24:10 - mmengine - INFO - Iter(train) [11000/19224] lr: 8.1591e-06 eta: 5:57:23 time: 1.8690 data_time: 0.0092 memory: 10251 loss: 0.2001 2024/07/24 22:24:10 - mmengine - INFO - Saving checkpoint at 11000 iterations 2024/07/24 22:24:28 - mmengine - INFO - Iter(train) [11010/19224] lr: 8.1425e-06 eta: 5:56:50 time: 1.7356 data_time: 0.2082 memory: 9948 loss: 0.1856 2024/07/24 22:24:56 - mmengine - INFO - Iter(train) [11020/19224] lr: 8.1260e-06 eta: 5:56:26 time: 2.8134 data_time: 0.0096 memory: 14368 loss: 0.1531 2024/07/24 22:25:27 - mmengine - INFO - Iter(train) [11030/19224] lr: 8.1094e-06 eta: 5:56:04 time: 3.1198 data_time: 0.0106 memory: 12015 loss: 0.1610 2024/07/24 22:25:57 - mmengine - INFO - Iter(train) [11040/19224] lr: 8.0929e-06 eta: 5:55:40 time: 2.9721 data_time: 0.0103 memory: 12066 loss: 0.1846 2024/07/24 22:26:25 - mmengine - INFO - Iter(train) [11050/19224] lr: 8.0764e-06 eta: 5:55:16 time: 2.8708 data_time: 0.0109 memory: 11484 loss: 0.1634 2024/07/24 22:26:52 - mmengine - INFO - Iter(train) [11060/19224] lr: 8.0598e-06 eta: 5:54:51 time: 2.7041 data_time: 0.0107 memory: 11417 loss: 0.1740 2024/07/24 22:27:18 - mmengine - INFO - Iter(train) [11070/19224] lr: 8.0433e-06 eta: 5:54:24 time: 2.5563 data_time: 0.0107 memory: 11190 loss: 0.1628 2024/07/24 22:27:42 - mmengine - INFO - Iter(train) [11080/19224] lr: 8.0268e-06 eta: 5:53:57 time: 2.4372 data_time: 0.0105 memory: 10975 loss: 0.1931 2024/07/24 22:28:03 - mmengine - INFO - Iter(train) [11090/19224] lr: 8.0103e-06 eta: 5:53:27 time: 2.1115 data_time: 0.0094 memory: 10654 loss: 0.1787 2024/07/24 22:28:23 - mmengine - INFO - Iter(train) [11100/19224] lr: 7.9938e-06 eta: 5:52:56 time: 1.9395 data_time: 0.0096 memory: 10273 loss: 0.1833 2024/07/24 22:28:39 - mmengine - INFO - Iter(train) [11110/19224] lr: 7.9773e-06 eta: 5:52:23 time: 1.5867 data_time: 0.0091 memory: 9929 loss: 0.1998 2024/07/24 22:29:08 - mmengine - INFO - Iter(train) [11120/19224] lr: 7.9608e-06 eta: 5:51:59 time: 2.9499 data_time: 0.0105 memory: 13771 loss: 0.1996 2024/07/24 22:29:38 - mmengine - INFO - Iter(train) [11130/19224] lr: 7.9443e-06 eta: 5:51:36 time: 2.9943 data_time: 0.0109 memory: 12097 loss: 0.1674 2024/07/24 22:30:09 - mmengine - INFO - Iter(train) [11140/19224] lr: 7.9278e-06 eta: 5:51:13 time: 3.0543 data_time: 0.0112 memory: 11729 loss: 0.1640 2024/07/24 22:30:36 - mmengine - INFO - Iter(train) [11150/19224] lr: 7.9113e-06 eta: 5:50:48 time: 2.7369 data_time: 0.0104 memory: 11376 loss: 0.1631 2024/07/24 22:31:03 - mmengine - INFO - Iter(train) [11160/19224] lr: 7.8949e-06 eta: 5:50:22 time: 2.6552 data_time: 0.0107 memory: 11332 loss: 0.1767 2024/07/24 22:31:29 - mmengine - INFO - Iter(train) [11170/19224] lr: 7.8784e-06 eta: 5:49:57 time: 2.6335 data_time: 0.0104 memory: 11198 loss: 0.1917 2024/07/24 22:31:54 - mmengine - INFO - Iter(train) [11180/19224] lr: 7.8619e-06 eta: 5:49:29 time: 2.4663 data_time: 0.0105 memory: 11013 loss: 0.1642 2024/07/24 22:32:17 - mmengine - INFO - Iter(train) [11190/19224] lr: 7.8455e-06 eta: 5:49:01 time: 2.3293 data_time: 0.0099 memory: 10786 loss: 0.3002 2024/07/24 22:32:37 - mmengine - INFO - Iter(train) [11200/19224] lr: 7.8290e-06 eta: 5:48:31 time: 1.9827 data_time: 0.0095 memory: 10454 loss: 0.1925 2024/07/24 22:32:53 - mmengine - INFO - Iter(train) [11210/19224] lr: 7.8126e-06 eta: 5:47:58 time: 1.6403 data_time: 0.0094 memory: 9944 loss: 0.1807 2024/07/24 22:33:25 - mmengine - INFO - Iter(train) [11220/19224] lr: 7.7961e-06 eta: 5:47:36 time: 3.1403 data_time: 0.0097 memory: 14439 loss: 0.2261 2024/07/24 22:33:57 - mmengine - INFO - Iter(train) [11230/19224] lr: 7.7797e-06 eta: 5:47:14 time: 3.2917 data_time: 0.0102 memory: 12516 loss: 0.1573 2024/07/24 22:34:28 - mmengine - INFO - Iter(train) [11240/19224] lr: 7.7633e-06 eta: 5:46:51 time: 3.0277 data_time: 0.0114 memory: 12057 loss: 0.1856 2024/07/24 22:34:56 - mmengine - INFO - Iter(train) [11250/19224] lr: 7.7469e-06 eta: 5:46:27 time: 2.8561 data_time: 0.0106 memory: 12203 loss: 0.1617 2024/07/24 22:35:24 - mmengine - INFO - Iter(train) [11260/19224] lr: 7.7305e-06 eta: 5:46:02 time: 2.7310 data_time: 0.0111 memory: 11535 loss: 0.1699 2024/07/24 22:35:49 - mmengine - INFO - Iter(train) [11270/19224] lr: 7.7141e-06 eta: 5:45:35 time: 2.5130 data_time: 0.0105 memory: 11183 loss: 0.1612 2024/07/24 22:36:13 - mmengine - INFO - Iter(train) [11280/19224] lr: 7.6977e-06 eta: 5:45:08 time: 2.3832 data_time: 0.0100 memory: 11019 loss: 0.1879 2024/07/24 22:36:32 - mmengine - INFO - Iter(train) [11290/19224] lr: 7.6813e-06 eta: 5:44:37 time: 1.9696 data_time: 0.0092 memory: 10584 loss: 0.1746 2024/07/24 22:36:50 - mmengine - INFO - Iter(train) [11300/19224] lr: 7.6649e-06 eta: 5:44:05 time: 1.7792 data_time: 0.0091 memory: 10114 loss: 0.1839 2024/07/24 22:37:06 - mmengine - INFO - Iter(train) [11310/19224] lr: 7.6485e-06 eta: 5:43:32 time: 1.5540 data_time: 0.0089 memory: 9891 loss: 0.1589 2024/07/24 22:37:34 - mmengine - INFO - Iter(train) [11320/19224] lr: 7.6321e-06 eta: 5:43:07 time: 2.8131 data_time: 0.0099 memory: 14140 loss: 0.1541 2024/07/24 22:38:04 - mmengine - INFO - Iter(train) [11330/19224] lr: 7.6158e-06 eta: 5:42:44 time: 3.0418 data_time: 0.0102 memory: 12298 loss: 0.1586 2024/07/24 22:38:33 - mmengine - INFO - Iter(train) [11340/19224] lr: 7.5994e-06 eta: 5:42:20 time: 2.8766 data_time: 0.0107 memory: 11762 loss: 0.1827 2024/07/24 22:39:01 - mmengine - INFO - Iter(train) [11350/19224] lr: 7.5831e-06 eta: 5:41:55 time: 2.8294 data_time: 0.0103 memory: 11591 loss: 0.1561 2024/07/24 22:39:28 - mmengine - INFO - Iter(train) [11360/19224] lr: 7.5667e-06 eta: 5:41:30 time: 2.6790 data_time: 0.0106 memory: 11321 loss: 0.1782 2024/07/24 22:39:54 - mmengine - INFO - Iter(train) [11370/19224] lr: 7.5504e-06 eta: 5:41:04 time: 2.6438 data_time: 0.0105 memory: 11217 loss: 0.1925 2024/07/24 22:40:19 - mmengine - INFO - Iter(train) [11380/19224] lr: 7.5341e-06 eta: 5:40:37 time: 2.4829 data_time: 0.0103 memory: 11097 loss: 0.1787 2024/07/24 22:40:42 - mmengine - INFO - Iter(train) [11390/19224] lr: 7.5177e-06 eta: 5:40:09 time: 2.2332 data_time: 0.0101 memory: 10924 loss: 0.1550 2024/07/24 22:41:00 - mmengine - INFO - Iter(train) [11400/19224] lr: 7.5014e-06 eta: 5:39:37 time: 1.8553 data_time: 0.0091 memory: 10393 loss: 0.1687 2024/07/24 22:41:15 - mmengine - INFO - Iter(train) [11410/19224] lr: 7.4851e-06 eta: 5:39:04 time: 1.5064 data_time: 0.0094 memory: 9665 loss: 0.1986 2024/07/24 22:41:45 - mmengine - INFO - Iter(train) [11420/19224] lr: 7.4688e-06 eta: 5:38:40 time: 2.9685 data_time: 0.0102 memory: 14331 loss: 0.1775 2024/07/24 22:42:16 - mmengine - INFO - Iter(train) [11430/19224] lr: 7.4525e-06 eta: 5:38:18 time: 3.1334 data_time: 0.0106 memory: 12187 loss: 0.1581 2024/07/24 22:42:46 - mmengine - INFO - Iter(train) [11440/19224] lr: 7.4362e-06 eta: 5:37:54 time: 2.9292 data_time: 0.0106 memory: 11942 loss: 0.1675 2024/07/24 22:43:14 - mmengine - INFO - Iter(train) [11450/19224] lr: 7.4199e-06 eta: 5:37:30 time: 2.8506 data_time: 0.0103 memory: 11613 loss: 0.1622 2024/07/24 22:43:41 - mmengine - INFO - Iter(train) [11460/19224] lr: 7.4037e-06 eta: 5:37:04 time: 2.6846 data_time: 0.0113 memory: 11380 loss: 0.1659 2024/07/24 22:44:07 - mmengine - INFO - Iter(train) [11470/19224] lr: 7.3874e-06 eta: 5:36:38 time: 2.6151 data_time: 0.0103 memory: 11276 loss: 0.1656 2024/07/24 22:44:32 - mmengine - INFO - Iter(train) [11480/19224] lr: 7.3712e-06 eta: 5:36:11 time: 2.4879 data_time: 0.0109 memory: 11149 loss: 0.1960 2024/07/24 22:44:55 - mmengine - INFO - Iter(train) [11490/19224] lr: 7.3549e-06 eta: 5:35:43 time: 2.3394 data_time: 0.0105 memory: 10956 loss: 0.2024 2024/07/24 22:45:16 - mmengine - INFO - Iter(train) [11500/19224] lr: 7.3387e-06 eta: 5:35:14 time: 2.0498 data_time: 0.0092 memory: 10472 loss: 0.1812 2024/07/24 22:45:33 - mmengine - INFO - Iter(train) [11510/19224] lr: 7.3224e-06 eta: 5:34:41 time: 1.6798 data_time: 0.0092 memory: 10165 loss: 0.1574 2024/07/24 22:46:00 - mmengine - INFO - Iter(train) [11520/19224] lr: 7.3062e-06 eta: 5:34:16 time: 2.7434 data_time: 0.0106 memory: 12807 loss: 0.2010 2024/07/24 22:46:31 - mmengine - INFO - Iter(train) [11530/19224] lr: 7.2900e-06 eta: 5:33:54 time: 3.0940 data_time: 0.0111 memory: 12120 loss: 0.1667 2024/07/24 22:47:00 - mmengine - INFO - Iter(train) [11540/19224] lr: 7.2738e-06 eta: 5:33:29 time: 2.8605 data_time: 0.0106 memory: 11745 loss: 0.1649 2024/07/24 22:47:27 - mmengine - INFO - Iter(train) [11550/19224] lr: 7.2576e-06 eta: 5:33:04 time: 2.7550 data_time: 0.0109 memory: 11526 loss: 0.1702 2024/07/24 22:47:53 - mmengine - INFO - Iter(train) [11560/19224] lr: 7.2414e-06 eta: 5:32:38 time: 2.5661 data_time: 0.0104 memory: 11198 loss: 0.1889 2024/07/24 22:48:19 - mmengine - INFO - Iter(train) [11570/19224] lr: 7.2252e-06 eta: 5:32:12 time: 2.6086 data_time: 0.0108 memory: 11163 loss: 0.1632 2024/07/24 22:48:44 - mmengine - INFO - Iter(train) [11580/19224] lr: 7.2090e-06 eta: 5:31:45 time: 2.5082 data_time: 0.0106 memory: 11037 loss: 0.1782 2024/07/24 22:49:07 - mmengine - INFO - Iter(train) [11590/19224] lr: 7.1928e-06 eta: 5:31:17 time: 2.2552 data_time: 0.0100 memory: 10798 loss: 0.1628 2024/07/24 22:49:27 - mmengine - INFO - Iter(train) [11600/19224] lr: 7.1767e-06 eta: 5:30:47 time: 1.9984 data_time: 0.0092 memory: 10348 loss: 0.1690 2024/07/24 22:49:43 - mmengine - INFO - Iter(train) [11610/19224] lr: 7.1605e-06 eta: 5:30:15 time: 1.6772 data_time: 0.0093 memory: 9935 loss: 0.1733 2024/07/24 22:50:13 - mmengine - INFO - Iter(train) [11620/19224] lr: 7.1443e-06 eta: 5:29:51 time: 2.9445 data_time: 0.0100 memory: 13234 loss: 0.1989 2024/07/24 22:50:44 - mmengine - INFO - Iter(train) [11630/19224] lr: 7.1282e-06 eta: 5:29:28 time: 3.1217 data_time: 0.0105 memory: 12111 loss: 0.1588 2024/07/24 22:51:14 - mmengine - INFO - Iter(train) [11640/19224] lr: 7.1121e-06 eta: 5:29:05 time: 2.9719 data_time: 0.0105 memory: 11818 loss: 0.1761 2024/07/24 22:51:42 - mmengine - INFO - Iter(train) [11650/19224] lr: 7.0959e-06 eta: 5:28:40 time: 2.8486 data_time: 0.0107 memory: 11555 loss: 0.1837 2024/07/24 22:52:10 - mmengine - INFO - Iter(train) [11660/19224] lr: 7.0798e-06 eta: 5:28:15 time: 2.7740 data_time: 0.0114 memory: 11430 loss: 0.1505 2024/07/24 22:52:36 - mmengine - INFO - Iter(train) [11670/19224] lr: 7.0637e-06 eta: 5:27:49 time: 2.6065 data_time: 0.0113 memory: 11252 loss: 0.1600 2024/07/24 22:53:01 - mmengine - INFO - Iter(train) [11680/19224] lr: 7.0476e-06 eta: 5:27:23 time: 2.4936 data_time: 0.0109 memory: 10996 loss: 0.2325 2024/07/24 22:53:23 - mmengine - INFO - Iter(train) [11690/19224] lr: 7.0315e-06 eta: 5:26:54 time: 2.1906 data_time: 0.0102 memory: 10704 loss: 0.2334 2024/07/24 22:53:43 - mmengine - INFO - Iter(train) [11700/19224] lr: 7.0154e-06 eta: 5:26:24 time: 2.0178 data_time: 0.0100 memory: 10502 loss: 0.1814 2024/07/24 22:53:58 - mmengine - INFO - Iter(train) [11710/19224] lr: 6.9994e-06 eta: 5:25:51 time: 1.5325 data_time: 0.0092 memory: 9935 loss: 0.2140 2024/07/24 22:54:32 - mmengine - INFO - Iter(train) [11720/19224] lr: 6.9833e-06 eta: 5:25:30 time: 3.3943 data_time: 0.0099 memory: 18895 loss: 0.1835 2024/07/24 22:55:03 - mmengine - INFO - Iter(train) [11730/19224] lr: 6.9672e-06 eta: 5:25:07 time: 3.0880 data_time: 0.0107 memory: 12480 loss: 0.1723 2024/07/24 22:55:33 - mmengine - INFO - Iter(train) [11740/19224] lr: 6.9512e-06 eta: 5:24:44 time: 3.0246 data_time: 0.0110 memory: 12460 loss: 0.1629 2024/07/24 22:56:04 - mmengine - INFO - Iter(train) [11750/19224] lr: 6.9352e-06 eta: 5:24:21 time: 3.0152 data_time: 0.0107 memory: 11640 loss: 0.1651 2024/07/24 22:56:31 - mmengine - INFO - Iter(train) [11760/19224] lr: 6.9191e-06 eta: 5:23:55 time: 2.7334 data_time: 0.0102 memory: 11790 loss: 0.1819 2024/07/24 22:56:57 - mmengine - INFO - Iter(train) [11770/19224] lr: 6.9031e-06 eta: 5:23:30 time: 2.6494 data_time: 0.0105 memory: 11267 loss: 0.1870 2024/07/24 22:57:22 - mmengine - INFO - Iter(train) [11780/19224] lr: 6.8871e-06 eta: 5:23:03 time: 2.4965 data_time: 0.0107 memory: 11149 loss: 0.1871 2024/07/24 22:57:46 - mmengine - INFO - Iter(train) [11790/19224] lr: 6.8711e-06 eta: 5:22:35 time: 2.3885 data_time: 0.0107 memory: 10836 loss: 0.2541 2024/07/24 22:58:05 - mmengine - INFO - Iter(train) [11800/19224] lr: 6.8551e-06 eta: 5:22:05 time: 1.9094 data_time: 0.0096 memory: 10258 loss: 0.2091 2024/07/24 22:58:22 - mmengine - INFO - Iter(train) [11810/19224] lr: 6.8391e-06 eta: 5:21:33 time: 1.6665 data_time: 0.0089 memory: 9948 loss: 0.1876 2024/07/24 22:58:52 - mmengine - INFO - Iter(train) [11820/19224] lr: 6.8231e-06 eta: 5:21:09 time: 2.9754 data_time: 0.0103 memory: 12940 loss: 0.1628 2024/07/24 22:59:24 - mmengine - INFO - Iter(train) [11830/19224] lr: 6.8071e-06 eta: 5:20:47 time: 3.1785 data_time: 0.0112 memory: 12313 loss: 0.1618 2024/07/24 22:59:53 - mmengine - INFO - Iter(train) [11840/19224] lr: 6.7912e-06 eta: 5:20:23 time: 2.9573 data_time: 0.0105 memory: 12085 loss: 0.1864 2024/07/24 23:00:24 - mmengine - INFO - Iter(train) [11850/19224] lr: 6.7752e-06 eta: 5:20:00 time: 3.1100 data_time: 0.0117 memory: 11528 loss: 0.1599 2024/07/24 23:00:51 - mmengine - INFO - Iter(train) [11860/19224] lr: 6.7593e-06 eta: 5:19:35 time: 2.7119 data_time: 0.0114 memory: 11323 loss: 0.1724 2024/07/24 23:01:17 - mmengine - INFO - Iter(train) [11870/19224] lr: 6.7434e-06 eta: 5:19:09 time: 2.6060 data_time: 0.0111 memory: 11269 loss: 0.1739 2024/07/24 23:01:43 - mmengine - INFO - Iter(train) [11880/19224] lr: 6.7274e-06 eta: 5:18:43 time: 2.5487 data_time: 0.0102 memory: 11042 loss: 0.1686 2024/07/24 23:02:07 - mmengine - INFO - Iter(train) [11890/19224] lr: 6.7115e-06 eta: 5:18:16 time: 2.4414 data_time: 0.0107 memory: 10934 loss: 0.1841 2024/07/24 23:02:27 - mmengine - INFO - Iter(train) [11900/19224] lr: 6.6956e-06 eta: 5:17:45 time: 1.9266 data_time: 0.0098 memory: 10457 loss: 0.1984 2024/07/24 23:02:43 - mmengine - INFO - Iter(train) [11910/19224] lr: 6.6797e-06 eta: 5:17:13 time: 1.6177 data_time: 0.0088 memory: 10029 loss: 0.1887 2024/07/24 23:03:15 - mmengine - INFO - Iter(train) [11920/19224] lr: 6.6638e-06 eta: 5:16:51 time: 3.2366 data_time: 0.0108 memory: 18474 loss: 0.1745 2024/07/24 23:03:46 - mmengine - INFO - Iter(train) [11930/19224] lr: 6.6480e-06 eta: 5:16:28 time: 3.0638 data_time: 0.0109 memory: 12141 loss: 0.1667 2024/07/24 23:04:15 - mmengine - INFO - Iter(train) [11940/19224] lr: 6.6321e-06 eta: 5:16:04 time: 2.9495 data_time: 0.0109 memory: 11798 loss: 0.1731 2024/07/24 23:04:44 - mmengine - INFO - Iter(train) [11950/19224] lr: 6.6162e-06 eta: 5:15:40 time: 2.8758 data_time: 0.0114 memory: 12118 loss: 0.1656 2024/07/24 23:05:11 - mmengine - INFO - Iter(train) [11960/19224] lr: 6.6004e-06 eta: 5:15:14 time: 2.7092 data_time: 0.0119 memory: 11352 loss: 0.1724 2024/07/24 23:05:38 - mmengine - INFO - Iter(train) [11970/19224] lr: 6.5845e-06 eta: 5:14:48 time: 2.6620 data_time: 0.0102 memory: 11202 loss: 0.1621 2024/07/24 23:06:05 - mmengine - INFO - Iter(train) [11980/19224] lr: 6.5687e-06 eta: 5:14:23 time: 2.7018 data_time: 0.0112 memory: 11178 loss: 0.1750 2024/07/24 23:06:29 - mmengine - INFO - Iter(train) [11990/19224] lr: 6.5529e-06 eta: 5:13:56 time: 2.4111 data_time: 0.0111 memory: 10947 loss: 0.1882 2024/07/24 23:06:49 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 23:06:49 - mmengine - INFO - Iter(train) [12000/19224] lr: 6.5371e-06 eta: 5:13:26 time: 2.0447 data_time: 0.0101 memory: 10587 loss: 0.2268 2024/07/24 23:06:49 - mmengine - INFO - Saving checkpoint at 12000 iterations 2024/07/24 23:07:09 - mmengine - INFO - Iter(train) [12010/19224] lr: 6.5213e-06 eta: 5:12:57 time: 1.9857 data_time: 0.2473 memory: 10113 loss: 0.1947 2024/07/24 23:07:41 - mmengine - INFO - Iter(train) [12020/19224] lr: 6.5055e-06 eta: 5:12:34 time: 3.1460 data_time: 0.0105 memory: 15563 loss: 0.2029 2024/07/24 23:08:11 - mmengine - INFO - Iter(train) [12030/19224] lr: 6.4897e-06 eta: 5:12:11 time: 3.0737 data_time: 0.0115 memory: 12199 loss: 0.1774 2024/07/24 23:08:41 - mmengine - INFO - Iter(train) [12040/19224] lr: 6.4740e-06 eta: 5:11:47 time: 2.9742 data_time: 0.0108 memory: 12132 loss: 0.1746 2024/07/24 23:09:09 - mmengine - INFO - Iter(train) [12050/19224] lr: 6.4582e-06 eta: 5:11:22 time: 2.8192 data_time: 0.0107 memory: 11546 loss: 0.1801 2024/07/24 23:09:36 - mmengine - INFO - Iter(train) [12060/19224] lr: 6.4424e-06 eta: 5:10:57 time: 2.6891 data_time: 0.0109 memory: 11343 loss: 0.1639 2024/07/24 23:10:02 - mmengine - INFO - Iter(train) [12070/19224] lr: 6.4267e-06 eta: 5:10:30 time: 2.5919 data_time: 0.0105 memory: 11188 loss: 0.1749 2024/07/24 23:10:26 - mmengine - INFO - Iter(train) [12080/19224] lr: 6.4110e-06 eta: 5:10:03 time: 2.4079 data_time: 0.0110 memory: 11017 loss: 0.1795 2024/07/24 23:10:47 - mmengine - INFO - Iter(train) [12090/19224] lr: 6.3953e-06 eta: 5:09:34 time: 2.0796 data_time: 0.0098 memory: 10717 loss: 0.1609 2024/07/24 23:11:06 - mmengine - INFO - Iter(train) [12100/19224] lr: 6.3795e-06 eta: 5:09:04 time: 1.8664 data_time: 0.0100 memory: 10242 loss: 0.1952 2024/07/24 23:11:22 - mmengine - INFO - Iter(train) [12110/19224] lr: 6.3638e-06 eta: 5:08:32 time: 1.5906 data_time: 0.0095 memory: 9961 loss: 0.1882 2024/07/24 23:11:53 - mmengine - INFO - Iter(train) [12120/19224] lr: 6.3482e-06 eta: 5:08:09 time: 3.1520 data_time: 0.0099 memory: 16358 loss: 0.1957 2024/07/24 23:12:23 - mmengine - INFO - Iter(train) [12130/19224] lr: 6.3325e-06 eta: 5:07:45 time: 3.0379 data_time: 0.0107 memory: 12192 loss: 0.1562 2024/07/24 23:12:53 - mmengine - INFO - Iter(train) [12140/19224] lr: 6.3168e-06 eta: 5:07:22 time: 3.0001 data_time: 0.0105 memory: 11823 loss: 0.1488 2024/07/24 23:13:22 - mmengine - INFO - Iter(train) [12150/19224] lr: 6.3012e-06 eta: 5:06:57 time: 2.8243 data_time: 0.0106 memory: 11573 loss: 0.1612 2024/07/24 23:13:49 - mmengine - INFO - Iter(train) [12160/19224] lr: 6.2855e-06 eta: 5:06:32 time: 2.6965 data_time: 0.0108 memory: 11334 loss: 0.1806 2024/07/24 23:14:15 - mmengine - INFO - Iter(train) [12170/19224] lr: 6.2699e-06 eta: 5:06:06 time: 2.6786 data_time: 0.0101 memory: 11255 loss: 0.1938 2024/07/24 23:14:41 - mmengine - INFO - Iter(train) [12180/19224] lr: 6.2542e-06 eta: 5:05:40 time: 2.5982 data_time: 0.0106 memory: 11118 loss: 0.1832 2024/07/24 23:15:05 - mmengine - INFO - Iter(train) [12190/19224] lr: 6.2386e-06 eta: 5:05:12 time: 2.3722 data_time: 0.0103 memory: 11024 loss: 0.1694 2024/07/24 23:15:24 - mmengine - INFO - Iter(train) [12200/19224] lr: 6.2230e-06 eta: 5:04:42 time: 1.9064 data_time: 0.0111 memory: 10329 loss: 0.1695 2024/07/24 23:15:41 - mmengine - INFO - Iter(train) [12210/19224] lr: 6.2074e-06 eta: 5:04:11 time: 1.6758 data_time: 0.0093 memory: 9992 loss: 0.2113 2024/07/24 23:16:11 - mmengine - INFO - Iter(train) [12220/19224] lr: 6.1919e-06 eta: 5:03:47 time: 3.0063 data_time: 0.0101 memory: 13461 loss: 0.1728 2024/07/24 23:16:41 - mmengine - INFO - Iter(train) [12230/19224] lr: 6.1763e-06 eta: 5:03:24 time: 3.0261 data_time: 0.0103 memory: 12224 loss: 0.1933 2024/07/24 23:17:10 - mmengine - INFO - Iter(train) [12240/19224] lr: 6.1607e-06 eta: 5:02:59 time: 2.8720 data_time: 0.0106 memory: 11691 loss: 0.1738 2024/07/24 23:17:38 - mmengine - INFO - Iter(train) [12250/19224] lr: 6.1452e-06 eta: 5:02:34 time: 2.7845 data_time: 0.0106 memory: 11408 loss: 0.1567 2024/07/24 23:18:05 - mmengine - INFO - Iter(train) [12260/19224] lr: 6.1296e-06 eta: 5:02:09 time: 2.6971 data_time: 0.0105 memory: 11299 loss: 0.1667 2024/07/24 23:18:30 - mmengine - INFO - Iter(train) [12270/19224] lr: 6.1141e-06 eta: 5:01:42 time: 2.5524 data_time: 0.0105 memory: 11117 loss: 0.1646 2024/07/24 23:18:55 - mmengine - INFO - Iter(train) [12280/19224] lr: 6.0986e-06 eta: 5:01:16 time: 2.5115 data_time: 0.0106 memory: 11033 loss: 0.1904 2024/07/24 23:19:17 - mmengine - INFO - Iter(train) [12290/19224] lr: 6.0831e-06 eta: 5:00:47 time: 2.1428 data_time: 0.0098 memory: 10760 loss: 0.1564 2024/07/24 23:19:36 - mmengine - INFO - Iter(train) [12300/19224] lr: 6.0676e-06 eta: 5:00:17 time: 1.9196 data_time: 0.0096 memory: 10225 loss: 0.1788 2024/07/24 23:19:50 - mmengine - INFO - Iter(train) [12310/19224] lr: 6.0521e-06 eta: 4:59:45 time: 1.4410 data_time: 0.0100 memory: 9974 loss: 0.1751 2024/07/24 23:20:21 - mmengine - INFO - Iter(train) [12320/19224] lr: 6.0366e-06 eta: 4:59:21 time: 3.0142 data_time: 0.0101 memory: 14323 loss: 0.1822 2024/07/24 23:20:52 - mmengine - INFO - Iter(train) [12330/19224] lr: 6.0212e-06 eta: 4:58:58 time: 3.1219 data_time: 0.0107 memory: 12337 loss: 0.1612 2024/07/24 23:21:21 - mmengine - INFO - Iter(train) [12340/19224] lr: 6.0057e-06 eta: 4:58:34 time: 2.9477 data_time: 0.0168 memory: 11887 loss: 0.1653 2024/07/24 23:21:50 - mmengine - INFO - Iter(train) [12350/19224] lr: 5.9903e-06 eta: 4:58:09 time: 2.8809 data_time: 0.0112 memory: 11573 loss: 0.1686 2024/07/24 23:22:18 - mmengine - INFO - Iter(train) [12360/19224] lr: 5.9748e-06 eta: 4:57:44 time: 2.7750 data_time: 0.0106 memory: 11355 loss: 0.1757 2024/07/24 23:22:44 - mmengine - INFO - Iter(train) [12370/19224] lr: 5.9594e-06 eta: 4:57:19 time: 2.6347 data_time: 0.0108 memory: 11444 loss: 0.1789 2024/07/24 23:23:09 - mmengine - INFO - Iter(train) [12380/19224] lr: 5.9440e-06 eta: 4:56:52 time: 2.4417 data_time: 0.0107 memory: 11041 loss: 0.2075 2024/07/24 23:23:29 - mmengine - INFO - Iter(train) [12390/19224] lr: 5.9286e-06 eta: 4:56:23 time: 2.0792 data_time: 0.0104 memory: 10585 loss: 0.1772 2024/07/24 23:23:48 - mmengine - INFO - Iter(train) [12400/19224] lr: 5.9133e-06 eta: 4:55:52 time: 1.8180 data_time: 0.0096 memory: 10099 loss: 0.2033 2024/07/24 23:24:02 - mmengine - INFO - Iter(train) [12410/19224] lr: 5.8979e-06 eta: 4:55:20 time: 1.4730 data_time: 0.0093 memory: 9686 loss: 0.1703 2024/07/24 23:24:32 - mmengine - INFO - Iter(train) [12420/19224] lr: 5.8825e-06 eta: 4:54:56 time: 2.9570 data_time: 0.0099 memory: 13344 loss: 0.1567 2024/07/24 23:25:02 - mmengine - INFO - Iter(train) [12430/19224] lr: 5.8672e-06 eta: 4:54:32 time: 3.0149 data_time: 0.0112 memory: 12130 loss: 0.1646 2024/07/24 23:25:32 - mmengine - INFO - Iter(train) [12440/19224] lr: 5.8518e-06 eta: 4:54:08 time: 2.9834 data_time: 0.0107 memory: 12057 loss: 0.1633 2024/07/24 23:25:59 - mmengine - INFO - Iter(train) [12450/19224] lr: 5.8365e-06 eta: 4:53:43 time: 2.7573 data_time: 0.0108 memory: 11663 loss: 0.1701 2024/07/24 23:26:27 - mmengine - INFO - Iter(train) [12460/19224] lr: 5.8212e-06 eta: 4:53:18 time: 2.7290 data_time: 0.0112 memory: 11279 loss: 0.1562 2024/07/24 23:26:53 - mmengine - INFO - Iter(train) [12470/19224] lr: 5.8059e-06 eta: 4:52:52 time: 2.5793 data_time: 0.0109 memory: 11165 loss: 0.1766 2024/07/24 23:27:17 - mmengine - INFO - Iter(train) [12480/19224] lr: 5.7906e-06 eta: 4:52:25 time: 2.4789 data_time: 0.0109 memory: 11043 loss: 0.1980 2024/07/24 23:27:41 - mmengine - INFO - Iter(train) [12490/19224] lr: 5.7753e-06 eta: 4:51:58 time: 2.3208 data_time: 0.0110 memory: 10708 loss: 0.1980 2024/07/24 23:28:00 - mmengine - INFO - Iter(train) [12500/19224] lr: 5.7601e-06 eta: 4:51:28 time: 1.9735 data_time: 0.0098 memory: 10437 loss: 0.1789 2024/07/24 23:28:17 - mmengine - INFO - Iter(train) [12510/19224] lr: 5.7448e-06 eta: 4:50:57 time: 1.6434 data_time: 0.0098 memory: 10102 loss: 0.1897 2024/07/24 23:28:47 - mmengine - INFO - Iter(train) [12520/19224] lr: 5.7296e-06 eta: 4:50:33 time: 3.0039 data_time: 0.0099 memory: 14357 loss: 0.1701 2024/07/24 23:29:18 - mmengine - INFO - Iter(train) [12530/19224] lr: 5.7144e-06 eta: 4:50:10 time: 3.1043 data_time: 0.0106 memory: 12322 loss: 0.1570 2024/07/24 23:29:48 - mmengine - INFO - Iter(train) [12540/19224] lr: 5.6991e-06 eta: 4:49:46 time: 3.0031 data_time: 0.0107 memory: 12005 loss: 0.1546 2024/07/24 23:30:19 - mmengine - INFO - Iter(train) [12550/19224] lr: 5.6839e-06 eta: 4:49:23 time: 3.0778 data_time: 0.0107 memory: 11559 loss: 0.1426 2024/07/24 23:30:47 - mmengine - INFO - Iter(train) [12560/19224] lr: 5.6688e-06 eta: 4:48:58 time: 2.7994 data_time: 0.0106 memory: 11411 loss: 0.1560 2024/07/24 23:31:13 - mmengine - INFO - Iter(train) [12570/19224] lr: 5.6536e-06 eta: 4:48:32 time: 2.6863 data_time: 0.0104 memory: 11297 loss: 0.1587 2024/07/24 23:31:39 - mmengine - INFO - Iter(train) [12580/19224] lr: 5.6384e-06 eta: 4:48:06 time: 2.5319 data_time: 0.0097 memory: 11143 loss: 0.1641 2024/07/24 23:32:02 - mmengine - INFO - Iter(train) [12590/19224] lr: 5.6233e-06 eta: 4:47:38 time: 2.3646 data_time: 0.0099 memory: 10909 loss: 0.1580 2024/07/24 23:32:23 - mmengine - INFO - Iter(train) [12600/19224] lr: 5.6081e-06 eta: 4:47:10 time: 2.0793 data_time: 0.0103 memory: 10584 loss: 0.1559 2024/07/24 23:32:40 - mmengine - INFO - Iter(train) [12610/19224] lr: 5.5930e-06 eta: 4:46:39 time: 1.6698 data_time: 0.0098 memory: 10123 loss: 0.1773 2024/07/24 23:33:14 - mmengine - INFO - Iter(train) [12620/19224] lr: 5.5779e-06 eta: 4:46:17 time: 3.4116 data_time: 0.0107 memory: 18250 loss: 0.2541 2024/07/24 23:33:45 - mmengine - INFO - Iter(train) [12630/19224] lr: 5.5628e-06 eta: 4:45:53 time: 3.0630 data_time: 0.0109 memory: 12282 loss: 0.1491 2024/07/24 23:34:15 - mmengine - INFO - Iter(train) [12640/19224] lr: 5.5477e-06 eta: 4:45:29 time: 3.0074 data_time: 0.0112 memory: 11891 loss: 0.1769 2024/07/24 23:34:44 - mmengine - INFO - Iter(train) [12650/19224] lr: 5.5326e-06 eta: 4:45:05 time: 2.9451 data_time: 0.0137 memory: 11825 loss: 0.1624 2024/07/24 23:35:12 - mmengine - INFO - Iter(train) [12660/19224] lr: 5.5175e-06 eta: 4:44:40 time: 2.7598 data_time: 0.0115 memory: 11419 loss: 0.1662 2024/07/24 23:35:39 - mmengine - INFO - Iter(train) [12670/19224] lr: 5.5025e-06 eta: 4:44:15 time: 2.7063 data_time: 0.0118 memory: 11317 loss: 0.1694 2024/07/24 23:36:04 - mmengine - INFO - Iter(train) [12680/19224] lr: 5.4874e-06 eta: 4:43:48 time: 2.5579 data_time: 0.0113 memory: 11145 loss: 0.1762 2024/07/24 23:36:28 - mmengine - INFO - Iter(train) [12690/19224] lr: 5.4724e-06 eta: 4:43:21 time: 2.3800 data_time: 0.0113 memory: 10925 loss: 0.2020 2024/07/24 23:36:49 - mmengine - INFO - Iter(train) [12700/19224] lr: 5.4574e-06 eta: 4:42:52 time: 2.0449 data_time: 0.0101 memory: 10521 loss: 0.1800 2024/07/24 23:37:06 - mmengine - INFO - Iter(train) [12710/19224] lr: 5.4424e-06 eta: 4:42:22 time: 1.7415 data_time: 0.0099 memory: 10203 loss: 0.1741 2024/07/24 23:37:38 - mmengine - INFO - Iter(train) [12720/19224] lr: 5.4274e-06 eta: 4:41:59 time: 3.2318 data_time: 0.0111 memory: 15899 loss: 0.2003 2024/07/24 23:38:09 - mmengine - INFO - Iter(train) [12730/19224] lr: 5.4124e-06 eta: 4:41:35 time: 3.0341 data_time: 0.0115 memory: 12132 loss: 0.1681 2024/07/24 23:38:39 - mmengine - INFO - Iter(train) [12740/19224] lr: 5.3975e-06 eta: 4:41:11 time: 2.9747 data_time: 0.0116 memory: 11901 loss: 0.1589 2024/07/24 23:39:07 - mmengine - INFO - Iter(train) [12750/19224] lr: 5.3825e-06 eta: 4:40:46 time: 2.8096 data_time: 0.0114 memory: 11477 loss: 0.1531 2024/07/24 23:39:34 - mmengine - INFO - Iter(train) [12760/19224] lr: 5.3676e-06 eta: 4:40:21 time: 2.7182 data_time: 0.0111 memory: 11333 loss: 0.1822 2024/07/24 23:40:00 - mmengine - INFO - Iter(train) [12770/19224] lr: 5.3527e-06 eta: 4:39:55 time: 2.6116 data_time: 0.0114 memory: 11179 loss: 0.1551 2024/07/24 23:40:24 - mmengine - INFO - Iter(train) [12780/19224] lr: 5.3377e-06 eta: 4:39:28 time: 2.4456 data_time: 0.0109 memory: 11084 loss: 0.1790 2024/07/24 23:40:46 - mmengine - INFO - Iter(train) [12790/19224] lr: 5.3228e-06 eta: 4:39:00 time: 2.1761 data_time: 0.0114 memory: 10781 loss: 0.1863 2024/07/24 23:41:06 - mmengine - INFO - Iter(train) [12800/19224] lr: 5.3080e-06 eta: 4:38:31 time: 1.9881 data_time: 0.0098 memory: 10402 loss: 0.1760 2024/07/24 23:41:21 - mmengine - INFO - Iter(train) [12810/19224] lr: 5.2931e-06 eta: 4:37:59 time: 1.4846 data_time: 0.0097 memory: 9857 loss: 0.2452 2024/07/24 23:41:54 - mmengine - INFO - Iter(train) [12820/19224] lr: 5.2782e-06 eta: 4:37:37 time: 3.3388 data_time: 0.0107 memory: 16719 loss: 0.1979 2024/07/24 23:42:25 - mmengine - INFO - Iter(train) [12830/19224] lr: 5.2634e-06 eta: 4:37:13 time: 3.1263 data_time: 0.0111 memory: 12318 loss: 0.1611 2024/07/24 23:42:55 - mmengine - INFO - Iter(train) [12840/19224] lr: 5.2486e-06 eta: 4:36:49 time: 2.9560 data_time: 0.0111 memory: 11901 loss: 0.1487 2024/07/24 23:43:23 - mmengine - INFO - Iter(train) [12850/19224] lr: 5.2337e-06 eta: 4:36:24 time: 2.8341 data_time: 0.0109 memory: 11638 loss: 0.1714 2024/07/24 23:43:52 - mmengine - INFO - Iter(train) [12860/19224] lr: 5.2189e-06 eta: 4:35:59 time: 2.8331 data_time: 0.0111 memory: 11444 loss: 0.1764 2024/07/24 23:44:19 - mmengine - INFO - Iter(train) [12870/19224] lr: 5.2042e-06 eta: 4:35:34 time: 2.6980 data_time: 0.0113 memory: 11281 loss: 0.1661 2024/07/24 23:44:44 - mmengine - INFO - Iter(train) [12880/19224] lr: 5.1894e-06 eta: 4:35:08 time: 2.5664 data_time: 0.0108 memory: 11104 loss: 0.1771 2024/07/24 23:45:07 - mmengine - INFO - Iter(train) [12890/19224] lr: 5.1746e-06 eta: 4:34:40 time: 2.2858 data_time: 0.0115 memory: 10836 loss: 0.1878 2024/07/24 23:45:27 - mmengine - INFO - Iter(train) [12900/19224] lr: 5.1599e-06 eta: 4:34:11 time: 1.9730 data_time: 0.0102 memory: 10404 loss: 0.1813 2024/07/24 23:45:43 - mmengine - INFO - Iter(train) [12910/19224] lr: 5.1451e-06 eta: 4:33:40 time: 1.5967 data_time: 0.0098 memory: 9959 loss: 0.1911 2024/07/24 23:46:16 - mmengine - INFO - Iter(train) [12920/19224] lr: 5.1304e-06 eta: 4:33:17 time: 3.2811 data_time: 0.0107 memory: 16719 loss: 0.1745 2024/07/24 23:46:48 - mmengine - INFO - Iter(train) [12930/19224] lr: 5.1157e-06 eta: 4:32:54 time: 3.2029 data_time: 0.0111 memory: 12196 loss: 0.1569 2024/07/24 23:47:18 - mmengine - INFO - Iter(train) [12940/19224] lr: 5.1010e-06 eta: 4:32:30 time: 2.9904 data_time: 0.0111 memory: 11898 loss: 0.1504 2024/07/24 23:47:47 - mmengine - INFO - Iter(train) [12950/19224] lr: 5.0863e-06 eta: 4:32:06 time: 2.9527 data_time: 0.0110 memory: 11692 loss: 0.1513 2024/07/24 23:48:15 - mmengine - INFO - Iter(train) [12960/19224] lr: 5.0717e-06 eta: 4:31:41 time: 2.8049 data_time: 0.0113 memory: 11524 loss: 0.1555 2024/07/24 23:48:42 - mmengine - INFO - Iter(train) [12970/19224] lr: 5.0570e-06 eta: 4:31:15 time: 2.6918 data_time: 0.0116 memory: 11278 loss: 0.1965 2024/07/24 23:49:07 - mmengine - INFO - Iter(train) [12980/19224] lr: 5.0424e-06 eta: 4:30:49 time: 2.4877 data_time: 0.0110 memory: 11126 loss: 0.1614 2024/07/24 23:49:30 - mmengine - INFO - Iter(train) [12990/19224] lr: 5.0277e-06 eta: 4:30:21 time: 2.2562 data_time: 0.0110 memory: 10840 loss: 0.1739 2024/07/24 23:49:49 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/24 23:49:49 - mmengine - INFO - Iter(train) [13000/19224] lr: 5.0131e-06 eta: 4:29:52 time: 1.9335 data_time: 0.0099 memory: 10437 loss: 0.1927 2024/07/24 23:49:49 - mmengine - INFO - Saving checkpoint at 13000 iterations 2024/07/24 23:50:05 - mmengine - INFO - Iter(train) [13010/19224] lr: 4.9985e-06 eta: 4:29:21 time: 1.5719 data_time: 0.1963 memory: 9748 loss: 0.1765 2024/07/24 23:50:36 - mmengine - INFO - Iter(train) [13020/19224] lr: 4.9840e-06 eta: 4:28:57 time: 3.1734 data_time: 0.0117 memory: 15923 loss: 0.1759 2024/07/24 23:51:08 - mmengine - INFO - Iter(train) [13030/19224] lr: 4.9694e-06 eta: 4:28:34 time: 3.2041 data_time: 0.0113 memory: 12723 loss: 0.1751 2024/07/24 23:51:39 - mmengine - INFO - Iter(train) [13040/19224] lr: 4.9548e-06 eta: 4:28:10 time: 3.0405 data_time: 0.0110 memory: 12003 loss: 0.1614 2024/07/24 23:52:09 - mmengine - INFO - Iter(train) [13050/19224] lr: 4.9403e-06 eta: 4:27:46 time: 3.0264 data_time: 0.0114 memory: 11845 loss: 0.1707 2024/07/24 23:52:37 - mmengine - INFO - Iter(train) [13060/19224] lr: 4.9258e-06 eta: 4:27:21 time: 2.7598 data_time: 0.0110 memory: 11477 loss: 0.1676 2024/07/24 23:53:02 - mmengine - INFO - Iter(train) [13070/19224] lr: 4.9113e-06 eta: 4:26:55 time: 2.5705 data_time: 0.0108 memory: 11263 loss: 0.1637 2024/07/24 23:53:26 - mmengine - INFO - Iter(train) [13080/19224] lr: 4.8968e-06 eta: 4:26:28 time: 2.4072 data_time: 0.0104 memory: 11001 loss: 0.1864 2024/07/24 23:53:48 - mmengine - INFO - Iter(train) [13090/19224] lr: 4.8823e-06 eta: 4:26:00 time: 2.1235 data_time: 0.0102 memory: 10618 loss: 0.2165 2024/07/24 23:54:06 - mmengine - INFO - Iter(train) [13100/19224] lr: 4.8678e-06 eta: 4:25:30 time: 1.8267 data_time: 0.0104 memory: 10241 loss: 0.1671 2024/07/24 23:54:20 - mmengine - INFO - Iter(train) [13110/19224] lr: 4.8534e-06 eta: 4:24:58 time: 1.4112 data_time: 0.0090 memory: 9953 loss: 0.1697 2024/07/24 23:54:47 - mmengine - INFO - Iter(train) [13120/19224] lr: 4.8389e-06 eta: 4:24:33 time: 2.7020 data_time: 0.0099 memory: 12953 loss: 0.1993 2024/07/24 23:55:17 - mmengine - INFO - Iter(train) [13130/19224] lr: 4.8245e-06 eta: 4:24:09 time: 2.9935 data_time: 0.0117 memory: 12143 loss: 0.1537 2024/07/24 23:55:46 - mmengine - INFO - Iter(train) [13140/19224] lr: 4.8101e-06 eta: 4:23:44 time: 2.8771 data_time: 0.0111 memory: 11729 loss: 0.1648 2024/07/24 23:56:13 - mmengine - INFO - Iter(train) [13150/19224] lr: 4.7957e-06 eta: 4:23:18 time: 2.6973 data_time: 0.0107 memory: 11386 loss: 0.1699 2024/07/24 23:56:40 - mmengine - INFO - Iter(train) [13160/19224] lr: 4.7813e-06 eta: 4:22:53 time: 2.7420 data_time: 0.0106 memory: 11236 loss: 0.1784 2024/07/24 23:57:06 - mmengine - INFO - Iter(train) [13170/19224] lr: 4.7670e-06 eta: 4:22:27 time: 2.5671 data_time: 0.0117 memory: 11143 loss: 0.1746 2024/07/24 23:57:31 - mmengine - INFO - Iter(train) [13180/19224] lr: 4.7526e-06 eta: 4:22:00 time: 2.4728 data_time: 0.0108 memory: 11065 loss: 0.1904 2024/07/24 23:57:55 - mmengine - INFO - Iter(train) [13190/19224] lr: 4.7383e-06 eta: 4:21:33 time: 2.4109 data_time: 0.0104 memory: 10846 loss: 0.1783 2024/07/24 23:58:17 - mmengine - INFO - Iter(train) [13200/19224] lr: 4.7240e-06 eta: 4:21:06 time: 2.1816 data_time: 0.0101 memory: 10646 loss: 0.1638 2024/07/24 23:58:34 - mmengine - INFO - Iter(train) [13210/19224] lr: 4.7097e-06 eta: 4:20:36 time: 1.7699 data_time: 0.0094 memory: 10050 loss: 0.1829 2024/07/24 23:59:04 - mmengine - INFO - Iter(train) [13220/19224] lr: 4.6954e-06 eta: 4:20:12 time: 2.9906 data_time: 0.0099 memory: 15347 loss: 0.1759 2024/07/24 23:59:34 - mmengine - INFO - Iter(train) [13230/19224] lr: 4.6811e-06 eta: 4:19:47 time: 3.0167 data_time: 0.0115 memory: 12109 loss: 0.1564 2024/07/25 00:00:06 - mmengine - INFO - Iter(train) [13240/19224] lr: 4.6668e-06 eta: 4:19:24 time: 3.1329 data_time: 0.0109 memory: 11796 loss: 0.1568 2024/07/25 00:00:33 - mmengine - INFO - Iter(train) [13250/19224] lr: 4.6526e-06 eta: 4:18:58 time: 2.7273 data_time: 0.0109 memory: 11439 loss: 0.1676 2024/07/25 00:00:59 - mmengine - INFO - Iter(train) [13260/19224] lr: 4.6384e-06 eta: 4:18:32 time: 2.6226 data_time: 0.0109 memory: 11229 loss: 0.1711 2024/07/25 00:01:24 - mmengine - INFO - Iter(train) [13270/19224] lr: 4.6242e-06 eta: 4:18:06 time: 2.5050 data_time: 0.0109 memory: 11199 loss: 0.1872 2024/07/25 00:01:49 - mmengine - INFO - Iter(train) [13280/19224] lr: 4.6100e-06 eta: 4:17:39 time: 2.4680 data_time: 0.0113 memory: 11009 loss: 0.1623 2024/07/25 00:02:13 - mmengine - INFO - Iter(train) [13290/19224] lr: 4.5958e-06 eta: 4:17:12 time: 2.3927 data_time: 0.0110 memory: 10888 loss: 0.1830 2024/07/25 00:02:35 - mmengine - INFO - Iter(train) [13300/19224] lr: 4.5816e-06 eta: 4:16:45 time: 2.1721 data_time: 0.0099 memory: 10592 loss: 0.1820 2024/07/25 00:02:53 - mmengine - INFO - Iter(train) [13310/19224] lr: 4.5675e-06 eta: 4:16:15 time: 1.8385 data_time: 0.0098 memory: 10188 loss: 0.2026 2024/07/25 00:03:26 - mmengine - INFO - Iter(train) [13320/19224] lr: 4.5533e-06 eta: 4:15:52 time: 3.3212 data_time: 0.0105 memory: 17914 loss: 0.1807 2024/07/25 00:03:56 - mmengine - INFO - Iter(train) [13330/19224] lr: 4.5392e-06 eta: 4:15:28 time: 2.9468 data_time: 0.0120 memory: 12010 loss: 0.1491 2024/07/25 00:04:24 - mmengine - INFO - Iter(train) [13340/19224] lr: 4.5251e-06 eta: 4:15:03 time: 2.8192 data_time: 0.0131 memory: 11670 loss: 0.1539 2024/07/25 00:04:50 - mmengine - INFO - Iter(train) [13350/19224] lr: 4.5110e-06 eta: 4:14:37 time: 2.6470 data_time: 0.0107 memory: 11360 loss: 0.1781 2024/07/25 00:05:16 - mmengine - INFO - Iter(train) [13360/19224] lr: 4.4969e-06 eta: 4:14:11 time: 2.5574 data_time: 0.0124 memory: 11210 loss: 0.1796 2024/07/25 00:05:41 - mmengine - INFO - Iter(train) [13370/19224] lr: 4.4829e-06 eta: 4:13:44 time: 2.5082 data_time: 0.0113 memory: 11028 loss: 0.1930 2024/07/25 00:06:05 - mmengine - INFO - Iter(train) [13380/19224] lr: 4.4688e-06 eta: 4:13:18 time: 2.3932 data_time: 0.0112 memory: 10906 loss: 0.2055 2024/07/25 00:06:25 - mmengine - INFO - Iter(train) [13390/19224] lr: 4.4548e-06 eta: 4:12:49 time: 1.9927 data_time: 0.0110 memory: 10494 loss: 0.1832 2024/07/25 00:06:42 - mmengine - INFO - Iter(train) [13400/19224] lr: 4.4408e-06 eta: 4:12:19 time: 1.7527 data_time: 0.0099 memory: 10122 loss: 0.1919 2024/07/25 00:06:57 - mmengine - INFO - Iter(train) [13410/19224] lr: 4.4268e-06 eta: 4:11:48 time: 1.4634 data_time: 0.0092 memory: 9698 loss: 0.1897 2024/07/25 00:07:29 - mmengine - INFO - Iter(train) [13420/19224] lr: 4.4128e-06 eta: 4:11:25 time: 3.2086 data_time: 0.0110 memory: 15849 loss: 0.1516 2024/07/25 00:08:00 - mmengine - INFO - Iter(train) [13430/19224] lr: 4.3989e-06 eta: 4:11:01 time: 3.1374 data_time: 0.0112 memory: 12210 loss: 0.1622 2024/07/25 00:08:29 - mmengine - INFO - Iter(train) [13440/19224] lr: 4.3849e-06 eta: 4:10:36 time: 2.8795 data_time: 0.0112 memory: 11825 loss: 0.1808 2024/07/25 00:08:57 - mmengine - INFO - Iter(train) [13450/19224] lr: 4.3710e-06 eta: 4:10:11 time: 2.8256 data_time: 0.0114 memory: 11413 loss: 0.1629 2024/07/25 00:09:24 - mmengine - INFO - Iter(train) [13460/19224] lr: 4.3571e-06 eta: 4:09:46 time: 2.6699 data_time: 0.0106 memory: 11441 loss: 0.1930 2024/07/25 00:09:50 - mmengine - INFO - Iter(train) [13470/19224] lr: 4.3432e-06 eta: 4:09:20 time: 2.5805 data_time: 0.0111 memory: 11182 loss: 0.1854 2024/07/25 00:10:14 - mmengine - INFO - Iter(train) [13480/19224] lr: 4.3293e-06 eta: 4:08:53 time: 2.3780 data_time: 0.0107 memory: 10952 loss: 0.2401 2024/07/25 00:10:36 - mmengine - INFO - Iter(train) [13490/19224] lr: 4.3154e-06 eta: 4:08:25 time: 2.1939 data_time: 0.0103 memory: 10747 loss: 0.2998 2024/07/25 00:10:55 - mmengine - INFO - Iter(train) [13500/19224] lr: 4.3016e-06 eta: 4:07:56 time: 1.8924 data_time: 0.0104 memory: 10262 loss: 0.1617 2024/07/25 00:11:10 - mmengine - INFO - Iter(train) [13510/19224] lr: 4.2877e-06 eta: 4:07:25 time: 1.5133 data_time: 0.0092 memory: 9836 loss: 0.2170 2024/07/25 00:11:39 - mmengine - INFO - Iter(train) [13520/19224] lr: 4.2739e-06 eta: 4:07:01 time: 2.9689 data_time: 0.0109 memory: 17245 loss: 0.1654 2024/07/25 00:12:10 - mmengine - INFO - Iter(train) [13530/19224] lr: 4.2601e-06 eta: 4:06:37 time: 3.0771 data_time: 0.0112 memory: 12289 loss: 0.1702 2024/07/25 00:12:40 - mmengine - INFO - Iter(train) [13540/19224] lr: 4.2463e-06 eta: 4:06:12 time: 2.9400 data_time: 0.0123 memory: 11823 loss: 0.1670 2024/07/25 00:13:07 - mmengine - INFO - Iter(train) [13550/19224] lr: 4.2325e-06 eta: 4:05:47 time: 2.7501 data_time: 0.0112 memory: 11415 loss: 0.1667 2024/07/25 00:13:33 - mmengine - INFO - Iter(train) [13560/19224] lr: 4.2188e-06 eta: 4:05:21 time: 2.5850 data_time: 0.0114 memory: 11282 loss: 0.1883 2024/07/25 00:13:59 - mmengine - INFO - Iter(train) [13570/19224] lr: 4.2050e-06 eta: 4:04:55 time: 2.5572 data_time: 0.0111 memory: 11242 loss: 0.2063 2024/07/25 00:14:22 - mmengine - INFO - Iter(train) [13580/19224] lr: 4.1913e-06 eta: 4:04:28 time: 2.3328 data_time: 0.0110 memory: 11068 loss: 0.1726 2024/07/25 00:14:42 - mmengine - INFO - Iter(train) [13590/19224] lr: 4.1776e-06 eta: 4:04:00 time: 2.0594 data_time: 0.0109 memory: 10607 loss: 0.1526 2024/07/25 00:15:01 - mmengine - INFO - Iter(train) [13600/19224] lr: 4.1639e-06 eta: 4:03:30 time: 1.8204 data_time: 0.0095 memory: 10174 loss: 0.1658 2024/07/25 00:15:14 - mmengine - INFO - Iter(train) [13610/19224] lr: 4.1503e-06 eta: 4:02:59 time: 1.2956 data_time: 0.0098 memory: 9721 loss: 0.2039 2024/07/25 00:15:43 - mmengine - INFO - Iter(train) [13620/19224] lr: 4.1366e-06 eta: 4:02:34 time: 2.9417 data_time: 0.0103 memory: 13688 loss: 0.1986 2024/07/25 00:16:14 - mmengine - INFO - Iter(train) [13630/19224] lr: 4.1230e-06 eta: 4:02:10 time: 3.0742 data_time: 0.0109 memory: 12511 loss: 0.1723 2024/07/25 00:16:43 - mmengine - INFO - Iter(train) [13640/19224] lr: 4.1093e-06 eta: 4:01:46 time: 2.9078 data_time: 0.0118 memory: 11905 loss: 0.1535 2024/07/25 00:17:11 - mmengine - INFO - Iter(train) [13650/19224] lr: 4.0957e-06 eta: 4:01:21 time: 2.8177 data_time: 0.0111 memory: 11586 loss: 0.1924 2024/07/25 00:17:39 - mmengine - INFO - Iter(train) [13660/19224] lr: 4.0821e-06 eta: 4:00:55 time: 2.7705 data_time: 0.0109 memory: 11550 loss: 0.1679 2024/07/25 00:18:05 - mmengine - INFO - Iter(train) [13670/19224] lr: 4.0686e-06 eta: 4:00:30 time: 2.6614 data_time: 0.0116 memory: 11368 loss: 0.2073 2024/07/25 00:18:30 - mmengine - INFO - Iter(train) [13680/19224] lr: 4.0550e-06 eta: 4:00:03 time: 2.5001 data_time: 0.0107 memory: 11138 loss: 0.1674 2024/07/25 00:18:54 - mmengine - INFO - Iter(train) [13690/19224] lr: 4.0415e-06 eta: 3:59:36 time: 2.3998 data_time: 0.0112 memory: 10964 loss: 0.1710 2024/07/25 00:19:14 - mmengine - INFO - Iter(train) [13700/19224] lr: 4.0280e-06 eta: 3:59:08 time: 2.0091 data_time: 0.0107 memory: 10470 loss: 0.1799 2024/07/25 00:19:30 - mmengine - INFO - Iter(train) [13710/19224] lr: 4.0145e-06 eta: 3:58:38 time: 1.5825 data_time: 0.0098 memory: 9907 loss: 0.1645 2024/07/25 00:20:01 - mmengine - INFO - Iter(train) [13720/19224] lr: 4.0010e-06 eta: 3:58:14 time: 3.0324 data_time: 0.0102 memory: 13513 loss: 0.1974 2024/07/25 00:20:32 - mmengine - INFO - Iter(train) [13730/19224] lr: 3.9875e-06 eta: 3:57:50 time: 3.0992 data_time: 0.0114 memory: 12194 loss: 0.1628 2024/07/25 00:21:01 - mmengine - INFO - Iter(train) [13740/19224] lr: 3.9740e-06 eta: 3:57:25 time: 2.9700 data_time: 0.0116 memory: 11839 loss: 0.1631 2024/07/25 00:21:29 - mmengine - INFO - Iter(train) [13750/19224] lr: 3.9606e-06 eta: 3:57:00 time: 2.7567 data_time: 0.0111 memory: 11484 loss: 0.1625 2024/07/25 00:21:56 - mmengine - INFO - Iter(train) [13760/19224] lr: 3.9472e-06 eta: 3:56:34 time: 2.6705 data_time: 0.0110 memory: 11291 loss: 0.1870 2024/07/25 00:22:22 - mmengine - INFO - Iter(train) [13770/19224] lr: 3.9338e-06 eta: 3:56:08 time: 2.6186 data_time: 0.0111 memory: 11221 loss: 0.1458 2024/07/25 00:22:47 - mmengine - INFO - Iter(train) [13780/19224] lr: 3.9204e-06 eta: 3:55:42 time: 2.4958 data_time: 0.0109 memory: 11013 loss: 0.1582 2024/07/25 00:23:10 - mmengine - INFO - Iter(train) [13790/19224] lr: 3.9070e-06 eta: 3:55:15 time: 2.2999 data_time: 0.0110 memory: 10846 loss: 0.1837 2024/07/25 00:23:29 - mmengine - INFO - Iter(train) [13800/19224] lr: 3.8937e-06 eta: 3:54:46 time: 1.9653 data_time: 0.0103 memory: 10403 loss: 0.3014 2024/07/25 00:23:45 - mmengine - INFO - Iter(train) [13810/19224] lr: 3.8804e-06 eta: 3:54:16 time: 1.5197 data_time: 0.0095 memory: 10059 loss: 0.1972 2024/07/25 00:24:15 - mmengine - INFO - Iter(train) [13820/19224] lr: 3.8670e-06 eta: 3:53:52 time: 3.0343 data_time: 0.0103 memory: 13515 loss: 0.1566 2024/07/25 00:24:46 - mmengine - INFO - Iter(train) [13830/19224] lr: 3.8537e-06 eta: 3:53:28 time: 3.1162 data_time: 0.0113 memory: 12253 loss: 0.1651 2024/07/25 00:25:16 - mmengine - INFO - Iter(train) [13840/19224] lr: 3.8405e-06 eta: 3:53:04 time: 2.9834 data_time: 0.0113 memory: 11736 loss: 0.1853 2024/07/25 00:25:44 - mmengine - INFO - Iter(train) [13850/19224] lr: 3.8272e-06 eta: 3:52:39 time: 2.8287 data_time: 0.0109 memory: 11553 loss: 0.1413 2024/07/25 00:26:12 - mmengine - INFO - Iter(train) [13860/19224] lr: 3.8140e-06 eta: 3:52:13 time: 2.7554 data_time: 0.0113 memory: 11358 loss: 0.1648 2024/07/25 00:26:37 - mmengine - INFO - Iter(train) [13870/19224] lr: 3.8007e-06 eta: 3:51:47 time: 2.5700 data_time: 0.0111 memory: 11229 loss: 0.1758 2024/07/25 00:27:03 - mmengine - INFO - Iter(train) [13880/19224] lr: 3.7875e-06 eta: 3:51:21 time: 2.5925 data_time: 0.0114 memory: 11072 loss: 0.1850 2024/07/25 00:27:28 - mmengine - INFO - Iter(train) [13890/19224] lr: 3.7743e-06 eta: 3:50:55 time: 2.4352 data_time: 0.0109 memory: 10977 loss: 0.1671 2024/07/25 00:27:50 - mmengine - INFO - Iter(train) [13900/19224] lr: 3.7612e-06 eta: 3:50:27 time: 2.2122 data_time: 0.0101 memory: 10790 loss: 0.1836 2024/07/25 00:28:05 - mmengine - INFO - Iter(train) [13910/19224] lr: 3.7480e-06 eta: 3:49:57 time: 1.5267 data_time: 0.0096 memory: 10060 loss: 0.1612 2024/07/25 00:28:36 - mmengine - INFO - Iter(train) [13920/19224] lr: 3.7349e-06 eta: 3:49:33 time: 3.0552 data_time: 0.0100 memory: 16069 loss: 0.2250 2024/07/25 00:29:07 - mmengine - INFO - Iter(train) [13930/19224] lr: 3.7217e-06 eta: 3:49:09 time: 3.1233 data_time: 0.0111 memory: 12351 loss: 0.1540 2024/07/25 00:29:37 - mmengine - INFO - Iter(train) [13940/19224] lr: 3.7086e-06 eta: 3:48:44 time: 2.9923 data_time: 0.0110 memory: 11903 loss: 0.1497 2024/07/25 00:30:07 - mmengine - INFO - Iter(train) [13950/19224] lr: 3.6955e-06 eta: 3:48:20 time: 3.0462 data_time: 0.0112 memory: 11591 loss: 0.1762 2024/07/25 00:30:35 - mmengine - INFO - Iter(train) [13960/19224] lr: 3.6825e-06 eta: 3:47:55 time: 2.7533 data_time: 0.0108 memory: 11362 loss: 0.1657 2024/07/25 00:31:01 - mmengine - INFO - Iter(train) [13970/19224] lr: 3.6694e-06 eta: 3:47:29 time: 2.6029 data_time: 0.0112 memory: 11169 loss: 0.1670 2024/07/25 00:31:25 - mmengine - INFO - Iter(train) [13980/19224] lr: 3.6564e-06 eta: 3:47:02 time: 2.4438 data_time: 0.0109 memory: 10995 loss: 0.1656 2024/07/25 00:31:48 - mmengine - INFO - Iter(train) [13990/19224] lr: 3.6434e-06 eta: 3:46:35 time: 2.2874 data_time: 0.0112 memory: 10769 loss: 0.2246 2024/07/25 00:32:08 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/25 00:32:08 - mmengine - INFO - Iter(train) [14000/19224] lr: 3.6304e-06 eta: 3:46:07 time: 1.9466 data_time: 0.0094 memory: 10454 loss: 0.1768 2024/07/25 00:32:08 - mmengine - INFO - Saving checkpoint at 14000 iterations 2024/07/25 00:32:27 - mmengine - INFO - Iter(train) [14010/19224] lr: 3.6174e-06 eta: 3:45:38 time: 1.8949 data_time: 0.1934 memory: 9973 loss: 0.1738 2024/07/25 00:32:56 - mmengine - INFO - Iter(train) [14020/19224] lr: 3.6044e-06 eta: 3:45:13 time: 2.9501 data_time: 0.0103 memory: 13403 loss: 0.1713 2024/07/25 00:33:26 - mmengine - INFO - Iter(train) [14030/19224] lr: 3.5915e-06 eta: 3:44:49 time: 3.0029 data_time: 0.0318 memory: 12061 loss: 0.1605 2024/07/25 00:33:55 - mmengine - INFO - Iter(train) [14040/19224] lr: 3.5786e-06 eta: 3:44:24 time: 2.8766 data_time: 0.0111 memory: 11670 loss: 0.1690 2024/07/25 00:34:21 - mmengine - INFO - Iter(train) [14050/19224] lr: 3.5657e-06 eta: 3:43:58 time: 2.6582 data_time: 0.0109 memory: 11339 loss: 0.1671 2024/07/25 00:34:48 - mmengine - INFO - Iter(train) [14060/19224] lr: 3.5528e-06 eta: 3:43:32 time: 2.6533 data_time: 0.0108 memory: 11252 loss: 0.1516 2024/07/25 00:35:13 - mmengine - INFO - Iter(train) [14070/19224] lr: 3.5399e-06 eta: 3:43:06 time: 2.5146 data_time: 0.0112 memory: 11185 loss: 0.1729 2024/07/25 00:35:37 - mmengine - INFO - Iter(train) [14080/19224] lr: 3.5271e-06 eta: 3:42:40 time: 2.4281 data_time: 0.0107 memory: 10965 loss: 0.1647 2024/07/25 00:36:00 - mmengine - INFO - Iter(train) [14090/19224] lr: 3.5142e-06 eta: 3:42:12 time: 2.2683 data_time: 0.0105 memory: 10720 loss: 0.1991 2024/07/25 00:36:19 - mmengine - INFO - Iter(train) [14100/19224] lr: 3.5014e-06 eta: 3:41:44 time: 1.8995 data_time: 0.0096 memory: 10349 loss: 0.1782 2024/07/25 00:36:34 - mmengine - INFO - Iter(train) [14110/19224] lr: 3.4886e-06 eta: 3:41:14 time: 1.4875 data_time: 0.0093 memory: 9707 loss: 0.1999 2024/07/25 00:37:06 - mmengine - INFO - Iter(train) [14120/19224] lr: 3.4758e-06 eta: 3:40:50 time: 3.2142 data_time: 0.0103 memory: 16768 loss: 0.1562 2024/07/25 00:37:36 - mmengine - INFO - Iter(train) [14130/19224] lr: 3.4631e-06 eta: 3:40:26 time: 2.9801 data_time: 0.0112 memory: 11841 loss: 0.1549 2024/07/25 00:38:05 - mmengine - INFO - Iter(train) [14140/19224] lr: 3.4504e-06 eta: 3:40:01 time: 2.8830 data_time: 0.0107 memory: 11747 loss: 0.1516 2024/07/25 00:38:33 - mmengine - INFO - Iter(train) [14150/19224] lr: 3.4376e-06 eta: 3:39:36 time: 2.8352 data_time: 0.0110 memory: 11622 loss: 0.1568 2024/07/25 00:39:00 - mmengine - INFO - Iter(train) [14160/19224] lr: 3.4249e-06 eta: 3:39:10 time: 2.6631 data_time: 0.0110 memory: 11312 loss: 0.1533 2024/07/25 00:39:26 - mmengine - INFO - Iter(train) [14170/19224] lr: 3.4122e-06 eta: 3:38:44 time: 2.6373 data_time: 0.0111 memory: 11178 loss: 0.1564 2024/07/25 00:39:51 - mmengine - INFO - Iter(train) [14180/19224] lr: 3.3996e-06 eta: 3:38:18 time: 2.4525 data_time: 0.0108 memory: 11060 loss: 0.1631 2024/07/25 00:40:12 - mmengine - INFO - Iter(train) [14190/19224] lr: 3.3869e-06 eta: 3:37:50 time: 2.1289 data_time: 0.0103 memory: 10687 loss: 0.2070 2024/07/25 00:40:30 - mmengine - INFO - Iter(train) [14200/19224] lr: 3.3743e-06 eta: 3:37:21 time: 1.8154 data_time: 0.0095 memory: 10189 loss: 0.1843 2024/07/25 00:40:45 - mmengine - INFO - Iter(train) [14210/19224] lr: 3.3617e-06 eta: 3:36:51 time: 1.5000 data_time: 0.0096 memory: 9758 loss: 0.1720 2024/07/25 00:41:13 - mmengine - INFO - Iter(train) [14220/19224] lr: 3.3491e-06 eta: 3:36:26 time: 2.8433 data_time: 0.0102 memory: 13497 loss: 0.1524 2024/07/25 00:41:44 - mmengine - INFO - Iter(train) [14230/19224] lr: 3.3365e-06 eta: 3:36:02 time: 3.0097 data_time: 0.0116 memory: 11998 loss: 0.1503 2024/07/25 00:42:12 - mmengine - INFO - Iter(train) [14240/19224] lr: 3.3240e-06 eta: 3:35:37 time: 2.8679 data_time: 0.0115 memory: 11954 loss: 0.1482 2024/07/25 00:42:40 - mmengine - INFO - Iter(train) [14250/19224] lr: 3.3114e-06 eta: 3:35:12 time: 2.7900 data_time: 0.0113 memory: 11720 loss: 0.2056 2024/07/25 00:43:08 - mmengine - INFO - Iter(train) [14260/19224] lr: 3.2989e-06 eta: 3:34:46 time: 2.7545 data_time: 0.0110 memory: 11424 loss: 0.1825 2024/07/25 00:43:34 - mmengine - INFO - Iter(train) [14270/19224] lr: 3.2864e-06 eta: 3:34:20 time: 2.5976 data_time: 0.0116 memory: 11224 loss: 0.1752 2024/07/25 00:43:58 - mmengine - INFO - Iter(train) [14280/19224] lr: 3.2740e-06 eta: 3:33:54 time: 2.4661 data_time: 0.0111 memory: 11153 loss: 0.1969 2024/07/25 00:44:20 - mmengine - INFO - Iter(train) [14290/19224] lr: 3.2615e-06 eta: 3:33:26 time: 2.1916 data_time: 0.0104 memory: 10862 loss: 0.1735 2024/07/25 00:44:40 - mmengine - INFO - Iter(train) [14300/19224] lr: 3.2491e-06 eta: 3:32:58 time: 1.9325 data_time: 0.0099 memory: 10429 loss: 0.1767 2024/07/25 00:44:55 - mmengine - INFO - Iter(train) [14310/19224] lr: 3.2366e-06 eta: 3:32:28 time: 1.5033 data_time: 0.0093 memory: 10004 loss: 0.1929 2024/07/25 00:45:26 - mmengine - INFO - Iter(train) [14320/19224] lr: 3.2242e-06 eta: 3:32:04 time: 3.1773 data_time: 0.0102 memory: 19084 loss: 0.1691 2024/07/25 00:45:56 - mmengine - INFO - Iter(train) [14330/19224] lr: 3.2119e-06 eta: 3:31:40 time: 2.9828 data_time: 0.0115 memory: 12139 loss: 0.1630 2024/07/25 00:46:25 - mmengine - INFO - Iter(train) [14340/19224] lr: 3.1995e-06 eta: 3:31:15 time: 2.8561 data_time: 0.0111 memory: 11718 loss: 0.1640 2024/07/25 00:46:52 - mmengine - INFO - Iter(train) [14350/19224] lr: 3.1872e-06 eta: 3:30:49 time: 2.7172 data_time: 0.0108 memory: 11486 loss: 0.1674 2024/07/25 00:47:19 - mmengine - INFO - Iter(train) [14360/19224] lr: 3.1748e-06 eta: 3:30:24 time: 2.6694 data_time: 0.0110 memory: 11353 loss: 0.1936 2024/07/25 00:47:45 - mmengine - INFO - Iter(train) [14370/19224] lr: 3.1625e-06 eta: 3:29:58 time: 2.6020 data_time: 0.0113 memory: 11275 loss: 0.1822 2024/07/25 00:48:10 - mmengine - INFO - Iter(train) [14380/19224] lr: 3.1503e-06 eta: 3:29:31 time: 2.5098 data_time: 0.0109 memory: 11157 loss: 0.1823 2024/07/25 00:48:34 - mmengine - INFO - Iter(train) [14390/19224] lr: 3.1380e-06 eta: 3:29:05 time: 2.4675 data_time: 0.0108 memory: 11091 loss: 0.1611 2024/07/25 00:48:55 - mmengine - INFO - Iter(train) [14400/19224] lr: 3.1257e-06 eta: 3:28:37 time: 2.0964 data_time: 0.0105 memory: 10785 loss: 0.1938 2024/07/25 00:49:09 - mmengine - INFO - Iter(train) [14410/19224] lr: 3.1135e-06 eta: 3:28:07 time: 1.3475 data_time: 0.0091 memory: 9825 loss: 0.2012 2024/07/25 00:49:40 - mmengine - INFO - Iter(train) [14420/19224] lr: 3.1013e-06 eta: 3:27:43 time: 3.1131 data_time: 0.2822 memory: 19416 loss: 0.1436 2024/07/25 00:50:13 - mmengine - INFO - Iter(train) [14430/19224] lr: 3.0891e-06 eta: 3:27:19 time: 3.2703 data_time: 0.0105 memory: 13209 loss: 0.1638 2024/07/25 00:50:42 - mmengine - INFO - Iter(train) [14440/19224] lr: 3.0770e-06 eta: 3:26:55 time: 2.9255 data_time: 0.0113 memory: 12104 loss: 0.1465 2024/07/25 00:51:10 - mmengine - INFO - Iter(train) [14450/19224] lr: 3.0648e-06 eta: 3:26:29 time: 2.7982 data_time: 0.0105 memory: 11733 loss: 0.1456 2024/07/25 00:51:38 - mmengine - INFO - Iter(train) [14460/19224] lr: 3.0527e-06 eta: 3:26:04 time: 2.7610 data_time: 0.0104 memory: 11481 loss: 0.1554 2024/07/25 00:52:03 - mmengine - INFO - Iter(train) [14470/19224] lr: 3.0406e-06 eta: 3:25:38 time: 2.5907 data_time: 0.0104 memory: 11301 loss: 0.1338 2024/07/25 00:52:30 - mmengine - INFO - Iter(train) [14480/19224] lr: 3.0285e-06 eta: 3:25:12 time: 2.6081 data_time: 0.0106 memory: 11186 loss: 0.1588 2024/07/25 00:52:54 - mmengine - INFO - Iter(train) [14490/19224] lr: 3.0164e-06 eta: 3:24:45 time: 2.4278 data_time: 0.0103 memory: 11022 loss: 0.1705 2024/07/25 00:53:15 - mmengine - INFO - Iter(train) [14500/19224] lr: 3.0044e-06 eta: 3:24:18 time: 2.1367 data_time: 0.0098 memory: 10890 loss: 0.1383 2024/07/25 00:53:33 - mmengine - INFO - Iter(train) [14510/19224] lr: 2.9923e-06 eta: 3:23:50 time: 1.8233 data_time: 0.0092 memory: 10213 loss: 0.1520 2024/07/25 00:53:53 - mmengine - INFO - Iter(train) [14520/19224] lr: 2.9803e-06 eta: 3:23:22 time: 1.9684 data_time: 0.0089 memory: 14140 loss: 0.1666 2024/07/25 00:54:24 - mmengine - INFO - Iter(train) [14530/19224] lr: 2.9684e-06 eta: 3:22:57 time: 3.0899 data_time: 0.0103 memory: 12272 loss: 0.1787 2024/07/25 00:54:53 - mmengine - INFO - Iter(train) [14540/19224] lr: 2.9564e-06 eta: 3:22:32 time: 2.8924 data_time: 0.0104 memory: 11761 loss: 0.1475 2024/07/25 00:55:22 - mmengine - INFO - Iter(train) [14550/19224] lr: 2.9444e-06 eta: 3:22:07 time: 2.8570 data_time: 0.0104 memory: 11781 loss: 0.1609 2024/07/25 00:55:50 - mmengine - INFO - Iter(train) [14560/19224] lr: 2.9325e-06 eta: 3:21:42 time: 2.8190 data_time: 0.0104 memory: 11484 loss: 0.1556 2024/07/25 00:56:17 - mmengine - INFO - Iter(train) [14570/19224] lr: 2.9206e-06 eta: 3:21:16 time: 2.7362 data_time: 0.0105 memory: 11267 loss: 0.1744 2024/07/25 00:56:43 - mmengine - INFO - Iter(train) [14580/19224] lr: 2.9087e-06 eta: 3:20:51 time: 2.6137 data_time: 0.0103 memory: 11127 loss: 0.1749 2024/07/25 00:57:07 - mmengine - INFO - Iter(train) [14590/19224] lr: 2.8968e-06 eta: 3:20:24 time: 2.3585 data_time: 0.0103 memory: 10931 loss: 0.1526 2024/07/25 00:57:26 - mmengine - INFO - Iter(train) [14600/19224] lr: 2.8850e-06 eta: 3:19:56 time: 1.9026 data_time: 0.0096 memory: 10358 loss: 0.1732 2024/07/25 00:57:44 - mmengine - INFO - Iter(train) [14610/19224] lr: 2.8732e-06 eta: 3:19:27 time: 1.7910 data_time: 0.0091 memory: 10063 loss: 0.1488 2024/07/25 00:58:01 - mmengine - INFO - Iter(train) [14620/19224] lr: 2.8614e-06 eta: 3:18:58 time: 1.6842 data_time: 0.0083 memory: 13746 loss: 0.1527 2024/07/25 00:58:35 - mmengine - INFO - Iter(train) [14630/19224] lr: 2.8496e-06 eta: 3:18:35 time: 3.4383 data_time: 0.0109 memory: 13218 loss: 0.1565 2024/07/25 00:59:06 - mmengine - INFO - Iter(train) [14640/19224] lr: 2.8378e-06 eta: 3:18:11 time: 3.0956 data_time: 0.0106 memory: 12375 loss: 0.1442 2024/07/25 00:59:34 - mmengine - INFO - Iter(train) [14650/19224] lr: 2.8261e-06 eta: 3:17:46 time: 2.8387 data_time: 0.0111 memory: 11593 loss: 0.1533 2024/07/25 01:00:02 - mmengine - INFO - Iter(train) [14660/19224] lr: 2.8143e-06 eta: 3:17:20 time: 2.7505 data_time: 0.0114 memory: 11502 loss: 0.1550 2024/07/25 01:00:30 - mmengine - INFO - Iter(train) [14670/19224] lr: 2.8026e-06 eta: 3:16:55 time: 2.8671 data_time: 0.0105 memory: 11259 loss: 0.1693 2024/07/25 01:00:56 - mmengine - INFO - Iter(train) [14680/19224] lr: 2.7909e-06 eta: 3:16:29 time: 2.5099 data_time: 0.0102 memory: 11154 loss: 0.1722 2024/07/25 01:01:20 - mmengine - INFO - Iter(train) [14690/19224] lr: 2.7793e-06 eta: 3:16:02 time: 2.3939 data_time: 0.0102 memory: 10955 loss: 0.1729 2024/07/25 01:01:41 - mmengine - INFO - Iter(train) [14700/19224] lr: 2.7676e-06 eta: 3:15:35 time: 2.1599 data_time: 0.0107 memory: 10763 loss: 0.1683 2024/07/25 01:01:59 - mmengine - INFO - Iter(train) [14710/19224] lr: 2.7560e-06 eta: 3:15:06 time: 1.7469 data_time: 0.0093 memory: 10238 loss: 0.1549 2024/07/25 01:02:16 - mmengine - INFO - Iter(train) [14720/19224] lr: 2.7444e-06 eta: 3:14:38 time: 1.7305 data_time: 0.0087 memory: 13749 loss: 0.1983 2024/07/25 01:02:50 - mmengine - INFO - Iter(train) [14730/19224] lr: 2.7328e-06 eta: 3:14:14 time: 3.3758 data_time: 0.0105 memory: 13376 loss: 0.1693 2024/07/25 01:03:20 - mmengine - INFO - Iter(train) [14740/19224] lr: 2.7213e-06 eta: 3:13:50 time: 3.0644 data_time: 0.0109 memory: 12111 loss: 0.1474 2024/07/25 01:03:50 - mmengine - INFO - Iter(train) [14750/19224] lr: 2.7097e-06 eta: 3:13:25 time: 2.9813 data_time: 0.0108 memory: 11889 loss: 0.1525 2024/07/25 01:04:19 - mmengine - INFO - Iter(train) [14760/19224] lr: 2.6982e-06 eta: 3:13:00 time: 2.8605 data_time: 0.0108 memory: 11718 loss: 0.1548 2024/07/25 01:04:46 - mmengine - INFO - Iter(train) [14770/19224] lr: 2.6867e-06 eta: 3:12:34 time: 2.7085 data_time: 0.0105 memory: 11437 loss: 0.1378 2024/07/25 01:05:13 - mmengine - INFO - Iter(train) [14780/19224] lr: 2.6752e-06 eta: 3:12:09 time: 2.7154 data_time: 0.0103 memory: 11275 loss: 0.1567 2024/07/25 01:05:38 - mmengine - INFO - Iter(train) [14790/19224] lr: 2.6638e-06 eta: 3:11:43 time: 2.5000 data_time: 0.0107 memory: 11137 loss: 0.1306 2024/07/25 01:06:00 - mmengine - INFO - Iter(train) [14800/19224] lr: 2.6523e-06 eta: 3:11:15 time: 2.2283 data_time: 0.0103 memory: 10914 loss: 0.1524 2024/07/25 01:06:19 - mmengine - INFO - Iter(train) [14810/19224] lr: 2.6409e-06 eta: 3:10:47 time: 1.8546 data_time: 0.0093 memory: 10204 loss: 0.1635 2024/07/25 01:06:37 - mmengine - INFO - Iter(train) [14820/19224] lr: 2.6295e-06 eta: 3:10:19 time: 1.7750 data_time: 0.0094 memory: 13400 loss: 0.1623 2024/07/25 01:07:08 - mmengine - INFO - Iter(train) [14830/19224] lr: 2.6181e-06 eta: 3:09:55 time: 3.1417 data_time: 0.0104 memory: 12599 loss: 0.1579 2024/07/25 01:07:37 - mmengine - INFO - Iter(train) [14840/19224] lr: 2.6068e-06 eta: 3:09:30 time: 2.8828 data_time: 0.0104 memory: 11911 loss: 0.1507 2024/07/25 01:08:05 - mmengine - INFO - Iter(train) [14850/19224] lr: 2.5954e-06 eta: 3:09:04 time: 2.8585 data_time: 0.0111 memory: 11733 loss: 0.1570 2024/07/25 01:08:33 - mmengine - INFO - Iter(train) [14860/19224] lr: 2.5841e-06 eta: 3:08:39 time: 2.7663 data_time: 0.0109 memory: 11479 loss: 0.1684 2024/07/25 01:09:00 - mmengine - INFO - Iter(train) [14870/19224] lr: 2.5728e-06 eta: 3:08:13 time: 2.6710 data_time: 0.0104 memory: 11580 loss: 0.1371 2024/07/25 01:09:25 - mmengine - INFO - Iter(train) [14880/19224] lr: 2.5616e-06 eta: 3:07:47 time: 2.5103 data_time: 0.0103 memory: 11408 loss: 0.1433 2024/07/25 01:09:49 - mmengine - INFO - Iter(train) [14890/19224] lr: 2.5503e-06 eta: 3:07:21 time: 2.4053 data_time: 0.0101 memory: 11013 loss: 0.1401 2024/07/25 01:10:10 - mmengine - INFO - Iter(train) [14900/19224] lr: 2.5391e-06 eta: 3:06:53 time: 2.1431 data_time: 0.0103 memory: 10715 loss: 0.1581 2024/07/25 01:10:28 - mmengine - INFO - Iter(train) [14910/19224] lr: 2.5279e-06 eta: 3:06:25 time: 1.8008 data_time: 0.0092 memory: 10296 loss: 0.1657 2024/07/25 01:10:47 - mmengine - INFO - Iter(train) [14920/19224] lr: 2.5167e-06 eta: 3:05:57 time: 1.8857 data_time: 0.0089 memory: 14087 loss: 0.1819 2024/07/25 01:11:18 - mmengine - INFO - Iter(train) [14930/19224] lr: 2.5055e-06 eta: 3:05:33 time: 3.1016 data_time: 0.0106 memory: 12260 loss: 0.1373 2024/07/25 01:11:48 - mmengine - INFO - Iter(train) [14940/19224] lr: 2.4944e-06 eta: 3:05:08 time: 2.9431 data_time: 0.0104 memory: 11905 loss: 0.1594 2024/07/25 01:12:17 - mmengine - INFO - Iter(train) [14950/19224] lr: 2.4833e-06 eta: 3:04:43 time: 2.9162 data_time: 0.0105 memory: 11747 loss: 0.1425 2024/07/25 01:12:44 - mmengine - INFO - Iter(train) [14960/19224] lr: 2.4722e-06 eta: 3:04:17 time: 2.7526 data_time: 0.0109 memory: 11430 loss: 0.1631 2024/07/25 01:13:11 - mmengine - INFO - Iter(train) [14970/19224] lr: 2.4611e-06 eta: 3:03:52 time: 2.7184 data_time: 0.0104 memory: 11291 loss: 0.1481 2024/07/25 01:13:38 - mmengine - INFO - Iter(train) [14980/19224] lr: 2.4500e-06 eta: 3:03:26 time: 2.6066 data_time: 0.0103 memory: 11203 loss: 0.1530 2024/07/25 01:14:02 - mmengine - INFO - Iter(train) [14990/19224] lr: 2.4390e-06 eta: 3:03:00 time: 2.4744 data_time: 0.0101 memory: 11019 loss: 0.1575 2024/07/25 01:14:24 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/25 01:14:24 - mmengine - INFO - Iter(train) [15000/19224] lr: 2.4280e-06 eta: 3:02:33 time: 2.2091 data_time: 0.0099 memory: 10806 loss: 0.1467 2024/07/25 01:14:24 - mmengine - INFO - Saving checkpoint at 15000 iterations 2024/07/25 01:14:45 - mmengine - INFO - Iter(train) [15010/19224] lr: 2.4170e-06 eta: 3:02:05 time: 2.0270 data_time: 0.1877 memory: 10245 loss: 0.1772 2024/07/25 01:15:06 - mmengine - INFO - Iter(train) [15020/19224] lr: 2.4060e-06 eta: 3:01:38 time: 2.1359 data_time: 0.0087 memory: 16632 loss: 0.2010 2024/07/25 01:15:41 - mmengine - INFO - Iter(train) [15030/19224] lr: 2.3951e-06 eta: 3:01:15 time: 3.5407 data_time: 0.0102 memory: 14140 loss: 0.1839 2024/07/25 01:16:13 - mmengine - INFO - Iter(train) [15040/19224] lr: 2.3841e-06 eta: 3:00:50 time: 3.1633 data_time: 0.0105 memory: 12402 loss: 0.1499 2024/07/25 01:16:43 - mmengine - INFO - Iter(train) [15050/19224] lr: 2.3732e-06 eta: 3:00:25 time: 2.9979 data_time: 0.0105 memory: 12083 loss: 0.1309 2024/07/25 01:17:12 - mmengine - INFO - Iter(train) [15060/19224] lr: 2.3623e-06 eta: 3:00:00 time: 2.9287 data_time: 0.0103 memory: 11756 loss: 0.1476 2024/07/25 01:17:40 - mmengine - INFO - Iter(train) [15070/19224] lr: 2.3515e-06 eta: 2:59:35 time: 2.7827 data_time: 0.0105 memory: 11562 loss: 0.1569 2024/07/25 01:18:07 - mmengine - INFO - Iter(train) [15080/19224] lr: 2.3406e-06 eta: 2:59:09 time: 2.6876 data_time: 0.0103 memory: 11307 loss: 0.1611 2024/07/25 01:18:32 - mmengine - INFO - Iter(train) [15090/19224] lr: 2.3298e-06 eta: 2:58:43 time: 2.4878 data_time: 0.0104 memory: 11093 loss: 0.1494 2024/07/25 01:18:56 - mmengine - INFO - Iter(train) [15100/19224] lr: 2.3190e-06 eta: 2:58:17 time: 2.3962 data_time: 0.0101 memory: 10963 loss: 0.1590 2024/07/25 01:19:17 - mmengine - INFO - Iter(train) [15110/19224] lr: 2.3082e-06 eta: 2:57:49 time: 2.1388 data_time: 0.0102 memory: 10639 loss: 0.1394 2024/07/25 01:19:41 - mmengine - INFO - Iter(train) [15120/19224] lr: 2.2975e-06 eta: 2:57:23 time: 2.3440 data_time: 0.0091 memory: 17245 loss: 0.1675 2024/07/25 01:20:16 - mmengine - INFO - Iter(train) [15130/19224] lr: 2.2868e-06 eta: 2:56:59 time: 3.5097 data_time: 0.0110 memory: 14611 loss: 0.1659 2024/07/25 01:20:46 - mmengine - INFO - Iter(train) [15140/19224] lr: 2.2760e-06 eta: 2:56:35 time: 3.0363 data_time: 0.0107 memory: 12217 loss: 0.1357 2024/07/25 01:21:15 - mmengine - INFO - Iter(train) [15150/19224] lr: 2.2654e-06 eta: 2:56:09 time: 2.8958 data_time: 0.0104 memory: 11832 loss: 0.1554 2024/07/25 01:21:43 - mmengine - INFO - Iter(train) [15160/19224] lr: 2.2547e-06 eta: 2:55:44 time: 2.7581 data_time: 0.0102 memory: 11832 loss: 0.1616 2024/07/25 01:22:10 - mmengine - INFO - Iter(train) [15170/19224] lr: 2.2440e-06 eta: 2:55:18 time: 2.7440 data_time: 0.0103 memory: 11282 loss: 0.1614 2024/07/25 01:22:36 - mmengine - INFO - Iter(train) [15180/19224] lr: 2.2334e-06 eta: 2:54:52 time: 2.5752 data_time: 0.0107 memory: 11196 loss: 0.1538 2024/07/25 01:23:00 - mmengine - INFO - Iter(train) [15190/19224] lr: 2.2228e-06 eta: 2:54:26 time: 2.3716 data_time: 0.0102 memory: 10936 loss: 0.1704 2024/07/25 01:23:21 - mmengine - INFO - Iter(train) [15200/19224] lr: 2.2122e-06 eta: 2:53:59 time: 2.1371 data_time: 0.0108 memory: 10638 loss: 0.1542 2024/07/25 01:23:39 - mmengine - INFO - Iter(train) [15210/19224] lr: 2.2017e-06 eta: 2:53:31 time: 1.8473 data_time: 0.0094 memory: 10263 loss: 0.1577 2024/07/25 01:23:58 - mmengine - INFO - Iter(train) [15220/19224] lr: 2.1911e-06 eta: 2:53:03 time: 1.8426 data_time: 0.0084 memory: 15275 loss: 0.1430 2024/07/25 01:24:29 - mmengine - INFO - Iter(train) [15230/19224] lr: 2.1806e-06 eta: 2:52:38 time: 3.1582 data_time: 0.0102 memory: 12395 loss: 0.1612 2024/07/25 01:25:00 - mmengine - INFO - Iter(train) [15240/19224] lr: 2.1701e-06 eta: 2:52:14 time: 3.0122 data_time: 0.0107 memory: 11945 loss: 0.1417 2024/07/25 01:25:28 - mmengine - INFO - Iter(train) [15250/19224] lr: 2.1597e-06 eta: 2:51:48 time: 2.8727 data_time: 0.0102 memory: 11718 loss: 0.1546 2024/07/25 01:25:56 - mmengine - INFO - Iter(train) [15260/19224] lr: 2.1492e-06 eta: 2:51:23 time: 2.7728 data_time: 0.0102 memory: 11501 loss: 0.1351 2024/07/25 01:26:23 - mmengine - INFO - Iter(train) [15270/19224] lr: 2.1388e-06 eta: 2:50:57 time: 2.6488 data_time: 0.0108 memory: 11299 loss: 0.1357 2024/07/25 01:26:47 - mmengine - INFO - Iter(train) [15280/19224] lr: 2.1284e-06 eta: 2:50:31 time: 2.4786 data_time: 0.0101 memory: 10969 loss: 0.1604 2024/07/25 01:27:10 - mmengine - INFO - Iter(train) [15290/19224] lr: 2.1180e-06 eta: 2:50:04 time: 2.3146 data_time: 0.0103 memory: 10825 loss: 0.1677 2024/07/25 01:27:32 - mmengine - INFO - Iter(train) [15300/19224] lr: 2.1077e-06 eta: 2:49:37 time: 2.1070 data_time: 0.0094 memory: 10492 loss: 0.1326 2024/07/25 01:27:51 - mmengine - INFO - Iter(train) [15310/19224] lr: 2.0973e-06 eta: 2:49:09 time: 1.9031 data_time: 0.0091 memory: 10216 loss: 0.1487 2024/07/25 01:28:09 - mmengine - INFO - Iter(train) [15320/19224] lr: 2.0870e-06 eta: 2:48:42 time: 1.8754 data_time: 0.0092 memory: 13685 loss: 0.1355 2024/07/25 01:28:40 - mmengine - INFO - Iter(train) [15330/19224] lr: 2.0767e-06 eta: 2:48:17 time: 3.0997 data_time: 0.0105 memory: 12526 loss: 0.1749 2024/07/25 01:29:09 - mmengine - INFO - Iter(train) [15340/19224] lr: 2.0665e-06 eta: 2:47:52 time: 2.8345 data_time: 0.0106 memory: 11713 loss: 0.1596 2024/07/25 01:29:36 - mmengine - INFO - Iter(train) [15350/19224] lr: 2.0562e-06 eta: 2:47:26 time: 2.7554 data_time: 0.0103 memory: 11625 loss: 0.1442 2024/07/25 01:30:06 - mmengine - INFO - Iter(train) [15360/19224] lr: 2.0460e-06 eta: 2:47:01 time: 2.9330 data_time: 0.0108 memory: 11312 loss: 0.1838 2024/07/25 01:30:31 - mmengine - INFO - Iter(train) [15370/19224] lr: 2.0358e-06 eta: 2:46:35 time: 2.5561 data_time: 0.0106 memory: 11183 loss: 0.1580 2024/07/25 01:30:56 - mmengine - INFO - Iter(train) [15380/19224] lr: 2.0256e-06 eta: 2:46:09 time: 2.4468 data_time: 0.0106 memory: 11049 loss: 0.1605 2024/07/25 01:31:19 - mmengine - INFO - Iter(train) [15390/19224] lr: 2.0155e-06 eta: 2:45:42 time: 2.2979 data_time: 0.0106 memory: 10842 loss: 0.1559 2024/07/25 01:31:40 - mmengine - INFO - Iter(train) [15400/19224] lr: 2.0053e-06 eta: 2:45:15 time: 2.1299 data_time: 0.0101 memory: 10550 loss: 0.2636 2024/07/25 01:31:59 - mmengine - INFO - Iter(train) [15410/19224] lr: 1.9952e-06 eta: 2:44:47 time: 1.8636 data_time: 0.0094 memory: 10218 loss: 0.1531 2024/07/25 01:32:20 - mmengine - INFO - Iter(train) [15420/19224] lr: 1.9851e-06 eta: 2:44:20 time: 2.1893 data_time: 0.0088 memory: 18895 loss: 0.1617 2024/07/25 01:32:54 - mmengine - INFO - Iter(train) [15430/19224] lr: 1.9751e-06 eta: 2:43:56 time: 3.3155 data_time: 0.0105 memory: 13558 loss: 0.1620 2024/07/25 01:33:24 - mmengine - INFO - Iter(train) [15440/19224] lr: 1.9650e-06 eta: 2:43:31 time: 3.0001 data_time: 0.0110 memory: 11868 loss: 0.1467 2024/07/25 01:33:52 - mmengine - INFO - Iter(train) [15450/19224] lr: 1.9550e-06 eta: 2:43:06 time: 2.8295 data_time: 0.0105 memory: 11656 loss: 0.1400 2024/07/25 01:34:20 - mmengine - INFO - Iter(train) [15460/19224] lr: 1.9450e-06 eta: 2:42:40 time: 2.8355 data_time: 0.0104 memory: 11666 loss: 0.1510 2024/07/25 01:34:47 - mmengine - INFO - Iter(train) [15470/19224] lr: 1.9351e-06 eta: 2:42:15 time: 2.6651 data_time: 0.0109 memory: 11357 loss: 0.1599 2024/07/25 01:35:14 - mmengine - INFO - Iter(train) [15480/19224] lr: 1.9251e-06 eta: 2:41:49 time: 2.6730 data_time: 0.0105 memory: 11557 loss: 0.1398 2024/07/25 01:35:39 - mmengine - INFO - Iter(train) [15490/19224] lr: 1.9152e-06 eta: 2:41:23 time: 2.5115 data_time: 0.0107 memory: 11049 loss: 0.1793 2024/07/25 01:36:01 - mmengine - INFO - Iter(train) [15500/19224] lr: 1.9053e-06 eta: 2:40:56 time: 2.2510 data_time: 0.0102 memory: 10791 loss: 0.1508 2024/07/25 01:36:20 - mmengine - INFO - Iter(train) [15510/19224] lr: 1.8954e-06 eta: 2:40:28 time: 1.8846 data_time: 0.0095 memory: 10273 loss: 0.1676 2024/07/25 01:36:39 - mmengine - INFO - Iter(train) [15520/19224] lr: 1.8855e-06 eta: 2:40:01 time: 1.8568 data_time: 0.0087 memory: 15416 loss: 0.1471 2024/07/25 01:37:11 - mmengine - INFO - Iter(train) [15530/19224] lr: 1.8757e-06 eta: 2:39:36 time: 3.1997 data_time: 0.0105 memory: 12528 loss: 0.1523 2024/07/25 01:37:41 - mmengine - INFO - Iter(train) [15540/19224] lr: 1.8659e-06 eta: 2:39:11 time: 3.0291 data_time: 0.0104 memory: 12078 loss: 0.1402 2024/07/25 01:38:10 - mmengine - INFO - Iter(train) [15550/19224] lr: 1.8561e-06 eta: 2:38:46 time: 2.9102 data_time: 0.0114 memory: 11729 loss: 0.1480 2024/07/25 01:38:37 - mmengine - INFO - Iter(train) [15560/19224] lr: 1.8463e-06 eta: 2:38:21 time: 2.7230 data_time: 0.0104 memory: 11395 loss: 0.1584 2024/07/25 01:39:04 - mmengine - INFO - Iter(train) [15570/19224] lr: 1.8366e-06 eta: 2:37:55 time: 2.6365 data_time: 0.0105 memory: 11221 loss: 0.1502 2024/07/25 01:39:28 - mmengine - INFO - Iter(train) [15580/19224] lr: 1.8269e-06 eta: 2:37:29 time: 2.4604 data_time: 0.0103 memory: 11049 loss: 0.1645 2024/07/25 01:39:51 - mmengine - INFO - Iter(train) [15590/19224] lr: 1.8172e-06 eta: 2:37:02 time: 2.2698 data_time: 0.0106 memory: 10835 loss: 0.1639 2024/07/25 01:40:11 - mmengine - INFO - Iter(train) [15600/19224] lr: 1.8075e-06 eta: 2:36:34 time: 1.9704 data_time: 0.0099 memory: 10441 loss: 0.1659 2024/07/25 01:40:29 - mmengine - INFO - Iter(train) [15610/19224] lr: 1.7979e-06 eta: 2:36:07 time: 1.8195 data_time: 0.0091 memory: 10050 loss: 0.1643 2024/07/25 01:40:49 - mmengine - INFO - Iter(train) [15620/19224] lr: 1.7882e-06 eta: 2:35:40 time: 2.0523 data_time: 0.0093 memory: 14357 loss: 0.1414 2024/07/25 01:41:20 - mmengine - INFO - Iter(train) [15630/19224] lr: 1.7786e-06 eta: 2:35:15 time: 3.0929 data_time: 0.0109 memory: 12061 loss: 0.1505 2024/07/25 01:41:49 - mmengine - INFO - Iter(train) [15640/19224] lr: 1.7691e-06 eta: 2:34:50 time: 2.9194 data_time: 0.0105 memory: 11695 loss: 0.1346 2024/07/25 01:42:18 - mmengine - INFO - Iter(train) [15650/19224] lr: 1.7595e-06 eta: 2:34:24 time: 2.8780 data_time: 0.0106 memory: 11830 loss: 0.1379 2024/07/25 01:42:46 - mmengine - INFO - Iter(train) [15660/19224] lr: 1.7500e-06 eta: 2:33:59 time: 2.7398 data_time: 0.0101 memory: 11308 loss: 0.1508 2024/07/25 01:43:12 - mmengine - INFO - Iter(train) [15670/19224] lr: 1.7405e-06 eta: 2:33:33 time: 2.6437 data_time: 0.0111 memory: 11179 loss: 0.1498 2024/07/25 01:43:38 - mmengine - INFO - Iter(train) [15680/19224] lr: 1.7310e-06 eta: 2:33:07 time: 2.6203 data_time: 0.0122 memory: 11087 loss: 0.1361 2024/07/25 01:44:03 - mmengine - INFO - Iter(train) [15690/19224] lr: 1.7215e-06 eta: 2:32:41 time: 2.4766 data_time: 0.0106 memory: 11149 loss: 0.1678 2024/07/25 01:44:25 - mmengine - INFO - Iter(train) [15700/19224] lr: 1.7121e-06 eta: 2:32:14 time: 2.1488 data_time: 0.0096 memory: 10798 loss: 0.1748 2024/07/25 01:44:43 - mmengine - INFO - Iter(train) [15710/19224] lr: 1.7027e-06 eta: 2:31:46 time: 1.8456 data_time: 0.0094 memory: 10215 loss: 0.1649 2024/07/25 01:45:02 - mmengine - INFO - Iter(train) [15720/19224] lr: 1.6933e-06 eta: 2:31:19 time: 1.8616 data_time: 0.0092 memory: 15692 loss: 0.1734 2024/07/25 01:45:34 - mmengine - INFO - Iter(train) [15730/19224] lr: 1.6839e-06 eta: 2:30:55 time: 3.2732 data_time: 0.0108 memory: 13180 loss: 0.1564 2024/07/25 01:46:04 - mmengine - INFO - Iter(train) [15740/19224] lr: 1.6746e-06 eta: 2:30:29 time: 2.9454 data_time: 0.0111 memory: 12139 loss: 0.1580 2024/07/25 01:46:32 - mmengine - INFO - Iter(train) [15750/19224] lr: 1.6652e-06 eta: 2:30:04 time: 2.8323 data_time: 0.0110 memory: 11770 loss: 0.1583 2024/07/25 01:47:00 - mmengine - INFO - Iter(train) [15760/19224] lr: 1.6559e-06 eta: 2:29:39 time: 2.8154 data_time: 0.0105 memory: 11486 loss: 0.1567 2024/07/25 01:47:27 - mmengine - INFO - Iter(train) [15770/19224] lr: 1.6467e-06 eta: 2:29:13 time: 2.6793 data_time: 0.0108 memory: 11417 loss: 0.1571 2024/07/25 01:47:53 - mmengine - INFO - Iter(train) [15780/19224] lr: 1.6374e-06 eta: 2:28:47 time: 2.5819 data_time: 0.0107 memory: 11236 loss: 0.1556 2024/07/25 01:48:17 - mmengine - INFO - Iter(train) [15790/19224] lr: 1.6282e-06 eta: 2:28:21 time: 2.4572 data_time: 0.0106 memory: 10985 loss: 0.1527 2024/07/25 01:48:39 - mmengine - INFO - Iter(train) [15800/19224] lr: 1.6190e-06 eta: 2:27:54 time: 2.1824 data_time: 0.0099 memory: 10832 loss: 0.1948 2024/07/25 01:48:57 - mmengine - INFO - Iter(train) [15810/19224] lr: 1.6098e-06 eta: 2:27:26 time: 1.8044 data_time: 0.0094 memory: 10241 loss: 0.1741 2024/07/25 01:49:17 - mmengine - INFO - Iter(train) [15820/19224] lr: 1.6007e-06 eta: 2:26:59 time: 1.9678 data_time: 0.0089 memory: 15849 loss: 0.2128 2024/07/25 01:49:49 - mmengine - INFO - Iter(train) [15830/19224] lr: 1.5915e-06 eta: 2:26:34 time: 3.2453 data_time: 0.0111 memory: 12872 loss: 0.1992 2024/07/25 01:50:20 - mmengine - INFO - Iter(train) [15840/19224] lr: 1.5824e-06 eta: 2:26:09 time: 3.0210 data_time: 0.0105 memory: 12146 loss: 0.1416 2024/07/25 01:50:49 - mmengine - INFO - Iter(train) [15850/19224] lr: 1.5733e-06 eta: 2:25:44 time: 2.9446 data_time: 0.0104 memory: 12017 loss: 0.1774 2024/07/25 01:51:18 - mmengine - INFO - Iter(train) [15860/19224] lr: 1.5643e-06 eta: 2:25:19 time: 2.8504 data_time: 0.0104 memory: 11524 loss: 0.1448 2024/07/25 01:51:45 - mmengine - INFO - Iter(train) [15870/19224] lr: 1.5552e-06 eta: 2:24:53 time: 2.7363 data_time: 0.0107 memory: 11295 loss: 0.1543 2024/07/25 01:52:11 - mmengine - INFO - Iter(train) [15880/19224] lr: 1.5462e-06 eta: 2:24:27 time: 2.5767 data_time: 0.0105 memory: 11217 loss: 0.1697 2024/07/25 01:52:35 - mmengine - INFO - Iter(train) [15890/19224] lr: 1.5372e-06 eta: 2:24:01 time: 2.4370 data_time: 0.0106 memory: 10907 loss: 0.1388 2024/07/25 01:52:55 - mmengine - INFO - Iter(train) [15900/19224] lr: 1.5283e-06 eta: 2:23:34 time: 2.0298 data_time: 0.0099 memory: 10681 loss: 0.1211 2024/07/25 01:53:12 - mmengine - INFO - Iter(train) [15910/19224] lr: 1.5193e-06 eta: 2:23:06 time: 1.6737 data_time: 0.0098 memory: 10025 loss: 0.1556 2024/07/25 01:53:32 - mmengine - INFO - Iter(train) [15920/19224] lr: 1.5104e-06 eta: 2:22:39 time: 1.9903 data_time: 0.0084 memory: 17914 loss: 0.1870 2024/07/25 01:54:05 - mmengine - INFO - Iter(train) [15930/19224] lr: 1.5015e-06 eta: 2:22:15 time: 3.3202 data_time: 0.0109 memory: 13731 loss: 0.1527 2024/07/25 01:54:36 - mmengine - INFO - Iter(train) [15940/19224] lr: 1.4927e-06 eta: 2:21:50 time: 3.0416 data_time: 0.0108 memory: 12045 loss: 0.1399 2024/07/25 01:55:04 - mmengine - INFO - Iter(train) [15950/19224] lr: 1.4838e-06 eta: 2:21:24 time: 2.8696 data_time: 0.0110 memory: 11640 loss: 0.1375 2024/07/25 01:55:32 - mmengine - INFO - Iter(train) [15960/19224] lr: 1.4750e-06 eta: 2:20:59 time: 2.7863 data_time: 0.0104 memory: 11448 loss: 0.1700 2024/07/25 01:55:58 - mmengine - INFO - Iter(train) [15970/19224] lr: 1.4662e-06 eta: 2:20:33 time: 2.5760 data_time: 0.0105 memory: 11207 loss: 0.1712 2024/07/25 01:56:23 - mmengine - INFO - Iter(train) [15980/19224] lr: 1.4574e-06 eta: 2:20:07 time: 2.5221 data_time: 0.0108 memory: 11096 loss: 0.1455 2024/07/25 01:56:45 - mmengine - INFO - Iter(train) [15990/19224] lr: 1.4487e-06 eta: 2:19:40 time: 2.2234 data_time: 0.0102 memory: 10871 loss: 0.1539 2024/07/25 01:57:05 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/25 01:57:05 - mmengine - INFO - Iter(train) [16000/19224] lr: 1.4400e-06 eta: 2:19:13 time: 1.9233 data_time: 0.0094 memory: 10354 loss: 0.1626 2024/07/25 01:57:05 - mmengine - INFO - Saving checkpoint at 16000 iterations 2024/07/25 01:57:24 - mmengine - INFO - Iter(train) [16010/19224] lr: 1.4313e-06 eta: 2:18:46 time: 1.9357 data_time: 0.1869 memory: 10000 loss: 0.1913 2024/07/25 01:57:45 - mmengine - INFO - Iter(train) [16020/19224] lr: 1.4226e-06 eta: 2:18:19 time: 2.0585 data_time: 0.0093 memory: 16719 loss: 0.1659 2024/07/25 01:58:19 - mmengine - INFO - Iter(train) [16030/19224] lr: 1.4140e-06 eta: 2:17:54 time: 3.3986 data_time: 0.0105 memory: 13555 loss: 0.1885 2024/07/25 01:58:51 - mmengine - INFO - Iter(train) [16040/19224] lr: 1.4053e-06 eta: 2:17:30 time: 3.2424 data_time: 0.0108 memory: 12255 loss: 0.1437 2024/07/25 01:59:20 - mmengine - INFO - Iter(train) [16050/19224] lr: 1.3967e-06 eta: 2:17:05 time: 2.9334 data_time: 0.0110 memory: 11839 loss: 0.1430 2024/07/25 01:59:49 - mmengine - INFO - Iter(train) [16060/19224] lr: 1.3882e-06 eta: 2:16:39 time: 2.8952 data_time: 0.0106 memory: 11577 loss: 0.1592 2024/07/25 02:00:19 - mmengine - INFO - Iter(train) [16070/19224] lr: 1.3796e-06 eta: 2:16:14 time: 3.0143 data_time: 0.0107 memory: 11336 loss: 0.1482 2024/07/25 02:00:46 - mmengine - INFO - Iter(train) [16080/19224] lr: 1.3711e-06 eta: 2:15:48 time: 2.6084 data_time: 0.0106 memory: 11230 loss: 0.1459 2024/07/25 02:01:10 - mmengine - INFO - Iter(train) [16090/19224] lr: 1.3626e-06 eta: 2:15:22 time: 2.4428 data_time: 0.0105 memory: 11027 loss: 0.1646 2024/07/25 02:01:33 - mmengine - INFO - Iter(train) [16100/19224] lr: 1.3541e-06 eta: 2:14:56 time: 2.2705 data_time: 0.0105 memory: 10806 loss: 0.1417 2024/07/25 02:01:50 - mmengine - INFO - Iter(train) [16110/19224] lr: 1.3456e-06 eta: 2:14:28 time: 1.7741 data_time: 0.0095 memory: 10274 loss: 0.1794 2024/07/25 02:02:09 - mmengine - INFO - Iter(train) [16120/19224] lr: 1.3372e-06 eta: 2:14:01 time: 1.8356 data_time: 0.0088 memory: 15899 loss: 0.1600 2024/07/25 02:02:42 - mmengine - INFO - Iter(train) [16130/19224] lr: 1.3288e-06 eta: 2:13:36 time: 3.2743 data_time: 0.0106 memory: 13063 loss: 0.1554 2024/07/25 02:03:11 - mmengine - INFO - Iter(train) [16140/19224] lr: 1.3204e-06 eta: 2:13:11 time: 2.9697 data_time: 0.0104 memory: 12180 loss: 0.1584 2024/07/25 02:03:40 - mmengine - INFO - Iter(train) [16150/19224] lr: 1.3121e-06 eta: 2:12:46 time: 2.9227 data_time: 0.0107 memory: 12172 loss: 0.1522 2024/07/25 02:04:08 - mmengine - INFO - Iter(train) [16160/19224] lr: 1.3038e-06 eta: 2:12:20 time: 2.7780 data_time: 0.0108 memory: 11535 loss: 0.1578 2024/07/25 02:04:35 - mmengine - INFO - Iter(train) [16170/19224] lr: 1.2954e-06 eta: 2:11:54 time: 2.6332 data_time: 0.0106 memory: 11312 loss: 0.1655 2024/07/25 02:05:00 - mmengine - INFO - Iter(train) [16180/19224] lr: 1.2872e-06 eta: 2:11:28 time: 2.5900 data_time: 0.0103 memory: 11299 loss: 0.1473 2024/07/25 02:05:25 - mmengine - INFO - Iter(train) [16190/19224] lr: 1.2789e-06 eta: 2:11:02 time: 2.4622 data_time: 0.0106 memory: 11091 loss: 0.1390 2024/07/25 02:05:49 - mmengine - INFO - Iter(train) [16200/19224] lr: 1.2707e-06 eta: 2:10:36 time: 2.3794 data_time: 0.0102 memory: 11000 loss: 0.1417 2024/07/25 02:06:08 - mmengine - INFO - Iter(train) [16210/19224] lr: 1.2625e-06 eta: 2:10:09 time: 1.9029 data_time: 0.0095 memory: 10374 loss: 0.1612 2024/07/25 02:06:25 - mmengine - INFO - Iter(train) [16220/19224] lr: 1.2543e-06 eta: 2:09:41 time: 1.7076 data_time: 0.0087 memory: 13118 loss: 0.1644 2024/07/25 02:06:58 - mmengine - INFO - Iter(train) [16230/19224] lr: 1.2461e-06 eta: 2:09:16 time: 3.3040 data_time: 0.0113 memory: 12963 loss: 0.1591 2024/07/25 02:07:28 - mmengine - INFO - Iter(train) [16240/19224] lr: 1.2380e-06 eta: 2:08:51 time: 2.9767 data_time: 0.0109 memory: 12120 loss: 0.1478 2024/07/25 02:07:57 - mmengine - INFO - Iter(train) [16250/19224] lr: 1.2299e-06 eta: 2:08:26 time: 2.9320 data_time: 0.0106 memory: 11885 loss: 0.1730 2024/07/25 02:08:25 - mmengine - INFO - Iter(train) [16260/19224] lr: 1.2218e-06 eta: 2:08:00 time: 2.8184 data_time: 0.0108 memory: 11548 loss: 0.1513 2024/07/25 02:08:51 - mmengine - INFO - Iter(train) [16270/19224] lr: 1.2138e-06 eta: 2:07:35 time: 2.5592 data_time: 0.0106 memory: 11278 loss: 0.1634 2024/07/25 02:09:15 - mmengine - INFO - Iter(train) [16280/19224] lr: 1.2057e-06 eta: 2:07:08 time: 2.4495 data_time: 0.0104 memory: 11068 loss: 0.1442 2024/07/25 02:09:39 - mmengine - INFO - Iter(train) [16290/19224] lr: 1.1977e-06 eta: 2:06:42 time: 2.3262 data_time: 0.0102 memory: 10895 loss: 0.1597 2024/07/25 02:09:58 - mmengine - INFO - Iter(train) [16300/19224] lr: 1.1897e-06 eta: 2:06:15 time: 1.9670 data_time: 0.0097 memory: 10507 loss: 0.1610 2024/07/25 02:10:14 - mmengine - INFO - Iter(train) [16310/19224] lr: 1.1818e-06 eta: 2:05:47 time: 1.6044 data_time: 0.0096 memory: 9962 loss: 0.1719 2024/07/25 02:10:32 - mmengine - INFO - Iter(train) [16320/19224] lr: 1.1738e-06 eta: 2:05:20 time: 1.7336 data_time: 0.0086 memory: 14425 loss: 0.1574 2024/07/25 02:11:05 - mmengine - INFO - Iter(train) [16330/19224] lr: 1.1659e-06 eta: 2:04:55 time: 3.2905 data_time: 0.0107 memory: 13275 loss: 0.2265 2024/07/25 02:11:35 - mmengine - INFO - Iter(train) [16340/19224] lr: 1.1581e-06 eta: 2:04:30 time: 3.0196 data_time: 0.0106 memory: 12097 loss: 0.1418 2024/07/25 02:12:03 - mmengine - INFO - Iter(train) [16350/19224] lr: 1.1502e-06 eta: 2:04:05 time: 2.8088 data_time: 0.0112 memory: 11593 loss: 0.1587 2024/07/25 02:12:30 - mmengine - INFO - Iter(train) [16360/19224] lr: 1.1424e-06 eta: 2:03:39 time: 2.7111 data_time: 0.0104 memory: 11374 loss: 0.1751 2024/07/25 02:12:56 - mmengine - INFO - Iter(train) [16370/19224] lr: 1.1346e-06 eta: 2:03:13 time: 2.6306 data_time: 0.0105 memory: 11297 loss: 0.1584 2024/07/25 02:13:21 - mmengine - INFO - Iter(train) [16380/19224] lr: 1.1268e-06 eta: 2:02:47 time: 2.4357 data_time: 0.0103 memory: 11065 loss: 0.1559 2024/07/25 02:13:44 - mmengine - INFO - Iter(train) [16390/19224] lr: 1.1190e-06 eta: 2:02:20 time: 2.3415 data_time: 0.0105 memory: 10882 loss: 0.1657 2024/07/25 02:14:04 - mmengine - INFO - Iter(train) [16400/19224] lr: 1.1113e-06 eta: 2:01:54 time: 2.0049 data_time: 0.0094 memory: 10373 loss: 0.1496 2024/07/25 02:14:21 - mmengine - INFO - Iter(train) [16410/19224] lr: 1.1036e-06 eta: 2:01:26 time: 1.7350 data_time: 0.0093 memory: 10115 loss: 0.1600 2024/07/25 02:14:40 - mmengine - INFO - Iter(train) [16420/19224] lr: 1.0959e-06 eta: 2:00:59 time: 1.9007 data_time: 0.0097 memory: 13882 loss: 0.1405 2024/07/25 02:15:11 - mmengine - INFO - Iter(train) [16430/19224] lr: 1.0883e-06 eta: 2:00:34 time: 3.0999 data_time: 0.0113 memory: 12313 loss: 0.1501 2024/07/25 02:15:40 - mmengine - INFO - Iter(train) [16440/19224] lr: 1.0806e-06 eta: 2:00:09 time: 2.8730 data_time: 0.0106 memory: 11733 loss: 0.1510 2024/07/25 02:16:08 - mmengine - INFO - Iter(train) [16450/19224] lr: 1.0730e-06 eta: 1:59:43 time: 2.7586 data_time: 0.0108 memory: 11528 loss: 0.1548 2024/07/25 02:16:35 - mmengine - INFO - Iter(train) [16460/19224] lr: 1.0654e-06 eta: 1:59:17 time: 2.6989 data_time: 0.0110 memory: 11389 loss: 0.1561 2024/07/25 02:17:01 - mmengine - INFO - Iter(train) [16470/19224] lr: 1.0579e-06 eta: 1:58:52 time: 2.6554 data_time: 0.0107 memory: 11274 loss: 0.1608 2024/07/25 02:17:27 - mmengine - INFO - Iter(train) [16480/19224] lr: 1.0504e-06 eta: 1:58:26 time: 2.5701 data_time: 0.0106 memory: 11205 loss: 0.1529 2024/07/25 02:17:52 - mmengine - INFO - Iter(train) [16490/19224] lr: 1.0429e-06 eta: 1:58:00 time: 2.4971 data_time: 0.0104 memory: 11054 loss: 0.1505 2024/07/25 02:18:16 - mmengine - INFO - Iter(train) [16500/19224] lr: 1.0354e-06 eta: 1:57:33 time: 2.4112 data_time: 0.0113 memory: 10905 loss: 0.1307 2024/07/25 02:18:36 - mmengine - INFO - Iter(train) [16510/19224] lr: 1.0279e-06 eta: 1:57:06 time: 1.9664 data_time: 0.0096 memory: 10598 loss: 0.1713 2024/07/25 02:18:56 - mmengine - INFO - Iter(train) [16520/19224] lr: 1.0205e-06 eta: 1:56:40 time: 2.0192 data_time: 0.0094 memory: 14913 loss: 0.1588 2024/07/25 02:19:28 - mmengine - INFO - Iter(train) [16530/19224] lr: 1.0131e-06 eta: 1:56:15 time: 3.1909 data_time: 0.0113 memory: 12745 loss: 0.1591 2024/07/25 02:19:59 - mmengine - INFO - Iter(train) [16540/19224] lr: 1.0057e-06 eta: 1:55:50 time: 3.0698 data_time: 0.0112 memory: 11910 loss: 0.1393 2024/07/25 02:20:28 - mmengine - INFO - Iter(train) [16550/19224] lr: 9.9837e-07 eta: 1:55:24 time: 2.9412 data_time: 0.0106 memory: 11631 loss: 0.1609 2024/07/25 02:20:56 - mmengine - INFO - Iter(train) [16560/19224] lr: 9.9104e-07 eta: 1:54:59 time: 2.7814 data_time: 0.0107 memory: 11353 loss: 0.1581 2024/07/25 02:21:23 - mmengine - INFO - Iter(train) [16570/19224] lr: 9.8374e-07 eta: 1:54:33 time: 2.7022 data_time: 0.0105 memory: 11301 loss: 0.1556 2024/07/25 02:21:48 - mmengine - INFO - Iter(train) [16580/19224] lr: 9.7647e-07 eta: 1:54:07 time: 2.5323 data_time: 0.0107 memory: 11179 loss: 0.1530 2024/07/25 02:22:13 - mmengine - INFO - Iter(train) [16590/19224] lr: 9.6922e-07 eta: 1:53:41 time: 2.4922 data_time: 0.0106 memory: 10955 loss: 0.1465 2024/07/25 02:22:35 - mmengine - INFO - Iter(train) [16600/19224] lr: 9.6200e-07 eta: 1:53:15 time: 2.2235 data_time: 0.0101 memory: 10709 loss: 0.1745 2024/07/25 02:22:54 - mmengine - INFO - Iter(train) [16610/19224] lr: 9.5480e-07 eta: 1:52:47 time: 1.8455 data_time: 0.0097 memory: 10366 loss: 0.1698 2024/07/25 02:23:09 - mmengine - INFO - Iter(train) [16620/19224] lr: 9.4763e-07 eta: 1:52:20 time: 1.5658 data_time: 0.0087 memory: 12923 loss: 0.1212 2024/07/25 02:23:41 - mmengine - INFO - Iter(train) [16630/19224] lr: 9.4049e-07 eta: 1:51:55 time: 3.1817 data_time: 0.0106 memory: 12661 loss: 0.1407 2024/07/25 02:24:11 - mmengine - INFO - Iter(train) [16640/19224] lr: 9.3337e-07 eta: 1:51:30 time: 2.9400 data_time: 0.0108 memory: 11786 loss: 0.1347 2024/07/25 02:24:40 - mmengine - INFO - Iter(train) [16650/19224] lr: 9.2627e-07 eta: 1:51:04 time: 2.8869 data_time: 0.0106 memory: 11676 loss: 0.1483 2024/07/25 02:25:07 - mmengine - INFO - Iter(train) [16660/19224] lr: 9.1920e-07 eta: 1:50:39 time: 2.7770 data_time: 0.0106 memory: 11415 loss: 0.1711 2024/07/25 02:25:34 - mmengine - INFO - Iter(train) [16670/19224] lr: 9.1216e-07 eta: 1:50:13 time: 2.6276 data_time: 0.0127 memory: 11282 loss: 0.1577 2024/07/25 02:26:00 - mmengine - INFO - Iter(train) [16680/19224] lr: 9.0514e-07 eta: 1:49:47 time: 2.6024 data_time: 0.0107 memory: 11205 loss: 0.1328 2024/07/25 02:26:22 - mmengine - INFO - Iter(train) [16690/19224] lr: 8.9815e-07 eta: 1:49:21 time: 2.2673 data_time: 0.0108 memory: 10895 loss: 0.1702 2024/07/25 02:26:42 - mmengine - INFO - Iter(train) [16700/19224] lr: 8.9119e-07 eta: 1:48:54 time: 1.9684 data_time: 0.0098 memory: 10339 loss: 0.1320 2024/07/25 02:27:00 - mmengine - INFO - Iter(train) [16710/19224] lr: 8.8425e-07 eta: 1:48:27 time: 1.8531 data_time: 0.0092 memory: 10115 loss: 0.1783 2024/07/25 02:27:23 - mmengine - INFO - Iter(train) [16720/19224] lr: 8.7734e-07 eta: 1:48:00 time: 2.2252 data_time: 0.0091 memory: 18895 loss: 0.1583 2024/07/25 02:27:58 - mmengine - INFO - Iter(train) [16730/19224] lr: 8.7045e-07 eta: 1:47:36 time: 3.5076 data_time: 0.0113 memory: 13632 loss: 0.1519 2024/07/25 02:28:28 - mmengine - INFO - Iter(train) [16740/19224] lr: 8.6359e-07 eta: 1:47:10 time: 2.9811 data_time: 0.0105 memory: 12063 loss: 0.1552 2024/07/25 02:28:56 - mmengine - INFO - Iter(train) [16750/19224] lr: 8.5675e-07 eta: 1:46:45 time: 2.8388 data_time: 0.0106 memory: 11679 loss: 0.1467 2024/07/25 02:29:24 - mmengine - INFO - Iter(train) [16760/19224] lr: 8.4994e-07 eta: 1:46:19 time: 2.7530 data_time: 0.0105 memory: 11484 loss: 0.1558 2024/07/25 02:29:50 - mmengine - INFO - Iter(train) [16770/19224] lr: 8.4316e-07 eta: 1:45:53 time: 2.6323 data_time: 0.0110 memory: 11314 loss: 0.1645 2024/07/25 02:30:18 - mmengine - INFO - Iter(train) [16780/19224] lr: 8.3640e-07 eta: 1:45:28 time: 2.8018 data_time: 0.0104 memory: 11127 loss: 0.1533 2024/07/25 02:30:42 - mmengine - INFO - Iter(train) [16790/19224] lr: 8.2967e-07 eta: 1:45:02 time: 2.4493 data_time: 0.0106 memory: 11005 loss: 0.2300 2024/07/25 02:31:04 - mmengine - INFO - Iter(train) [16800/19224] lr: 8.2296e-07 eta: 1:44:35 time: 2.1258 data_time: 0.0095 memory: 10542 loss: 0.1665 2024/07/25 02:31:22 - mmengine - INFO - Iter(train) [16810/19224] lr: 8.1628e-07 eta: 1:44:08 time: 1.8011 data_time: 0.0092 memory: 10271 loss: 0.1458 2024/07/25 02:31:41 - mmengine - INFO - Iter(train) [16820/19224] lr: 8.0963e-07 eta: 1:43:41 time: 1.9501 data_time: 0.0087 memory: 15197 loss: 0.1579 2024/07/25 02:32:14 - mmengine - INFO - Iter(train) [16830/19224] lr: 8.0300e-07 eta: 1:43:17 time: 3.2919 data_time: 0.0110 memory: 13542 loss: 0.1516 2024/07/25 02:32:44 - mmengine - INFO - Iter(train) [16840/19224] lr: 7.9640e-07 eta: 1:42:51 time: 2.9788 data_time: 0.0111 memory: 12127 loss: 0.1504 2024/07/25 02:33:12 - mmengine - INFO - Iter(train) [16850/19224] lr: 7.8983e-07 eta: 1:42:26 time: 2.8332 data_time: 0.0109 memory: 11638 loss: 0.1498 2024/07/25 02:33:40 - mmengine - INFO - Iter(train) [16860/19224] lr: 7.8328e-07 eta: 1:42:00 time: 2.7567 data_time: 0.0114 memory: 11386 loss: 0.1516 2024/07/25 02:34:07 - mmengine - INFO - Iter(train) [16870/19224] lr: 7.7675e-07 eta: 1:41:34 time: 2.7423 data_time: 0.0106 memory: 11256 loss: 0.1459 2024/07/25 02:34:33 - mmengine - INFO - Iter(train) [16880/19224] lr: 7.7026e-07 eta: 1:41:08 time: 2.5772 data_time: 0.0104 memory: 11163 loss: 0.1423 2024/07/25 02:34:57 - mmengine - INFO - Iter(train) [16890/19224] lr: 7.6379e-07 eta: 1:40:42 time: 2.4527 data_time: 0.0108 memory: 11008 loss: 0.1701 2024/07/25 02:35:20 - mmengine - INFO - Iter(train) [16900/19224] lr: 7.5734e-07 eta: 1:40:16 time: 2.2447 data_time: 0.0111 memory: 10801 loss: 0.1303 2024/07/25 02:35:39 - mmengine - INFO - Iter(train) [16910/19224] lr: 7.5092e-07 eta: 1:39:49 time: 1.9073 data_time: 0.0112 memory: 10267 loss: 0.1399 2024/07/25 02:35:58 - mmengine - INFO - Iter(train) [16920/19224] lr: 7.4453e-07 eta: 1:39:22 time: 1.9463 data_time: 0.0089 memory: 13808 loss: 0.1542 2024/07/25 02:36:31 - mmengine - INFO - Iter(train) [16930/19224] lr: 7.3817e-07 eta: 1:38:57 time: 3.2710 data_time: 0.0108 memory: 13328 loss: 0.1575 2024/07/25 02:37:00 - mmengine - INFO - Iter(train) [16940/19224] lr: 7.3183e-07 eta: 1:38:32 time: 2.9222 data_time: 0.0102 memory: 12085 loss: 0.1447 2024/07/25 02:37:28 - mmengine - INFO - Iter(train) [16950/19224] lr: 7.2551e-07 eta: 1:38:06 time: 2.7871 data_time: 0.0106 memory: 11715 loss: 0.1638 2024/07/25 02:37:56 - mmengine - INFO - Iter(train) [16960/19224] lr: 7.1923e-07 eta: 1:37:41 time: 2.7425 data_time: 0.0105 memory: 11446 loss: 0.1509 2024/07/25 02:38:22 - mmengine - INFO - Iter(train) [16970/19224] lr: 7.1297e-07 eta: 1:37:15 time: 2.6652 data_time: 0.0107 memory: 11295 loss: 0.1674 2024/07/25 02:38:47 - mmengine - INFO - Iter(train) [16980/19224] lr: 7.0673e-07 eta: 1:36:49 time: 2.5063 data_time: 0.0105 memory: 11154 loss: 0.1661 2024/07/25 02:39:12 - mmengine - INFO - Iter(train) [16990/19224] lr: 7.0052e-07 eta: 1:36:23 time: 2.4103 data_time: 0.0108 memory: 11024 loss: 0.1843 2024/07/25 02:39:33 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/25 02:39:33 - mmengine - INFO - Iter(train) [17000/19224] lr: 6.9434e-07 eta: 1:35:56 time: 2.1599 data_time: 0.0098 memory: 10688 loss: 0.1494 2024/07/25 02:39:33 - mmengine - INFO - Saving checkpoint at 17000 iterations 2024/07/25 02:39:53 - mmengine - INFO - Iter(train) [17010/19224] lr: 6.8819e-07 eta: 1:35:30 time: 2.0314 data_time: 0.1924 memory: 10190 loss: 0.1645 2024/07/25 02:40:11 - mmengine - INFO - Iter(train) [17020/19224] lr: 6.8206e-07 eta: 1:35:03 time: 1.7111 data_time: 0.0094 memory: 13121 loss: 0.1884 2024/07/25 02:40:44 - mmengine - INFO - Iter(train) [17030/19224] lr: 6.7596e-07 eta: 1:34:38 time: 3.3380 data_time: 0.0107 memory: 12887 loss: 0.1653 2024/07/25 02:41:14 - mmengine - INFO - Iter(train) [17040/19224] lr: 6.6988e-07 eta: 1:34:12 time: 3.0021 data_time: 0.0106 memory: 12125 loss: 0.1382 2024/07/25 02:41:43 - mmengine - INFO - Iter(train) [17050/19224] lr: 6.6383e-07 eta: 1:33:47 time: 2.9330 data_time: 0.0113 memory: 11945 loss: 0.1709 2024/07/25 02:42:12 - mmengine - INFO - Iter(train) [17060/19224] lr: 6.5781e-07 eta: 1:33:21 time: 2.8293 data_time: 0.0106 memory: 11598 loss: 0.1568 2024/07/25 02:42:40 - mmengine - INFO - Iter(train) [17070/19224] lr: 6.5182e-07 eta: 1:32:56 time: 2.8143 data_time: 0.0106 memory: 11956 loss: 0.1438 2024/07/25 02:43:06 - mmengine - INFO - Iter(train) [17080/19224] lr: 6.4585e-07 eta: 1:32:30 time: 2.6327 data_time: 0.0104 memory: 11220 loss: 0.1449 2024/07/25 02:43:30 - mmengine - INFO - Iter(train) [17090/19224] lr: 6.3990e-07 eta: 1:32:04 time: 2.4083 data_time: 0.0106 memory: 10910 loss: 0.1575 2024/07/25 02:43:50 - mmengine - INFO - Iter(train) [17100/19224] lr: 6.3399e-07 eta: 1:31:37 time: 2.0196 data_time: 0.0096 memory: 10511 loss: 0.1728 2024/07/25 02:44:07 - mmengine - INFO - Iter(train) [17110/19224] lr: 6.2810e-07 eta: 1:31:10 time: 1.7154 data_time: 0.0098 memory: 10064 loss: 0.1679 2024/07/25 02:44:25 - mmengine - INFO - Iter(train) [17120/19224] lr: 6.2223e-07 eta: 1:30:43 time: 1.7288 data_time: 0.0086 memory: 13706 loss: 0.1563 2024/07/25 02:44:57 - mmengine - INFO - Iter(train) [17130/19224] lr: 6.1640e-07 eta: 1:30:18 time: 3.2499 data_time: 0.0107 memory: 12915 loss: 0.1487 2024/07/25 02:45:27 - mmengine - INFO - Iter(train) [17140/19224] lr: 6.1059e-07 eta: 1:29:53 time: 2.9273 data_time: 0.0107 memory: 11849 loss: 0.1590 2024/07/25 02:45:54 - mmengine - INFO - Iter(train) [17150/19224] lr: 6.0480e-07 eta: 1:29:27 time: 2.7629 data_time: 0.0109 memory: 11680 loss: 0.1430 2024/07/25 02:46:21 - mmengine - INFO - Iter(train) [17160/19224] lr: 5.9905e-07 eta: 1:29:01 time: 2.7007 data_time: 0.0108 memory: 11351 loss: 0.1407 2024/07/25 02:46:47 - mmengine - INFO - Iter(train) [17170/19224] lr: 5.9332e-07 eta: 1:28:36 time: 2.5704 data_time: 0.0107 memory: 11227 loss: 0.1467 2024/07/25 02:47:13 - mmengine - INFO - Iter(train) [17180/19224] lr: 5.8761e-07 eta: 1:28:10 time: 2.6079 data_time: 0.0106 memory: 11163 loss: 0.1355 2024/07/25 02:47:37 - mmengine - INFO - Iter(train) [17190/19224] lr: 5.8194e-07 eta: 1:27:44 time: 2.4271 data_time: 0.0105 memory: 11027 loss: 0.1506 2024/07/25 02:47:59 - mmengine - INFO - Iter(train) [17200/19224] lr: 5.7629e-07 eta: 1:27:17 time: 2.1492 data_time: 0.0102 memory: 10692 loss: 0.1764 2024/07/25 02:48:16 - mmengine - INFO - Iter(train) [17210/19224] lr: 5.7066e-07 eta: 1:26:50 time: 1.7348 data_time: 0.0094 memory: 10044 loss: 0.1730 2024/07/25 02:48:34 - mmengine - INFO - Iter(train) [17220/19224] lr: 5.6507e-07 eta: 1:26:24 time: 1.8370 data_time: 0.0091 memory: 14619 loss: 0.1749 2024/07/25 02:49:07 - mmengine - INFO - Iter(train) [17230/19224] lr: 5.5950e-07 eta: 1:25:58 time: 3.2365 data_time: 0.0109 memory: 13120 loss: 0.1356 2024/07/25 02:49:36 - mmengine - INFO - Iter(train) [17240/19224] lr: 5.5396e-07 eta: 1:25:33 time: 2.9233 data_time: 0.0104 memory: 11816 loss: 0.1544 2024/07/25 02:50:05 - mmengine - INFO - Iter(train) [17250/19224] lr: 5.4844e-07 eta: 1:25:07 time: 2.8587 data_time: 0.0105 memory: 11692 loss: 0.1422 2024/07/25 02:50:33 - mmengine - INFO - Iter(train) [17260/19224] lr: 5.4295e-07 eta: 1:24:42 time: 2.7883 data_time: 0.0115 memory: 11501 loss: 0.1406 2024/07/25 02:51:00 - mmengine - INFO - Iter(train) [17270/19224] lr: 5.3749e-07 eta: 1:24:16 time: 2.7125 data_time: 0.0109 memory: 11365 loss: 0.1397 2024/07/25 02:51:25 - mmengine - INFO - Iter(train) [17280/19224] lr: 5.3205e-07 eta: 1:23:50 time: 2.5716 data_time: 0.0109 memory: 11178 loss: 0.1743 2024/07/25 02:51:49 - mmengine - INFO - Iter(train) [17290/19224] lr: 5.2664e-07 eta: 1:23:24 time: 2.3931 data_time: 0.0106 memory: 10945 loss: 0.1511 2024/07/25 02:52:09 - mmengine - INFO - Iter(train) [17300/19224] lr: 5.2126e-07 eta: 1:22:58 time: 2.0024 data_time: 0.0100 memory: 10525 loss: 0.1688 2024/07/25 02:52:26 - mmengine - INFO - Iter(train) [17310/19224] lr: 5.1591e-07 eta: 1:22:31 time: 1.7050 data_time: 0.0096 memory: 10061 loss: 0.1565 2024/07/25 02:52:45 - mmengine - INFO - Iter(train) [17320/19224] lr: 5.1058e-07 eta: 1:22:04 time: 1.8543 data_time: 0.0092 memory: 16719 loss: 0.1554 2024/07/25 02:53:17 - mmengine - INFO - Iter(train) [17330/19224] lr: 5.0528e-07 eta: 1:21:39 time: 3.2106 data_time: 0.0111 memory: 12688 loss: 0.1397 2024/07/25 02:53:47 - mmengine - INFO - Iter(train) [17340/19224] lr: 5.0001e-07 eta: 1:21:13 time: 2.9845 data_time: 0.0107 memory: 11894 loss: 0.1699 2024/07/25 02:54:15 - mmengine - INFO - Iter(train) [17350/19224] lr: 4.9476e-07 eta: 1:20:48 time: 2.8356 data_time: 0.0109 memory: 11825 loss: 0.1510 2024/07/25 02:54:42 - mmengine - INFO - Iter(train) [17360/19224] lr: 4.8954e-07 eta: 1:20:22 time: 2.6730 data_time: 0.0105 memory: 11484 loss: 0.1615 2024/07/25 02:55:09 - mmengine - INFO - Iter(train) [17370/19224] lr: 4.8435e-07 eta: 1:19:56 time: 2.6934 data_time: 0.0107 memory: 11253 loss: 0.1518 2024/07/25 02:55:34 - mmengine - INFO - Iter(train) [17380/19224] lr: 4.7918e-07 eta: 1:19:30 time: 2.5320 data_time: 0.0105 memory: 11121 loss: 0.1548 2024/07/25 02:55:59 - mmengine - INFO - Iter(train) [17390/19224] lr: 4.7404e-07 eta: 1:19:04 time: 2.4675 data_time: 0.0106 memory: 11028 loss: 0.1517 2024/07/25 02:56:21 - mmengine - INFO - Iter(train) [17400/19224] lr: 4.6893e-07 eta: 1:18:38 time: 2.1969 data_time: 0.0100 memory: 10769 loss: 0.1778 2024/07/25 02:56:41 - mmengine - INFO - Iter(train) [17410/19224] lr: 4.6384e-07 eta: 1:18:12 time: 2.0625 data_time: 0.0096 memory: 10454 loss: 0.1708 2024/07/25 02:57:03 - mmengine - INFO - Iter(train) [17420/19224] lr: 4.5879e-07 eta: 1:17:45 time: 2.1092 data_time: 0.0091 memory: 16069 loss: 0.1743 2024/07/25 02:57:35 - mmengine - INFO - Iter(train) [17430/19224] lr: 4.5376e-07 eta: 1:17:20 time: 3.1958 data_time: 0.0111 memory: 12807 loss: 0.1672 2024/07/25 02:58:04 - mmengine - INFO - Iter(train) [17440/19224] lr: 4.4875e-07 eta: 1:16:55 time: 2.9701 data_time: 0.0109 memory: 11935 loss: 0.1469 2024/07/25 02:58:33 - mmengine - INFO - Iter(train) [17450/19224] lr: 4.4378e-07 eta: 1:16:29 time: 2.8298 data_time: 0.0114 memory: 11629 loss: 0.1646 2024/07/25 02:58:59 - mmengine - INFO - Iter(train) [17460/19224] lr: 4.3883e-07 eta: 1:16:03 time: 2.6947 data_time: 0.0111 memory: 11327 loss: 0.1341 2024/07/25 02:59:25 - mmengine - INFO - Iter(train) [17470/19224] lr: 4.3390e-07 eta: 1:15:37 time: 2.5602 data_time: 0.0106 memory: 11173 loss: 0.1548 2024/07/25 02:59:50 - mmengine - INFO - Iter(train) [17480/19224] lr: 4.2901e-07 eta: 1:15:11 time: 2.4540 data_time: 0.0105 memory: 11024 loss: 0.1736 2024/07/25 03:00:16 - mmengine - INFO - Iter(train) [17490/19224] lr: 4.2414e-07 eta: 1:14:45 time: 2.6467 data_time: 0.0107 memory: 10913 loss: 0.1754 2024/07/25 03:00:38 - mmengine - INFO - Iter(train) [17500/19224] lr: 4.1930e-07 eta: 1:14:19 time: 2.1494 data_time: 0.0102 memory: 10689 loss: 0.1296 2024/07/25 03:00:55 - mmengine - INFO - Iter(train) [17510/19224] lr: 4.1449e-07 eta: 1:13:52 time: 1.7700 data_time: 0.0092 memory: 10203 loss: 0.1730 2024/07/25 03:01:15 - mmengine - INFO - Iter(train) [17520/19224] lr: 4.0970e-07 eta: 1:13:26 time: 1.9826 data_time: 0.0087 memory: 19084 loss: 0.1523 2024/07/25 03:01:48 - mmengine - INFO - Iter(train) [17530/19224] lr: 4.0494e-07 eta: 1:13:01 time: 3.2491 data_time: 0.0105 memory: 13492 loss: 0.1436 2024/07/25 03:02:16 - mmengine - INFO - Iter(train) [17540/19224] lr: 4.0021e-07 eta: 1:12:35 time: 2.8836 data_time: 0.0113 memory: 12185 loss: 0.1500 2024/07/25 03:02:43 - mmengine - INFO - Iter(train) [17550/19224] lr: 3.9550e-07 eta: 1:12:09 time: 2.6673 data_time: 0.0114 memory: 11548 loss: 0.1500 2024/07/25 03:03:10 - mmengine - INFO - Iter(train) [17560/19224] lr: 3.9083e-07 eta: 1:11:44 time: 2.6537 data_time: 0.0111 memory: 11361 loss: 0.1657 2024/07/25 03:03:35 - mmengine - INFO - Iter(train) [17570/19224] lr: 3.8618e-07 eta: 1:11:18 time: 2.5775 data_time: 0.0107 memory: 11233 loss: 0.1499 2024/07/25 03:04:00 - mmengine - INFO - Iter(train) [17580/19224] lr: 3.8155e-07 eta: 1:10:52 time: 2.4795 data_time: 0.0115 memory: 11115 loss: 0.1394 2024/07/25 03:04:25 - mmengine - INFO - Iter(train) [17590/19224] lr: 3.7696e-07 eta: 1:10:26 time: 2.4302 data_time: 0.0116 memory: 11001 loss: 0.1478 2024/07/25 03:04:46 - mmengine - INFO - Iter(train) [17600/19224] lr: 3.7239e-07 eta: 1:09:59 time: 2.1153 data_time: 0.0098 memory: 10663 loss: 0.1206 2024/07/25 03:05:04 - mmengine - INFO - Iter(train) [17610/19224] lr: 3.6785e-07 eta: 1:09:33 time: 1.8149 data_time: 0.0095 memory: 10204 loss: 0.1511 2024/07/25 03:05:19 - mmengine - INFO - Iter(train) [17620/19224] lr: 3.6334e-07 eta: 1:09:06 time: 1.5538 data_time: 0.0091 memory: 14660 loss: 0.1649 2024/07/25 03:05:50 - mmengine - INFO - Iter(train) [17630/19224] lr: 3.5885e-07 eta: 1:08:41 time: 3.0671 data_time: 0.0107 memory: 12204 loss: 0.1622 2024/07/25 03:06:20 - mmengine - INFO - Iter(train) [17640/19224] lr: 3.5439e-07 eta: 1:08:15 time: 2.9584 data_time: 0.0105 memory: 11910 loss: 0.1419 2024/07/25 03:06:48 - mmengine - INFO - Iter(train) [17650/19224] lr: 3.4996e-07 eta: 1:07:50 time: 2.8524 data_time: 0.0106 memory: 11875 loss: 0.1568 2024/07/25 03:07:16 - mmengine - INFO - Iter(train) [17660/19224] lr: 3.4555e-07 eta: 1:07:24 time: 2.7874 data_time: 0.0107 memory: 11508 loss: 0.1320 2024/07/25 03:07:43 - mmengine - INFO - Iter(train) [17670/19224] lr: 3.4118e-07 eta: 1:06:58 time: 2.7448 data_time: 0.0110 memory: 11324 loss: 0.1746 2024/07/25 03:08:09 - mmengine - INFO - Iter(train) [17680/19224] lr: 3.3683e-07 eta: 1:06:32 time: 2.5909 data_time: 0.0114 memory: 11183 loss: 0.1466 2024/07/25 03:08:33 - mmengine - INFO - Iter(train) [17690/19224] lr: 3.3251e-07 eta: 1:06:06 time: 2.4094 data_time: 0.0104 memory: 11068 loss: 0.1797 2024/07/25 03:08:54 - mmengine - INFO - Iter(train) [17700/19224] lr: 3.2821e-07 eta: 1:05:40 time: 2.0835 data_time: 0.0097 memory: 10614 loss: 0.1650 2024/07/25 03:09:12 - mmengine - INFO - Iter(train) [17710/19224] lr: 3.2395e-07 eta: 1:05:13 time: 1.8170 data_time: 0.0094 memory: 10302 loss: 0.1466 2024/07/25 03:09:31 - mmengine - INFO - Iter(train) [17720/19224] lr: 3.1971e-07 eta: 1:04:47 time: 1.8954 data_time: 0.0087 memory: 14669 loss: 0.1623 2024/07/25 03:10:05 - mmengine - INFO - Iter(train) [17730/19224] lr: 3.1549e-07 eta: 1:04:22 time: 3.3868 data_time: 0.0106 memory: 13507 loss: 0.1603 2024/07/25 03:10:36 - mmengine - INFO - Iter(train) [17740/19224] lr: 3.1131e-07 eta: 1:03:56 time: 3.0409 data_time: 0.0113 memory: 12148 loss: 0.1482 2024/07/25 03:11:04 - mmengine - INFO - Iter(train) [17750/19224] lr: 3.0715e-07 eta: 1:03:31 time: 2.8390 data_time: 0.0113 memory: 11718 loss: 0.1498 2024/07/25 03:11:31 - mmengine - INFO - Iter(train) [17760/19224] lr: 3.0302e-07 eta: 1:03:05 time: 2.7360 data_time: 0.0107 memory: 11593 loss: 0.1551 2024/07/25 03:11:57 - mmengine - INFO - Iter(train) [17770/19224] lr: 2.9892e-07 eta: 1:02:39 time: 2.5519 data_time: 0.0105 memory: 11285 loss: 0.1514 2024/07/25 03:12:23 - mmengine - INFO - Iter(train) [17780/19224] lr: 2.9485e-07 eta: 1:02:13 time: 2.5570 data_time: 0.0104 memory: 11177 loss: 0.1445 2024/07/25 03:12:47 - mmengine - INFO - Iter(train) [17790/19224] lr: 2.9080e-07 eta: 1:01:47 time: 2.4306 data_time: 0.0105 memory: 10996 loss: 0.1594 2024/07/25 03:13:08 - mmengine - INFO - Iter(train) [17800/19224] lr: 2.8678e-07 eta: 1:01:21 time: 2.0913 data_time: 0.0100 memory: 10646 loss: 0.1336 2024/07/25 03:13:26 - mmengine - INFO - Iter(train) [17810/19224] lr: 2.8279e-07 eta: 1:00:55 time: 1.8464 data_time: 0.0092 memory: 10323 loss: 0.1651 2024/07/25 03:13:43 - mmengine - INFO - Iter(train) [17820/19224] lr: 2.7882e-07 eta: 1:00:28 time: 1.7188 data_time: 0.0085 memory: 13200 loss: 0.1681 2024/07/25 03:14:15 - mmengine - INFO - Iter(train) [17830/19224] lr: 2.7489e-07 eta: 1:00:03 time: 3.1861 data_time: 0.0116 memory: 12516 loss: 0.1459 2024/07/25 03:14:46 - mmengine - INFO - Iter(train) [17840/19224] lr: 2.7098e-07 eta: 0:59:37 time: 3.0528 data_time: 0.0109 memory: 12043 loss: 0.1748 2024/07/25 03:15:14 - mmengine - INFO - Iter(train) [17850/19224] lr: 2.6710e-07 eta: 0:59:12 time: 2.8680 data_time: 0.0109 memory: 11912 loss: 0.1471 2024/07/25 03:15:43 - mmengine - INFO - Iter(train) [17860/19224] lr: 2.6324e-07 eta: 0:58:46 time: 2.8699 data_time: 0.0109 memory: 11670 loss: 0.1623 2024/07/25 03:16:10 - mmengine - INFO - Iter(train) [17870/19224] lr: 2.5942e-07 eta: 0:58:20 time: 2.7200 data_time: 0.0107 memory: 11550 loss: 0.1596 2024/07/25 03:16:37 - mmengine - INFO - Iter(train) [17880/19224] lr: 2.5562e-07 eta: 0:57:54 time: 2.6844 data_time: 0.0105 memory: 11297 loss: 0.1513 2024/07/25 03:17:02 - mmengine - INFO - Iter(train) [17890/19224] lr: 2.5185e-07 eta: 0:57:28 time: 2.4726 data_time: 0.0112 memory: 11154 loss: 0.1564 2024/07/25 03:17:25 - mmengine - INFO - Iter(train) [17900/19224] lr: 2.4810e-07 eta: 0:57:02 time: 2.2868 data_time: 0.0106 memory: 10958 loss: 0.1767 2024/07/25 03:17:43 - mmengine - INFO - Iter(train) [17910/19224] lr: 2.4439e-07 eta: 0:56:36 time: 1.8658 data_time: 0.0095 memory: 10307 loss: 0.1538 2024/07/25 03:18:04 - mmengine - INFO - Iter(train) [17920/19224] lr: 2.4070e-07 eta: 0:56:10 time: 2.0413 data_time: 0.0090 memory: 15347 loss: 0.1536 2024/07/25 03:18:36 - mmengine - INFO - Iter(train) [17930/19224] lr: 2.3704e-07 eta: 0:55:44 time: 3.1955 data_time: 0.0107 memory: 12838 loss: 0.1541 2024/07/25 03:19:05 - mmengine - INFO - Iter(train) [17940/19224] lr: 2.3341e-07 eta: 0:55:19 time: 2.8775 data_time: 0.0106 memory: 11989 loss: 0.1528 2024/07/25 03:19:33 - mmengine - INFO - Iter(train) [17950/19224] lr: 2.2980e-07 eta: 0:54:53 time: 2.8350 data_time: 0.0107 memory: 11542 loss: 0.1495 2024/07/25 03:20:00 - mmengine - INFO - Iter(train) [17960/19224] lr: 2.2623e-07 eta: 0:54:27 time: 2.7436 data_time: 0.0113 memory: 11393 loss: 0.1587 2024/07/25 03:20:27 - mmengine - INFO - Iter(train) [17970/19224] lr: 2.2268e-07 eta: 0:54:02 time: 2.6632 data_time: 0.0107 memory: 11279 loss: 0.1640 2024/07/25 03:20:52 - mmengine - INFO - Iter(train) [17980/19224] lr: 2.1915e-07 eta: 0:53:36 time: 2.5281 data_time: 0.0113 memory: 11097 loss: 0.1544 2024/07/25 03:21:16 - mmengine - INFO - Iter(train) [17990/19224] lr: 2.1566e-07 eta: 0:53:10 time: 2.3900 data_time: 0.0119 memory: 11009 loss: 0.1606 2024/07/25 03:21:37 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/25 03:21:37 - mmengine - INFO - Iter(train) [18000/19224] lr: 2.1220e-07 eta: 0:52:43 time: 2.1133 data_time: 0.0097 memory: 10642 loss: 0.1572 2024/07/25 03:21:37 - mmengine - INFO - Saving checkpoint at 18000 iterations 2024/07/25 03:21:58 - mmengine - INFO - Iter(train) [18010/19224] lr: 2.0876e-07 eta: 0:52:17 time: 2.0593 data_time: 0.1874 memory: 10302 loss: 0.1419 2024/07/25 03:22:15 - mmengine - INFO - Iter(train) [18020/19224] lr: 2.0535e-07 eta: 0:51:51 time: 1.7141 data_time: 0.0088 memory: 15543 loss: 0.2187 2024/07/25 03:22:49 - mmengine - INFO - Iter(train) [18030/19224] lr: 2.0196e-07 eta: 0:51:25 time: 3.3725 data_time: 0.0113 memory: 13107 loss: 0.1530 2024/07/25 03:23:20 - mmengine - INFO - Iter(train) [18040/19224] lr: 1.9861e-07 eta: 0:51:00 time: 3.1091 data_time: 0.0115 memory: 12246 loss: 0.1476 2024/07/25 03:23:49 - mmengine - INFO - Iter(train) [18050/19224] lr: 1.9528e-07 eta: 0:50:34 time: 2.8998 data_time: 0.0110 memory: 11811 loss: 0.1395 2024/07/25 03:24:17 - mmengine - INFO - Iter(train) [18060/19224] lr: 1.9198e-07 eta: 0:50:09 time: 2.8552 data_time: 0.0113 memory: 11550 loss: 0.1629 2024/07/25 03:24:45 - mmengine - INFO - Iter(train) [18070/19224] lr: 1.8871e-07 eta: 0:49:43 time: 2.7225 data_time: 0.0111 memory: 11368 loss: 0.1488 2024/07/25 03:25:10 - mmengine - INFO - Iter(train) [18080/19224] lr: 1.8547e-07 eta: 0:49:17 time: 2.5403 data_time: 0.0107 memory: 11170 loss: 0.1478 2024/07/25 03:25:34 - mmengine - INFO - Iter(train) [18090/19224] lr: 1.8225e-07 eta: 0:48:51 time: 2.3998 data_time: 0.0109 memory: 10960 loss: 0.1532 2024/07/25 03:25:54 - mmengine - INFO - Iter(train) [18100/19224] lr: 1.7906e-07 eta: 0:48:25 time: 2.0055 data_time: 0.0114 memory: 10617 loss: 0.1705 2024/07/25 03:26:12 - mmengine - INFO - Iter(train) [18110/19224] lr: 1.7590e-07 eta: 0:47:59 time: 1.8328 data_time: 0.0093 memory: 10161 loss: 0.1489 2024/07/25 03:26:34 - mmengine - INFO - Iter(train) [18120/19224] lr: 1.7277e-07 eta: 0:47:32 time: 2.1113 data_time: 0.0093 memory: 15840 loss: 0.1787 2024/07/25 03:27:06 - mmengine - INFO - Iter(train) [18130/19224] lr: 1.6967e-07 eta: 0:47:07 time: 3.2571 data_time: 0.0106 memory: 13601 loss: 0.1492 2024/07/25 03:27:35 - mmengine - INFO - Iter(train) [18140/19224] lr: 1.6659e-07 eta: 0:46:41 time: 2.9089 data_time: 0.0110 memory: 12132 loss: 0.1351 2024/07/25 03:28:04 - mmengine - INFO - Iter(train) [18150/19224] lr: 1.6354e-07 eta: 0:46:16 time: 2.8592 data_time: 0.0111 memory: 11649 loss: 0.1422 2024/07/25 03:28:31 - mmengine - INFO - Iter(train) [18160/19224] lr: 1.6052e-07 eta: 0:45:50 time: 2.6707 data_time: 0.0108 memory: 11424 loss: 0.1523 2024/07/25 03:28:57 - mmengine - INFO - Iter(train) [18170/19224] lr: 1.5753e-07 eta: 0:45:24 time: 2.6296 data_time: 0.0119 memory: 11271 loss: 0.1470 2024/07/25 03:29:22 - mmengine - INFO - Iter(train) [18180/19224] lr: 1.5457e-07 eta: 0:44:58 time: 2.5532 data_time: 0.0112 memory: 11141 loss: 0.1481 2024/07/25 03:29:47 - mmengine - INFO - Iter(train) [18190/19224] lr: 1.5163e-07 eta: 0:44:32 time: 2.4361 data_time: 0.0108 memory: 10965 loss: 0.2242 2024/07/25 03:30:10 - mmengine - INFO - Iter(train) [18200/19224] lr: 1.4872e-07 eta: 0:44:06 time: 2.2850 data_time: 0.0099 memory: 10589 loss: 0.1362 2024/07/25 03:30:26 - mmengine - INFO - Iter(train) [18210/19224] lr: 1.4584e-07 eta: 0:43:40 time: 1.6889 data_time: 0.0094 memory: 10037 loss: 0.1995 2024/07/25 03:30:44 - mmengine - INFO - Iter(train) [18220/19224] lr: 1.4299e-07 eta: 0:43:14 time: 1.7952 data_time: 0.0090 memory: 15677 loss: 0.1782 2024/07/25 03:31:18 - mmengine - INFO - Iter(train) [18230/19224] lr: 1.4016e-07 eta: 0:42:48 time: 3.3455 data_time: 0.0111 memory: 14435 loss: 0.1350 2024/07/25 03:31:48 - mmengine - INFO - Iter(train) [18240/19224] lr: 1.3737e-07 eta: 0:42:23 time: 3.0589 data_time: 0.0112 memory: 12017 loss: 0.1514 2024/07/25 03:32:18 - mmengine - INFO - Iter(train) [18250/19224] lr: 1.3460e-07 eta: 0:41:57 time: 2.9665 data_time: 0.0107 memory: 11820 loss: 0.1310 2024/07/25 03:32:46 - mmengine - INFO - Iter(train) [18260/19224] lr: 1.3186e-07 eta: 0:41:31 time: 2.7364 data_time: 0.0104 memory: 11459 loss: 0.1446 2024/07/25 03:33:12 - mmengine - INFO - Iter(train) [18270/19224] lr: 1.2914e-07 eta: 0:41:05 time: 2.6223 data_time: 0.0108 memory: 11307 loss: 0.1566 2024/07/25 03:33:37 - mmengine - INFO - Iter(train) [18280/19224] lr: 1.2646e-07 eta: 0:40:39 time: 2.5347 data_time: 0.0105 memory: 11111 loss: 0.1534 2024/07/25 03:34:02 - mmengine - INFO - Iter(train) [18290/19224] lr: 1.2380e-07 eta: 0:40:14 time: 2.4487 data_time: 0.0108 memory: 11027 loss: 0.1739 2024/07/25 03:34:24 - mmengine - INFO - Iter(train) [18300/19224] lr: 1.2117e-07 eta: 0:39:48 time: 2.2062 data_time: 0.0107 memory: 10727 loss: 0.1564 2024/07/25 03:34:42 - mmengine - INFO - Iter(train) [18310/19224] lr: 1.1857e-07 eta: 0:39:21 time: 1.7871 data_time: 0.0096 memory: 10143 loss: 0.1688 2024/07/25 03:35:01 - mmengine - INFO - Iter(train) [18320/19224] lr: 1.1600e-07 eta: 0:38:55 time: 1.9110 data_time: 0.0094 memory: 14006 loss: 0.1651 2024/07/25 03:35:35 - mmengine - INFO - Iter(train) [18330/19224] lr: 1.1346e-07 eta: 0:38:30 time: 3.4059 data_time: 0.0115 memory: 13362 loss: 0.1597 2024/07/25 03:36:05 - mmengine - INFO - Iter(train) [18340/19224] lr: 1.1094e-07 eta: 0:38:04 time: 3.0365 data_time: 0.0108 memory: 12210 loss: 0.1317 2024/07/25 03:36:34 - mmengine - INFO - Iter(train) [18350/19224] lr: 1.0845e-07 eta: 0:37:38 time: 2.9384 data_time: 0.0106 memory: 11991 loss: 0.1674 2024/07/25 03:37:02 - mmengine - INFO - Iter(train) [18360/19224] lr: 1.0599e-07 eta: 0:37:13 time: 2.8062 data_time: 0.0105 memory: 11475 loss: 0.1516 2024/07/25 03:37:30 - mmengine - INFO - Iter(train) [18370/19224] lr: 1.0356e-07 eta: 0:36:47 time: 2.7485 data_time: 0.0111 memory: 11389 loss: 0.1492 2024/07/25 03:37:56 - mmengine - INFO - Iter(train) [18380/19224] lr: 1.0115e-07 eta: 0:36:21 time: 2.6454 data_time: 0.0108 memory: 11211 loss: 0.1813 2024/07/25 03:38:22 - mmengine - INFO - Iter(train) [18390/19224] lr: 9.8778e-08 eta: 0:35:55 time: 2.5278 data_time: 0.0110 memory: 11050 loss: 0.1782 2024/07/25 03:38:45 - mmengine - INFO - Iter(train) [18400/19224] lr: 9.6430e-08 eta: 0:35:29 time: 2.3046 data_time: 0.0114 memory: 10975 loss: 0.1934 2024/07/25 03:39:04 - mmengine - INFO - Iter(train) [18410/19224] lr: 9.4110e-08 eta: 0:35:03 time: 1.9004 data_time: 0.0094 memory: 10248 loss: 0.1634 2024/07/25 03:39:24 - mmengine - INFO - Iter(train) [18420/19224] lr: 9.1819e-08 eta: 0:34:37 time: 2.0051 data_time: 0.0090 memory: 13948 loss: 0.1708 2024/07/25 03:39:57 - mmengine - INFO - Iter(train) [18430/19224] lr: 8.9555e-08 eta: 0:34:11 time: 3.2821 data_time: 0.0106 memory: 13225 loss: 0.1688 2024/07/25 03:40:27 - mmengine - INFO - Iter(train) [18440/19224] lr: 8.7320e-08 eta: 0:33:46 time: 3.0108 data_time: 0.0112 memory: 11968 loss: 0.1521 2024/07/25 03:40:55 - mmengine - INFO - Iter(train) [18450/19224] lr: 8.5112e-08 eta: 0:33:20 time: 2.8395 data_time: 0.0108 memory: 11542 loss: 0.1508 2024/07/25 03:41:22 - mmengine - INFO - Iter(train) [18460/19224] lr: 8.2933e-08 eta: 0:32:54 time: 2.6782 data_time: 0.0106 memory: 11275 loss: 0.1559 2024/07/25 03:41:47 - mmengine - INFO - Iter(train) [18470/19224] lr: 8.0782e-08 eta: 0:32:28 time: 2.5450 data_time: 0.0108 memory: 11210 loss: 0.1620 2024/07/25 03:42:12 - mmengine - INFO - Iter(train) [18480/19224] lr: 7.8659e-08 eta: 0:32:02 time: 2.5047 data_time: 0.0105 memory: 11070 loss: 0.1488 2024/07/25 03:42:37 - mmengine - INFO - Iter(train) [18490/19224] lr: 7.6564e-08 eta: 0:31:37 time: 2.4221 data_time: 0.0104 memory: 10918 loss: 0.1537 2024/07/25 03:42:59 - mmengine - INFO - Iter(train) [18500/19224] lr: 7.4497e-08 eta: 0:31:11 time: 2.2404 data_time: 0.0097 memory: 10650 loss: 0.1615 2024/07/25 03:43:18 - mmengine - INFO - Iter(train) [18510/19224] lr: 7.2459e-08 eta: 0:30:44 time: 1.9005 data_time: 0.0093 memory: 10231 loss: 0.1378 2024/07/25 03:43:38 - mmengine - INFO - Iter(train) [18520/19224] lr: 7.0449e-08 eta: 0:30:18 time: 1.9712 data_time: 0.0090 memory: 13967 loss: 0.1567 2024/07/25 03:44:10 - mmengine - INFO - Iter(train) [18530/19224] lr: 6.8467e-08 eta: 0:29:53 time: 3.1739 data_time: 0.0107 memory: 12577 loss: 0.1390 2024/07/25 03:44:40 - mmengine - INFO - Iter(train) [18540/19224] lr: 6.6513e-08 eta: 0:29:27 time: 3.0323 data_time: 0.0109 memory: 12019 loss: 0.1689 2024/07/25 03:45:09 - mmengine - INFO - Iter(train) [18550/19224] lr: 6.4587e-08 eta: 0:29:01 time: 2.9154 data_time: 0.0108 memory: 11718 loss: 0.1653 2024/07/25 03:45:37 - mmengine - INFO - Iter(train) [18560/19224] lr: 6.2689e-08 eta: 0:28:36 time: 2.7897 data_time: 0.0106 memory: 11437 loss: 0.1449 2024/07/25 03:46:04 - mmengine - INFO - Iter(train) [18570/19224] lr: 6.0820e-08 eta: 0:28:10 time: 2.6726 data_time: 0.0106 memory: 11274 loss: 0.1653 2024/07/25 03:46:30 - mmengine - INFO - Iter(train) [18580/19224] lr: 5.8979e-08 eta: 0:27:44 time: 2.6362 data_time: 0.0106 memory: 11198 loss: 0.1454 2024/07/25 03:46:55 - mmengine - INFO - Iter(train) [18590/19224] lr: 5.7166e-08 eta: 0:27:18 time: 2.4859 data_time: 0.0108 memory: 11127 loss: 0.1744 2024/07/25 03:47:17 - mmengine - INFO - Iter(train) [18600/19224] lr: 5.5381e-08 eta: 0:26:52 time: 2.2530 data_time: 0.0101 memory: 10804 loss: 0.1523 2024/07/25 03:47:35 - mmengine - INFO - Iter(train) [18610/19224] lr: 5.3625e-08 eta: 0:26:26 time: 1.7469 data_time: 0.0094 memory: 10130 loss: 0.1450 2024/07/25 03:47:52 - mmengine - INFO - Iter(train) [18620/19224] lr: 5.1897e-08 eta: 0:26:00 time: 1.7126 data_time: 0.0084 memory: 14439 loss: 0.2033 2024/07/25 03:48:25 - mmengine - INFO - Iter(train) [18630/19224] lr: 5.0197e-08 eta: 0:25:34 time: 3.3223 data_time: 0.0117 memory: 13560 loss: 0.1586 2024/07/25 03:48:55 - mmengine - INFO - Iter(train) [18640/19224] lr: 4.8525e-08 eta: 0:25:09 time: 3.0098 data_time: 0.0102 memory: 12008 loss: 0.1559 2024/07/25 03:49:24 - mmengine - INFO - Iter(train) [18650/19224] lr: 4.6881e-08 eta: 0:24:43 time: 2.9147 data_time: 0.0108 memory: 11825 loss: 0.1547 2024/07/25 03:49:52 - mmengine - INFO - Iter(train) [18660/19224] lr: 4.5266e-08 eta: 0:24:17 time: 2.7133 data_time: 0.0104 memory: 11497 loss: 0.1462 2024/07/25 03:50:18 - mmengine - INFO - Iter(train) [18670/19224] lr: 4.3679e-08 eta: 0:23:51 time: 2.6408 data_time: 0.0104 memory: 11318 loss: 0.1468 2024/07/25 03:50:43 - mmengine - INFO - Iter(train) [18680/19224] lr: 4.2120e-08 eta: 0:23:25 time: 2.5203 data_time: 0.0106 memory: 11128 loss: 0.1800 2024/07/25 03:51:07 - mmengine - INFO - Iter(train) [18690/19224] lr: 4.0590e-08 eta: 0:22:59 time: 2.4141 data_time: 0.0111 memory: 11015 loss: 0.1571 2024/07/25 03:51:29 - mmengine - INFO - Iter(train) [18700/19224] lr: 3.9088e-08 eta: 0:22:34 time: 2.2189 data_time: 0.0102 memory: 10741 loss: 0.1450 2024/07/25 03:51:48 - mmengine - INFO - Iter(train) [18710/19224] lr: 3.7614e-08 eta: 0:22:07 time: 1.8687 data_time: 0.0095 memory: 10347 loss: 0.1787 2024/07/25 03:52:09 - mmengine - INFO - Iter(train) [18720/19224] lr: 3.6168e-08 eta: 0:21:42 time: 2.1124 data_time: 0.0091 memory: 18474 loss: 0.1659 2024/07/25 03:52:42 - mmengine - INFO - Iter(train) [18730/19224] lr: 3.4751e-08 eta: 0:21:16 time: 3.2320 data_time: 0.0106 memory: 13301 loss: 0.1585 2024/07/25 03:53:11 - mmengine - INFO - Iter(train) [18740/19224] lr: 3.3362e-08 eta: 0:20:50 time: 2.9573 data_time: 0.0116 memory: 12005 loss: 0.1502 2024/07/25 03:53:40 - mmengine - INFO - Iter(train) [18750/19224] lr: 3.2001e-08 eta: 0:20:24 time: 2.8330 data_time: 0.0103 memory: 11715 loss: 0.1466 2024/07/25 03:54:07 - mmengine - INFO - Iter(train) [18760/19224] lr: 3.0668e-08 eta: 0:19:59 time: 2.7270 data_time: 0.0105 memory: 11499 loss: 0.1716 2024/07/25 03:54:33 - mmengine - INFO - Iter(train) [18770/19224] lr: 2.9364e-08 eta: 0:19:33 time: 2.6667 data_time: 0.0102 memory: 11319 loss: 0.1515 2024/07/25 03:54:59 - mmengine - INFO - Iter(train) [18780/19224] lr: 2.8088e-08 eta: 0:19:07 time: 2.5575 data_time: 0.0104 memory: 11176 loss: 0.1407 2024/07/25 03:55:24 - mmengine - INFO - Iter(train) [18790/19224] lr: 2.6840e-08 eta: 0:18:41 time: 2.4509 data_time: 0.0103 memory: 11014 loss: 0.1748 2024/07/25 03:55:47 - mmengine - INFO - Iter(train) [18800/19224] lr: 2.5621e-08 eta: 0:18:15 time: 2.3016 data_time: 0.0110 memory: 10890 loss: 0.1789 2024/07/25 03:56:06 - mmengine - INFO - Iter(train) [18810/19224] lr: 2.4430e-08 eta: 0:17:49 time: 1.9283 data_time: 0.0097 memory: 10504 loss: 0.1545 2024/07/25 03:56:28 - mmengine - INFO - Iter(train) [18820/19224] lr: 2.3267e-08 eta: 0:17:23 time: 2.2437 data_time: 0.0100 memory: 18443 loss: 0.1734 2024/07/25 03:57:00 - mmengine - INFO - Iter(train) [18830/19224] lr: 2.2133e-08 eta: 0:16:57 time: 3.2073 data_time: 0.0112 memory: 13343 loss: 0.1476 2024/07/25 03:57:29 - mmengine - INFO - Iter(train) [18840/19224] lr: 2.1027e-08 eta: 0:16:32 time: 2.8826 data_time: 0.0106 memory: 11871 loss: 0.1471 2024/07/25 03:57:58 - mmengine - INFO - Iter(train) [18850/19224] lr: 1.9949e-08 eta: 0:16:06 time: 2.8390 data_time: 0.0106 memory: 11751 loss: 0.1416 2024/07/25 03:58:25 - mmengine - INFO - Iter(train) [18860/19224] lr: 1.8900e-08 eta: 0:15:40 time: 2.7293 data_time: 0.0110 memory: 11504 loss: 0.1389 2024/07/25 03:58:52 - mmengine - INFO - Iter(train) [18870/19224] lr: 1.7879e-08 eta: 0:15:14 time: 2.7096 data_time: 0.0106 memory: 11217 loss: 0.1582 2024/07/25 03:59:17 - mmengine - INFO - Iter(train) [18880/19224] lr: 1.6886e-08 eta: 0:14:48 time: 2.5264 data_time: 0.0106 memory: 11043 loss: 0.1514 2024/07/25 03:59:40 - mmengine - INFO - Iter(train) [18890/19224] lr: 1.5921e-08 eta: 0:14:23 time: 2.2279 data_time: 0.0109 memory: 10906 loss: 0.1562 2024/07/25 04:00:00 - mmengine - INFO - Iter(train) [18900/19224] lr: 1.4985e-08 eta: 0:13:57 time: 1.9996 data_time: 0.0095 memory: 10455 loss: 0.1431 2024/07/25 04:00:20 - mmengine - INFO - Iter(train) [18910/19224] lr: 1.4077e-08 eta: 0:13:31 time: 2.0578 data_time: 0.0095 memory: 10259 loss: 0.1751 2024/07/25 04:00:39 - mmengine - INFO - Iter(train) [18920/19224] lr: 1.3198e-08 eta: 0:13:05 time: 1.8801 data_time: 0.0092 memory: 13569 loss: 0.1693 2024/07/25 04:01:10 - mmengine - INFO - Iter(train) [18930/19224] lr: 1.2347e-08 eta: 0:12:39 time: 3.0632 data_time: 0.0110 memory: 12165 loss: 0.1348 2024/07/25 04:01:39 - mmengine - INFO - Iter(train) [18940/19224] lr: 1.1524e-08 eta: 0:12:13 time: 2.9316 data_time: 0.0104 memory: 11844 loss: 0.1315 2024/07/25 04:02:07 - mmengine - INFO - Iter(train) [18950/19224] lr: 1.0730e-08 eta: 0:11:47 time: 2.8325 data_time: 0.0108 memory: 11636 loss: 0.1789 2024/07/25 04:02:35 - mmengine - INFO - Iter(train) [18960/19224] lr: 9.9638e-09 eta: 0:11:22 time: 2.8222 data_time: 0.0108 memory: 11593 loss: 0.1392 2024/07/25 04:03:02 - mmengine - INFO - Iter(train) [18970/19224] lr: 9.2261e-09 eta: 0:10:56 time: 2.6726 data_time: 0.0108 memory: 11321 loss: 0.1468 2024/07/25 04:03:27 - mmengine - INFO - Iter(train) [18980/19224] lr: 8.5168e-09 eta: 0:10:30 time: 2.5023 data_time: 0.0103 memory: 11065 loss: 0.1605 2024/07/25 04:03:51 - mmengine - INFO - Iter(train) [18990/19224] lr: 7.8358e-09 eta: 0:10:04 time: 2.3813 data_time: 0.0105 memory: 10905 loss: 0.1698 2024/07/25 04:04:13 - mmengine - INFO - Exp name: internvl_v2_internlm2_2b_qlora_finetune_copy_20240724_142532 2024/07/25 04:04:13 - mmengine - INFO - Iter(train) [19000/19224] lr: 7.1832e-09 eta: 0:09:38 time: 2.1619 data_time: 0.0102 memory: 10687 loss: 0.2859 2024/07/25 04:04:13 - mmengine - INFO - Saving checkpoint at 19000 iterations 2024/07/25 04:04:34 - mmengine - INFO - Iter(train) [19010/19224] lr: 6.5590e-09 eta: 0:09:12 time: 2.1508 data_time: 0.2102 memory: 10401 loss: 0.1580 2024/07/25 04:04:54 - mmengine - INFO - Iter(train) [19020/19224] lr: 5.9631e-09 eta: 0:08:46 time: 1.9628 data_time: 0.0093 memory: 16358 loss: 0.1590 2024/07/25 04:05:26 - mmengine - INFO - Iter(train) [19030/19224] lr: 5.3955e-09 eta: 0:08:21 time: 3.1808 data_time: 0.0113 memory: 12441 loss: 0.1432 2024/07/25 04:05:56 - mmengine - INFO - Iter(train) [19040/19224] lr: 4.8564e-09 eta: 0:07:55 time: 2.9996 data_time: 0.0105 memory: 12203 loss: 0.1403 2024/07/25 04:06:23 - mmengine - INFO - Iter(train) [19050/19224] lr: 4.3456e-09 eta: 0:07:29 time: 2.7851 data_time: 0.0103 memory: 11524 loss: 0.1542 2024/07/25 04:06:51 - mmengine - INFO - Iter(train) [19060/19224] lr: 3.8632e-09 eta: 0:07:03 time: 2.7558 data_time: 0.0101 memory: 11386 loss: 0.1713 2024/07/25 04:07:17 - mmengine - INFO - Iter(train) [19070/19224] lr: 3.4091e-09 eta: 0:06:37 time: 2.5964 data_time: 0.0107 memory: 11194 loss: 0.1780 2024/07/25 04:07:42 - mmengine - INFO - Iter(train) [19080/19224] lr: 2.9835e-09 eta: 0:06:12 time: 2.5005 data_time: 0.0110 memory: 11199 loss: 0.1741 2024/07/25 04:08:05 - mmengine - INFO - Iter(train) [19090/19224] lr: 2.5862e-09 eta: 0:05:46 time: 2.2649 data_time: 0.0101 memory: 10930 loss: 0.1582 2024/07/25 04:08:25 - mmengine - INFO - Iter(train) [19100/19224] lr: 2.2172e-09 eta: 0:05:20 time: 2.0543 data_time: 0.0094 memory: 10494 loss: 0.1186 2024/07/25 04:08:43 - mmengine - INFO - Iter(train) [19110/19224] lr: 1.8767e-09 eta: 0:04:54 time: 1.8402 data_time: 0.0092 memory: 10225 loss: 0.1481 2024/07/25 04:09:03 - mmengine - INFO - Iter(train) [19120/19224] lr: 1.5645e-09 eta: 0:04:28 time: 1.9339 data_time: 0.0087 memory: 18250 loss: 0.1625 2024/07/25 04:09:34 - mmengine - INFO - Iter(train) [19130/19224] lr: 1.2807e-09 eta: 0:04:02 time: 3.0968 data_time: 0.0110 memory: 12514 loss: 0.1665 2024/07/25 04:10:04 - mmengine - INFO - Iter(train) [19140/19224] lr: 1.0253e-09 eta: 0:03:36 time: 2.9786 data_time: 0.0105 memory: 12005 loss: 0.1517 2024/07/25 04:10:32 - mmengine - INFO - Iter(train) [19150/19224] lr: 7.9822e-10 eta: 0:03:11 time: 2.8906 data_time: 0.0108 memory: 12010 loss: 0.1428 2024/07/25 04:11:00 - mmengine - INFO - Iter(train) [19160/19224] lr: 5.9955e-10 eta: 0:02:45 time: 2.7405 data_time: 0.0108 memory: 11533 loss: 0.1386 2024/07/25 04:11:26 - mmengine - INFO - Iter(train) [19170/19224] lr: 4.2927e-10 eta: 0:02:19 time: 2.6075 data_time: 0.0107 memory: 11234 loss: 0.1859 2024/07/25 04:11:51 - mmengine - INFO - Iter(train) [19180/19224] lr: 2.8736e-10 eta: 0:01:53 time: 2.5384 data_time: 0.0114 memory: 11094 loss: 0.1414 2024/07/25 04:12:14 - mmengine - INFO - Iter(train) [19190/19224] lr: 1.7384e-10 eta: 0:01:27 time: 2.2657 data_time: 0.0101 memory: 10839 loss: 0.1676 2024/07/25 04:12:33 - mmengine - INFO - Iter(train) [19200/19224] lr: 8.8692e-11 eta: 0:01:01 time: 1.9189 data_time: 0.0093 memory: 10269 loss: 0.1517 2024/07/25 04:12:50 - mmengine - INFO - Iter(train) [19210/19224] lr: 3.1929e-11 eta: 0:00:36 time: 1.7128 data_time: 0.0092 memory: 10041 loss: 0.1615 2024/07/25 04:13:05 - mmengine - INFO - Iter(train) [19220/19224] lr: 3.5477e-12 eta: 0:00:10 time: 1.4886 data_time: 0.0081 memory: 12383 loss: 0.1337 2024/07/25 04:13:14 - mmengine - INFO - Saving checkpoint at 19224 iterations