When run on Ascend, load model error
#2
by
lrisguan
- opened
Hello, admire your works! But when I run on Ascend, I encounter some errors like:
- When using mindspore==2.7.0
- with dtype=mindspore.float16
[MS_ALLOC_CONF]Runtime config: enable_vmm:True vmm_align_size:2MB
Some weights of DeepseekOCRForCausalLM were not initialized from the model checkpoint at lvyufeng/DeepSeek-OCR-Community-Latest and are newly initialized: ['model.vision_model.embeddings.position_ids']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
error:
----------------------------------------------------
- Kernel select failed:
----------------------------------------------------
Select CPU operator[Stack] fail! Unsupported data type!
The supported data types are input[Float16], output[Float16]; input[Float32], output[Float32]; input[Float64], output[Float64]; input[Int8], output[Int8]; input[Int16], output[Int16]; input[Int32], output[Int32]; input[Int64], output[Int64]; input[UInt8], output[UInt8]; input[UInt16], output[UInt16]; input[UInt32], output[UInt32]; input[UInt64], output[UInt64]; input[Complex64], output[Complex64]; input[Complex128], output[Complex128]; input[Bool], output[Bool]; , but get input[BFloat16 ] and output[BFloat16 ]
node: @pynative_kernel_graph4000000041:CNode_1{[0]: ValueNode<Primitive> Stack, [1]: @pynative_kernel_graph4000000041:param_Parameter_2}
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/cpu/hal/hardware/cpu_device_context.cc:514 SetOperatorInfo
when with dtype=mindspore.float32, errors the same with above.
- When using mindspore==2.7.1:
[MS_ALLOC_CONF] config: enable_vmm:True vmm_align_size:2MB
Some weights of DeepseekOCRForCausalLM were not initialized from the model checkpoint at lvyufeng/DeepSeek-OCR-Community-Latest and are newly initialized: ['model.vision_model.embeddings.position_ids']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
error:Invalid sync host to device for tensor storage info in device address:0xfffe8c184b40 device type:1 tensor storage info:0x3db59f40TensorStorageInfo(shape=[1, 257] strides=[257, 1] storage_offset=0 ori_shape=[257] ori_strides=[1] is_contiguous=1 ori_size=0) size:2056 format:-1 dtype:35 device id:0 device name:CPU shape vector:{1 257 } device point:0xfffe84cf6670 ptr:0x3c3c1d20 from mem pool:0 deleter:1 from persist mem:0 need recycle:0 padding type: and:0x3d72a0c0 device type:2 tensor storage info:0 size:2056 format:-1 dtype:35 device id:0 device name:Ascend shape vector:{1 257 } device point:0x3d1f18d0 ptr:0x12c2cdc6ec00 from mem pool:1 deleter:1 from persist mem:0 need recycle:0 padding type:
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/ascend/res_manager/ascend_res_manager.cc:1159 AsyncHostToDevice
- Environment:
all the same with model card wrote, but mindspore==2.7.0 and mindspore==2.7.1
CANN=8.2.RC2
- code:
try:
import os
import mindnlp
import mindspore as ms
from transformers import AutoModel, AutoTokenizer
ms.set_device("Ascend")
model_name = 'lvyufeng/DeepSeek-OCR-Community-Latest'
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, dtype=ms.float32, _attn_implementation='sdpa', trust_remote_code=True, use_safetensors=True, device_map='npu:0')
model = model.eval()
# prompt = "<image>\nFree OCR. "
prompt = "<image>\n<|grounding|>Convert the document to markdown. "
image_file = 'tmp.png'
output_path = './'
# infer(self, tokenizer, prompt='', image_file='', output_path = ' ', base_size = 1024, image_size = 640, crop_mode = True, test_compress = False, save_results = False):
# Tiny: base_size = 512, image_size = 512, crop_mode = False
# Small: base_size = 640, image_size = 640, crop_mode = False
# Base: base_size = 1024, image_size = 1024, crop_mode = False
# Large: base_size = 1280, image_size = 1280, crop_mode = False
# Gundam: base_size = 1024, image_size = 640, crop_mode = True
res = model.infer(tokenizer, prompt=prompt, image_file=image_file, output_path = output_path, base_size = 1024, image_size = 640, crop_mode=True, save_results = True, test_compress = True)
except Exception as e:
print(f"error:{e}")
on 910B?
No, on 310p3, ,actually Atlas 300I Duo.
No, on 310p3, ,actually Atlas 300I Duo.
I found the problem, mindspore do not support bfloat16 dtype on CPU, but most of preprocessing and postprocesing of transformers use torch.device('cpu'), I will fix it later.
all the issue have been fixed, please try again
Sorry for late to get the news! I tested on my machine. It seems there still some erorrs, but may be caused by the operator itself.
the errors like:
Some weights of DeepseekOCRForCausalLM were not initialized from the model checkpoint at lvyufeng/DeepSeek-OCR-Community-Latest and are newly initialized: ['model.vision_model.embeddings.position_ids']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
The following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:1 for open-end generation.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
[ERROR] OP(3342821,python):2025-12-22-14:44:47.392.624 [kernel_mgr.cpp:205][NNOP][ParseDynamicKernelConfig][3342976] errno[561108] OpName:[aclnnUpsampleLinear1d_81_ResizeD] Parse dynamic kernel config fail.
[ERROR] OP(3342821,python):2025-12-22-14:44:47.392.787 [kernel_workspace.cpp:91][NNOP][GetWorkspace][3342976] errno[561000] OpName:[aclnnUpsampleLinear1d_81_ResizeD] AclOpKernelInit failed opType
[ERROR] OP(3342821,python):2025-12-22-14:44:47.392.825 [resize_d.cpp:80][NNOP][ResizeD][3342976] errno[561103] OpName:[aclnnUpsampleLinear1d_81_ResizeD] ResizeDAiCore ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] OP(3342821,python):2025-12-22-14:44:47.392.855 [aclnn_upsample_linear_1d.cpp:264][NNOP][aclnnUpsampleLinear1dGetWorkspaceSize][3342976] errno[561000] OpName:[aclnnUpsampleLinear1d_81] check ResizeDOut != nullptr failed.
[ERROR] OP(3342821,python):2025-12-22-14:44:47.392.867 [aclnn_upsample_linear_1d.cpp:176][NNOP][~UniqueExecutor][3342976] errno[561102] OpName:[aclnnUpsampleLinear1d_81] When aclnnUpsampleLinear1dGetWorkspaceSize do success, ReleaseTo(executor) should be called before return.
Traceback (most recent call last):
File "/root/code/.code/ocr/ocr.py", line 33, in <module>
res = model.infer(tokenizer, prompt=prompt, image_file=image_file, output_path = output_path, base_size = 1024, image_size = 640, crop_mode=True, save_results = True, test_compress = True)
File "/root/.cache/huggingface/modules/transformers_modules/lvyufeng/DeepSeek_hyphen_OCR_hyphen_Community_hyphen_Latest/87108be92a07e9aa9e87ea16894e83ed174a7f75/modeling_deepseekocr.py", line 942, in infer
output_ids = self.generate(
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/utils/_contextlib.py", line 117, in decorate_context
return func(*args, **kwargs)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/transformers/generation/utils.py", line 2564, in generate
result = decoding_method(
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/transformers/generation/utils.py", line 2784, in _sample
outputs = self(**model_inputs, return_dict=True)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 826, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 840, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/lvyufeng/DeepSeek_hyphen_OCR_hyphen_Community_hyphen_Latest/87108be92a07e9aa9e87ea16894e83ed174a7f75/modeling_deepseekocr.py", line 598, in forward
outputs = self.model(
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 826, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 840, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/lvyufeng/DeepSeek_hyphen_OCR_hyphen_Community_hyphen_Latest/87108be92a07e9aa9e87ea16894e83ed174a7f75/modeling_deepseekocr.py", line 458, in forward
local_features_1 = sam_model(patches)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 826, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 840, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/lvyufeng/DeepSeek_hyphen_OCR_hyphen_Community_hyphen_Latest/87108be92a07e9aa9e87ea16894e83ed174a7f75/deepencoder.py", line 705, in forward
x = blk(x)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 826, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 840, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/lvyufeng/DeepSeek_hyphen_OCR_hyphen_Community_hyphen_Latest/87108be92a07e9aa9e87ea16894e83ed174a7f75/deepencoder.py", line 769, in forward
x = self.attn(x)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 826, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/nn/modules/module.py", line 840, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/lvyufeng/DeepSeek_hyphen_OCR_hyphen_Community_hyphen_Latest/87108be92a07e9aa9e87ea16894e83ed174a7f75/deepencoder.py", line 824, in forward
q, k, v = qkv.reshape(3, B * self.num_heads, H * W, -1).unbind(0)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/_tensor.py", line 2433, in unbind
return ops.unbind(self, dim)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/ops/array.py", line 476, in unbind
return execute("unstack_view", input, dim)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/executor.py", line 6, in execute
out, device = dispatcher.dispatch(func_name, *args, **kwargs)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/dispatcher.py", line 57, in dispatch
return func(*args, **kwargs), device
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindtorch/_apis/npu.py", line 1430, in unstack_view
return pyboost.unstack_ext_view_op(input, dim)
File "/root/tools/anaconda3/envs/ocr-9/lib/python3.9/site-packages/mindspore/ops/auto_generate/gen_ops_prim.py", line 17039, in __call__
res = pyboost_unstack_ext_view(self, [input, dim])
RuntimeError: aclnnUpsampleLinear1dGetWorkspaceSize call failed, please check!
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EZ9999: Inner Error!
EZ9999: [PID: 3342821] 2025-12-22-14:44:47.392.670 Parse dynamic kernel config fail.[THREAD:3342976]
TraceBack (most recent call last):
AclOpKernelInit failed opType[THREAD:3342976]
ResizeDAiCore ADD_TO_LAUNCHER_LIST_AICORE failed.[THREAD:3342976]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn/en for error code description)
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ops/kernel/ascend/pyboost/customize/upsample_linear1d.cc:42 operator()
sorry about thst, 310p lack some operator，but i don't have 310p device，i am trying to find a machine to test