params seems to be a dict object containing the modelâs state_dict in its ['model'] key as well as other objects such as the optimizerâs state_dict in ['optimizer'], the training results etc. as given in the error message.
Index the 'model' key and it might work,
Yes, while checking with above solution, I got the following error:
ERROR - Exception caught while uploading models: âModelâ object has no attribute âcopyâ
File "/home/user/anaconda3/envs/dali/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1455, in load_state_dict
state_dict = state_dict.copy()
File "/home/user/anaconda3/envs/dali/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1177, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Model' object has no attribute 'copy'
It seems youâve saved the model object directly instead if its state_dict, so you could try to assign the model directly via model = params['model'].
I have got the above output as a param from torch.load(âyolov5s.ptâ map_location=âcpuâ).
I have used model = params['model'] .
got the following issue:
File "models/load.py", line 73, in attempt_load
model.append(ckpt['ema' if ckpt.get('ema') else 'model'].float().fuse().eval())
File "/home/user/anaconda3/envs/dali/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1177, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Model' object has no attribute 'append'
I assume models.load.py and attempt_load are custom methods, so you would need to check your code to see how the model loading is expected.
Based on the new error it seems as if the used model object is supposed to be a list.
while loading model using CPU, it only usages 17.5MB size of memory
see the following profiling result (check line number 35)
Line Mem usage Increment Occurrences Line Contents
30 386.0 MiB 386.0 MiB 1 @profile
31 def attempt_load(weights, map_location=None):
32 # Loads an ensemble of models weights=[a,b,c] or a single model weights=[a] or weights=a
33 386.0 MiB 0.0 MiB 1 model = Ensemble()
34 422.7 MiB 0.0 MiB 2 for w in weights if isinstance(weights, list) else [weights]:
35 403.6 MiB 17.5 MiB 1 ckpt = torch.load(w, map_location='cpu')
Now, I tried the load same model on GPU cuda:0 but memory behavior is drastically changed,
memory cpu ram surges to 2713.8 MiB, see the profiling result below(line no. 35):
Line Mem usage Increment Occurrences Line Contents
30 385.5 MiB 385.5 MiB 1 @profile
31 def attempt_load(weights, map_location=None):
32 # Loads an ensemble of models weights=[a,b,c] or a single model weights=[a] or weights=a
33 385.5 MiB 0.0 MiB 1 model = Ensemble()
34 3099.6 MiB 0.0 MiB 2 for w in weights if isinstance(weights, list) else [weights]:
35 3099.3 MiB 2713.8 MiB 1 ckpt = torch.load(w, map_location='cuda:0')
thatâs why I was looking for some solution referred from (link) to load model first on cpu then transfer to gpu, with the above discussion I was trying to find a solution using state_dict but not resolved that yet, Want to understand why this memory surges even I changed the load device?
Assuming you are profiling the host RAM, the memory increase would be caused by loading all CUDA libs and is expected.
The linked issue targets an OOM on the device, which I donât know if you are running into.
with the above way, I was able to load state_dict which contains weights and biases, using load_state_dict to the existing model. which resolve my above-mentioned state_dict issue.
Now, I want to load the same model which I already loaded in CPU to GPU but the model is not able to migrate from cpu to gpu, even not getting any error but model stayed only on cpu even I tried to migrate it to GPU with following way:
I have tested model in above-mentioned way,
observation is : model.named_parameters() and model.named_buffers() shows the blank result, inside that no weights and bias. looks like as you mentioned it is âunregisteredâ.
So how can I resolve this to do register those?
Did you manipulate the internal parameters, buffers, or submodules somehow by e.g. using setattr etc.?
Parameters, buffers, and submodules (containing both) are directly registered once they are assigned to the parent module via: