Best way to find a specific tensor within a model

1

I have a fairly complex module that I am trying to call copy.deepcopy on. I run into the error:

RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment.  If you were attempting to deepcopy a module, this may be because of a torch.nn.utils.weight_norm usage, see https://github.com/pytorch/pytorch/pull/103001

I see some posts about calling detach() or general advice about how to change things to allow this to work. But I don’t know how to track down what tensor is actually the problem. From within the debugger I can get to _ tensor.py_(86) __ deepcopy__() and print self to get the value of the tensor. However when I go up in the debugger to where deepcopy is first called on net, both of the following do not print out the values in my tensor:

for x in net.buffers(): print(x)
for x in net.parameters(): print(x)

Anyone know how to actually find the tensor that is causing this problem? I’m guessing its a grad tensor or something that is not being cleared but no idea how to track it down. Being able to find it by name would be especially helpful.

2

Changing to:

for x in net.buffers(): print(x.shape)
for x in net.parameters(): print(x.shape)

Also isnt showing anything that even has the same shape, so maybe its not a grad tensor.

3

Hi,

Would you be able to share the stack trace from the deepcopy call?
In the frames above the current Tensor being copied, you should see the deepcopy of the Module holding it happening. From there you should be able to see which Module it is.

1 Like
4

It doesn’t look like it. Everything in between where I call deep copy (currently inside a loop while I dig into this) is in the copy library rather than the torch library.

  File "/myCode/copycode.py", line 398, in copycode                                                                                                                                                                                                       
    for x in net.__dir__():                                                                                                                                                                                                                                                    
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 172, in deepcopy                                                                                                                                                                                     
    y = _reconstruct(x, memo, *rv)                                                                                                                                                                                                                                             
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 270, in _reconstruct                                                                                                                                                                                 
    state = deepcopy(state, memo)                                                                                                                                                                                                                                              
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 146, in deepcopy                                                                                                                                                                                     
    y = copier(x, memo)                                                                                                                                                                                                                                                        
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 230, in _deepcopy_dict                                                                                                                                                                               
    y[deepcopy(key, memo)] = deepcopy(value, memo)                                                                                                                                                                                                                             
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 172, in deepcopy                                                                                                                                                                                     
    y = _reconstruct(x, memo, *rv)                                                                                                                                                                                                                                             
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 296, in _reconstruct                                                                                                                                                                                 
    value = deepcopy(value, memo)                                                                                                                                                                                                                                              
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 172, in deepcopy                                                                                                                                                                                     
    y = _reconstruct(x, memo, *rv)                                                                                                                                                                                                                                             
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 270, in _reconstruct                                                                                                                                                                                 
    state = deepcopy(state, memo)                                                                                                                                                                                                                                              
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 146, in deepcopy                                                                                                                                                                                     
    y = copier(x, memo)                                                                                                                                                                                                                                                        
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/copy.py", line 153, in deepcopy
    y = copier(memo)
  File "/home/myCode/.conda/envs/conda_env/lib/python3.9/site-packages/torch/_tensor.py", line 86, in __deepcopy__
    raise RuntimeError(
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment.  If you were attempting to deepcopy a module, this may be because of a torch.nn.utils.weight_norm usage, see https://github.com/pytorch/pytorch/pull/103001
5

Just found it. I used the following to get to the bottom of it if anyone else encounters this error in the future, either with deepcopy or another function where one elusive tensor is causing problems.

    for x in net.dec.gru_rnn.myProblemModule.__dir__():
        # Skip member variables that start with _
        if x[0] == '_':
            continue
        # Skip member variables that are methods
        if  'method' in str(type(getattr(net.dec.gru_rnn.myProblemModule,x))):
            continue
        print(x);
        copy.deepcopy(getattr(net.dec.gru_rnn.myProblemModule,x))

I started with just net and then iterated up to net.dec.gru_rnn.myProblemModule each time print(x) let me know which module was causing deepcopy to throw the error.

1 Like