Very Strange Things (New Beginner)

reinforcement-learning
1

In one of my trials, I try to compute loss for each of my data then collect the “prob” in a list. Then, in each iteration(time step), I try to update my net. But with RunError: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 2]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

But, I don’t know where is the wrong lies, since I don’t have any in-place operation. Could any one help me?

My code Like:

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import matplotlib.pyplot as plt
from torch.distributions import Categorical
from torch.nn.utils import clip_grad_norm_

torch.autograd.set_detect_anomaly(True)

class Policy(nn.Module):
    def __init__(self, n_input, n_output, hidden_dim = 128):
        super(Policy,self).__init__()
        self.n_input = n_input
        self.n_output = n_output
        self.lin1 = nn.Linear(self.n_input, hidden_dim)
        self.lin2 = nn.Linear(hidden_dim, self.n_output)
    
    def forward(self,state):
        x = F.relu(self.lin1(state))
        x = self.lin2(x)
        return x

p = Policy(4,2)
lists = []
optimizer = optim.Adam(p.parameters(), lr=0.01)

#psudeo data
data = [[1,2,3,4],[2,1,3,4],[3,1,3,4]]
data = torch.tensor(data).float()

for i in range(data.shape[0]):
    x = data[i,:].unsqueeze(0)
    prob = p(x).max()
    lists.append(prob)

#BIG PROBLEM HERE!!!
for t in range(len(lists)):
    print("t:",t)
    loss = -lists[t]
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
2

Oh, I think something wrong in self.lin2, since the weight of lin2 is (128,2)
`
[W python_anomaly_mode.cpp:60] Warning: Error detected in AddmmBackward. Traceback of forward call that caused the error:
File “/Users/kakusou/Work/VScode/mimic/PG/test.py”, line 34, in
prob = p(x).max()
File “/Users/kakusou/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl
result = self.forward(*input, **kwargs)
File “/Users/kakusou/Work/VScode/mimic/PG/test.py”, line 22, in forward
x = self.lin2(x)
File “/Users/kakusou/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl
result = self.forward(*input, **kwargs)
File “/Users/kakusou/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py”, line 91, in forward
return F.linear(input, self.weight, self.bias)
File “/Users/kakusou/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py”, line 1674, in linear
ret = torch.addmm(bias, input, weight.t())
(function print_stack)

`

3

And, If I run this code in torch1.4.0, no error at all.
But, in torch1.6.0, it continues showing the above error. I just don’t understand it

4

The issue is that the stored probs in lists are invalid after the first optimizer.step() and parameter update.
I.e. the prob tensors were creates with a model using the initial parameter set P0.
After the first loss.backward() and optimizer.step() operation these parameters will be updated to P1.
In the next loss.backward() and optimizer.step() iteration you are trying to calculate the gradients of the “old” loss w.r.t the new parameters, which is wrong since these already parameters didn’t calculate the loss.
Older PyTorch versions didn’t implement this check and would thus yield wrong results.

The proper way would be to recalculate the prob after each update or to calculate all gradients before calling optimizer.step().

1 Like