I’m currently developing a transversal machine learning tool that is able to support multiple ML frameworks and therefore I’m doing things a little differently when compared to the regular pytorch workflow.
My model inherits from nn.Module and has the regular init and forward methods. However, to fit the framework, I had to add an update method that calls the forward, computes the loss, calls loss.backward() and calls optimizer.step.
This is the current update bit from my code:
def update(self, input=None, output=None):
# input_source, input_target, tags, input_editor=None, input_client=None
input_source = input['source']
input_target = input['target']
input_editor = input['editor']
input_client = input['client']
tags = output['tags']
inputs = [input_source, input_target]
if input_editor is not None:
inputs += [input_editor]
if input_client is not None:
inputs += [input_client]
if self.cuda:
inputs = [Variable(inpt.cuda()) for inpt in inputs]
tags = Variable(tags.cuda())
else:
inputs = [Variable(inpt) for inpt in inputs]
tags = Variable(tags)
self.optimizer.zero_grad()
out = self.forward(*inputs)
loss = self.loss_fn(out, tags)
loss.backward()
self.optimizer.step()
return loss
Everything is working fine, EXCEPT the update bit of the weights. The update method is being called in a train_loop function that calls model.update(**data). I’m printing the losses from every epoch and the values are exactly the same (with different types of data).
Try seeing if model.parameters() gives you all the parameters you are expecting. Maybe you have defined some of the parameters as Variable instead of nn.Parameter, so the logic behind parameters() can’t see your weights.
Could you try seeing if list(model.parameters()) prints all the parameters of your model?
Also, verify that you are not setting requires_grad=True to all parameters of your network, as it would avoid backprop through the network
The problem seems to be the loss.backward() and optimizer.step() calls that are not updating the weights.
While debugging I did:
a = list(self.parameters())[0]
loss.backward()
self.optimizer.step()
b = list(self.parameters())[0]
torch.equal(a.data, b.data)
And it returned True so the parameter is not being updated at all. I don’t have a clue of what may be wrong with my code anymore. Here’s a small reproducible example of my code:
The example that you showed has a problem with it.
Indeed, when you do list(model.parameters())[0], you are taking a reference of the tensor.
Try adding a clone() instead
a = list(self.parameters())[0].clone()
loss.backward()
self.optimizer.step()
b = list(self.parameters())[0].clone()
torch.equal(a.data, b.data)
Did you remove your custom modules too? I guess you must have repacked the data into a new Variables somewhere, thus breaking the history graph. You might want to check if list(model.parameters())[0].grad is not None. If it is, you must be breaking the graph somewhere (you should never need to take the .data do some op, and wrap it in a Variable).
It is not None it is a full zero matrix!! And that explains why the weights are not being updated…
But I checked again and this only happens because I use BCELoss AND a softmax activation for the output! With a sigmoid activation, the weights are updated…
Am I doing something wrong? I’m pretty much recreating some other code I had in Keras where I used a categorical cross entropy loss (with one-hot encodings) and a final softmax activation. In pytorch I’m not doing one-hot encodings and I’m using BCELoss.
Upon checking a = list(model.parameters())[0].clone() and b = a = list(model.parameters())[0].clone() before and after the call to loss.backward() and optimizer.step(). a==b returns false
Upon printing list(model.parameters())[0].grad it returns a matrix of all 0’s.
I’ve been struggling with this problem for 3 days now. Please help me out. The original question can be found at LSTM Model not training
@smth@apaszke@fmassa please take a look at this. I can share the complete code if I am missing out on providing some details now.
Hi all. I get the similar problem. My bug is probably that I use the wrong combination of Softmax and Loss function, so value of grads are super small.
For me,
My model doesn’t seem to be training.
Upon checking a = list(model.parameters())[0].clone() and b = a = list(model.parameters())[0].clone() before and after the call to loss.backward() and optimizer.step(). a==b returns false
Upon printing list(model.parameters())[0].grad it returns a matrix of all super small number like in order of 10^-8.
In my case, it was because the Softmax function calculate the zero value tensor. So the NLL loss was unchanging and the weights were not being updated.
I am also dealing with the same problem. I get p.grad is None for all the parameters in the network. How do I check if the computation graph is broken somewhere?
@Gkv@RGaonkar I have the same problem here.
My weights never be updated and my model list(model.parameters())[0].grad is None . And my Network didn’t return any grad_fn in tensor.
It turns out I have to return the function instead of variable. I don’t know it is correct or not but it worked for me.
Bug source code
class Model(nn.Module):
def __init__(self, size_in, size_out):
super(Model, self).__init__()
self.ReLU1 = nn.ReLU()
self.conv1 = nn.Conv2d(in_channels=size_in, out_channels=size_out,
kernel_size=(3, 3),
stride=(1, 1),
padding=(1, 1),
dilation=1,
groups=1,
bias=False)
def forward(self, x):
x = self.ReLU1(x)
x = self.conv1(x)
return x
what do you mean by
verify that you are not setting requires_grad=True to all parameters of your network, as it would avoid backprop through the network
Ignore this message. The problem was that I had too shallow of a network! Sorry about the noise! It converged after I added a couple of conv layers.
I am having a very similar issue. Initially, I had a 3 conv+relu+maxpool followed by 2 linears + relu and sigmoid for binary classification. It would not learn – all the parameters would be set to something very close to 0, and the output would be “random”. I reduced the size of the network to Conv/relu + Linear/relu + Linear/sigmoid => same problem.
similar problem faced by me as I was using relu in the layer and then apply softmax on the layer and then cross entropy so make sure that last layer should not contain any activation function if you are using softmax