Gradient of output is zero

1

Hi,

For my model, output.grad is zero right from the first iteration. My model is as follows -

import torch.nn as nn
import math
import torch.utils.model_zoo as model_zoo
import torch
from torch.autograd.variable import Variable
from torchvision import datasets, models, transforms

class ResnetCombinedFull2(nn.Module):
    def __init__(self,LOAD_VIS_URL=None):
        print('Using MSE Version')
        super(ResnetCombinedFull2, self).__init__()
        
        model = models.resnet18(pretrained = True)
        ct = 0
        for child in model.children():
            for childdd in child.children():
                if ct < 6 :
                    for param in childdd.parameters():
                        param.requires_grad = False
                ct += 1
        
        self.vismodel = nn.Sequential(*list(model.children())[:-1])
        self.projective = nn.Linear(512,400)
        self.nonlinearity = nn.ReLU(inplace=True)
        self.projective2 = nn.Linear(400,300)
    
    def forward(self,x):
        x = self.vismodel(x)
        x = torch.squeeze(x)
        x = self.projective(x)
        x = self.nonlinearity(x)
        x = self.projective2(x)
        x = x/x.norm(2, 1).clamp(min=1e-12).expand_as(x)
        return x

Can someone help me out with this?

Thanks!

2

How are you using ResnetCombinedFull2 and how are you getting the output?

3

Hi,

As simple as, creating an object using -

model = ResnetCombinedFull2()

And then passing in input variable through it using -

output = model(input_variable)

In fact, I noticed that it is zero for a simple fine tuning script too, so it’s not a problem of my model but just something not very intuitive. Any insights are helpful, thanks!

4

Why are you looking at gradient of the output instead of input_variable.grad? How are you using that?

1 Like
5

I was just exploring gradients of all variables, and was a little unsure about what’s usual behavior. But you’re right I should be looking at input_variable.grad, I’ll look into that and get back if something looks out of place. Thanks!