Is this custom loss function properly implemented?

Hi, I read PyTorch with Example and I tried to implement network with learned parameters and custom loss function. Following is abstract of what I did:

  • Loaded learned model of ‘model.pth’ which was converted from ‘.t7’ by using convert_torch_to_pytorch
  • Defined custom loss function of triplet_loss() which is unsure of working properly

Below is the code:

# define custom loss function

from torch import exp
from torch.autograd import Variable

class LambdaBase(nn.Sequential):
    def __init__(self, fn, *args):
        super(LambdaBase, self).__init__(*args)
        self.lambda_func = fn

    def forward_prepare(self, input):
        output = []
        for module in self._modules.values():
            output.append(module(input))
        return output if output else input

class Lambda(LambdaBase):
    def forward(self, input):
        return self.lambda_func(self.forward_prepare(input))

model = nn.Sequential( # Sequential,
	nn.Conv2d(3,64,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Conv2d(64,64,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Dropout(0.25),
	nn.MaxPool2d((4, 4),(4, 4)),
	nn.BatchNorm2d(64,0.001,0.9,True),
	nn.Conv2d(64,128,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Conv2d(128,128,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Dropout(0.25),
	nn.MaxPool2d((4, 4),(4, 4)),
	nn.BatchNorm2d(128,0.001,0.9,True),
	nn.Conv2d(128,256,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Conv2d(256,256,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Dropout(0.25),
	nn.MaxPool2d((4, 4),(4, 4)),
	nn.BatchNorm2d(256,0.001,0.9,True),
	nn.Conv2d(256,128,(1, 1)),
	nn.ReLU(),
	Lambda(lambda x: x.view(x.size(0),-1)), # Reshape,
	nn.Sequential(Lambda(lambda x: x.view(1,-1) if 1==len(x.size()) else x ),nn.Linear(3072,128)), # Linear,
)

model.load_state_dict(torch.load('model.pth'))

def triplet_loss(vec_ref, vec_pos, vec_neg):
    
    dist_pos = torch.norm(vec_ref - vec_pos, 2)
    dist_neg = torch.norm(vec_ref - vec_neg, 2)
    
    reg_dist_pos = exp(dist_pos)/(exp(dist_pos) + exp(dist_neg))
    reg_dist_neg = exp(dist_neg)/(exp(dist_pos) + exp(dist_neg))
    
    loss = reg_dist_pos
    
    return loss

learning_rate = 1e-4
triplet_samples = 100

for t in range(triplet_samples):
    # extract feature of t th image triplet
    # img_***[t] is tensor of image
    y_ref = model(img_ref[t])
    y_pos = model(img_pos[t])
    y_neg = model(img_var[t])
    
    loss = triplet_loss(y_ref, y_pos, y_neg)
    
    print(t, loss.data[0])
    model.zero_grad()
    
    loss.backward()
    
    for param in model.parameters():
        param.data -= learning_rate * param.grad.data

I tried this code on one image triplet and worked and computed loss. But I’m not sure whether the loss.backward() properly works. Could you explain it is properly implemented or not?

Thank you.

PyTorch is pretty good at automatically differentiating stuff. If loss.backward() isn’t throwing any errors then chances are it is working properly.

I notice that triplet_loss calculates reg_dist_neg but never uses it. Is that intentional?

@jpeg729 Thank you for reply.

triplet_loss calculates reg_dist_neg but never uses it. Is that intentional?

Yes, It’s intentional. I’ll use value of reg_dist_neg after addtional modification of triplet_loss.

Now I have another question that:
When I forward() same image Tensor two times, I’ll get two different output. Is this because of the dropout layer?
When I want to use this network as feature extractor, what should I do?

You are correct. The dropout layer randomly affects the output during training. After training just do

model.train(False) 
# or equivalently
model.eval()

# dropout is now switched off

stable_prediction = model(input)
1 Like

I really thank you!
I could extract specific value for one image after applied your suggestion.