Loss function conditional on two outputs

Hello,

I have a model that outputs two values, one for a classification task, and other for a regression task. Because I don’t know if it is even possible to use in a single loss function multiple output / target pairs, my model outputs a single tensor where input[:8] are the probabilities for the classification task, and input[8] is the regressed scalar, so the loss function:

def species_length_loss(input, target):
    input_species = input[:,:8]
    input_length  = input[:,8]

    target_species = target[:,0].long()
    target_length  = target[:,1]

    if target_species != 7: #doesn't work b/c target_species is N (batch size)
        loss = F.mse_loss(input_length, target_length) * 1e-4 + F.nll_loss(input_species, target_species)
    else:
        loss = F.nll_loss(input_species, target_species)

    return loss

I’ve placed the if statement to illustrate what I want to do, but even if just leave loss = F.mse_loss(input_length, target_length) * 1e-4 + F.nll_loss(input_species, target_species) (w/o the statement) I get the error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Any ideas of suggestions how to accomplish this?

Thanks

1 Like

I think I have something that works:

import torch
import torch.autograd as autograd
import torch.nn as nn
input = autograd.Variable(torch.randn(3,9).double(), requires_grad=True)
target = autograd.Variable(torch.randn(3,2).random_(0,8).long(), requires_grad=False)

input_species = input[:,:8]
input_length  = input[:,8]
target_species = target[:,0]
target_length  = target[:,1].double()
loss = nn.MSELoss()(input_length, target_length) * 1e-4 + nn.NLLLoss()(input_species, target_species)
loss.backward()

Is this what you’re looking for?

1 Like

This has me puzzled. Your code works but when I try to run the same (well almost):

def species_length_loss(input, target):
    target = th.autograd.Variable(th.randn(32,2).random_(0,8).float(), requires_grad=False).cuda() # JUST TO TEST

    input_species = input[:,:8]
    input_length  = input[:, 8]

    target_species = target[:,0].long()
    target_length  = target[:,1]

    loss = nn.MSELoss()(input_length, target_length) * 1e-4 # + F.nll_loss(input_species, target_species) 
    return loss

It fails as before. I’ve made sure target requires_grad=False but still I’m getting the same error:

Traceback (most recent call last):
  File "fish.py", line 191, in <module>
    trainer.fit_loader(train_loader, valid_loader, num_epoch=200, verbose=1, cuda_device=0)
  File "build/bdist.linux-x86_64/egg/torchsample/modules/module_trainer.py", line 367, in fit_loader
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/variable.py", line 156, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/__init__.py", line 98, in backward
    variables, grad_variables, retain_graph)
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation```

Could it be the way the outputs are defined?

def forward(self, x):
        x = self.features(x)
        x = x.view(-1,self.num_ftrs )
        if False:
            x = F.relu(self.class_fc1(x))
            species = self.class_fc2(x)
            species = nn.LogSoftmax()(species)
            length = self.reg_fc2(x)
            species_length = (species, length)
        else:
            species_length = F.relu(self.class_fc1(x))
            species_length = self.class_fc2(species_length)
            species_length[:8] = nn.LogSoftmax()(species_length[:8])
            species_length[8]  = F.relu(species_length[8])
            #length  = F.relu(self.reg_fc1(x))
            #length  = F.relu(self.reg_fc2(length))
        return species_length```

You do not need to store the values in one array. You can simply return a tuple. This will avoid the inplace operations in species_length
in your forward function you can write:

def forward(self, x):
        x = self.features(x)
        x = x.view(-1,self.num_ftrs )
        if False:
            x = F.relu(self.class_fc1(x))
            species = self.class_fc2(x)
            species = nn.LogSoftmax()(species)
            length = self.reg_fc2(x)
            # species_length = (species, length)
            return species, length
        else:
            species_length = F.relu(self.class_fc1(x))
            species_length = self.class_fc2(species_length)
            # species_length[:8] = nn.LogSoftmax()(species_length[:8])
            species_length = nn.LogSoftmax()(species_length)
            # species_length[8]  = F.relu(species_length[8])
            #length  = F.relu(self.reg_fc1(x))
            #length  = F.relu(self.reg_fc2(length))
            return species_length, F.relu(species_length[8])
1 Like

OK. Works now:

Model:

def forward(self, x):
        x = self.features(x)
        x = x.view(-1,self.num_ftrs )
        species_length = F.relu(self.class_fc1(x))
        species_length = self.class_fc2(species_length)
        _species = species_length.clone()[:,:8,...]
        _length  = species_length.clone()[:, 8,...]
        species_length[:,:8,...] = nn.LogSoftmax()(_species)
        species_length[:, 8,...] = F.relu(_length)
        return species_length

Loss:

def species_length_loss(input, target):

    input_species = input[:,:8]
    input_length  = input[:, 8]

    target_species = target[:,0].long()
    target_length  = target[:,1]

    input_length = input_length * (target_species != 7).float()

    loss = nn.MSELoss()(input_length, target_length) * 1e-4 + F.nll_loss(input_species, target_species) 

    return loss