Apply math operation dynamically between two nn.Modules or loss functions

1

I would like to apply math operations dynamically between two loss functions or nn.Modules. I am not sure how exactly to implement it. But any help is really appreciated.

For example: In below example I would like to add two loss functions.

nn.L1Loss() + nn.CosineEmbeddingLoss()

If I do this it gives me error:

----> 1 nn.L1Loss() + nn.CosineEmbeddingLoss()
TypeError: unsupported operand type(s) for +: 'L1Loss' and 'CosineEmbeddingLoss'`

I also tried creating a wrapper with forward function and torch operations like below, but it doesn’t work either.

class Execute_Op(nn.Module):
    def __init__(self):
        super().__init__()
        
    def forward(self, x, y, op):
        if op == 'add':
            return torch.add(x, y)
        elif op == 'subtract':
            return torch.subtract(x - y)

exec_op = Execute_Op()
exec_op(nn.L1Loss(), nn.CosineEmbeddingLoss(), 'add')

It gives error like below:

Execute_Op.forward(self, x, y, op)
      5 def forward(self, x, y, op):
      6     if op == 'add':
----> 7         return torch.add(x, y)
      8     elif op == 'subtract':
      9         return x - y

TypeError: add(): argument 'input' (position 1) must be Tensor, not L1Loss
2

You could either use the functional APIs or create objects for both loss functions:

criterion1 = nn.L1Loss()
criterion2 = nn.CosneEmbeddingLoss()
...
output = model(input)
loss1 = criterion1(output, target1)
loss2 = criterion2(output, target2)
loss = loss1 + loss2
loss.backward() # gradient calculation using both loss functions
3

Thanks for the response @ptrblck.
Your response makes sense for a general case.
But I am looking for a dynamic way to create loss functions where I can combine the loss equation without passing the inputs such as output or target, and then at the run time, I can pass the input values. Is there a way or workaround to do that?

4

No, you won’t be able to “combine” objects directly, but could initialize them and add both losses “in a dynamic way” (i.e. if your conditions are met) during the forward pass.
E.g. you could create a list of loss functions based on your conditions and apply them (during the loss calculation).