I can’t agree more
Writing a loss function is no different from writing a neural network, or an autograd function.
Here’s an example of writing a mean-square-error loss function:
def mse_loss(input, target):
return ((input - target) ** 2).sum() / input.data.nelement()
@smth but will this version have ability to backpropagate ? i think we need to perform those functions on autograd Variable ??
Yes, smth’s function is taking Variables as input. So you will be able to backpropagate.
I struggled with this myself, so I’ve started building a tutorial for such stuff in PyTorch. You can find a section on custom losses there too (Section 5). Github link - https://github.com/Spandan-Madan/A-Collection-of-important-tasks-in-pytorch
I wrote this up quickly in my free time so it must have some typos etc. If you think there’s things you would like to see there but are missing, feel free to create an issue on GitHub to make suggestions. Hope this helps!
I have already implemented my own loss in python, but it is too slow. Is there any tutorials which can teach me
to speed it up?(there is a for loop in my loss)
if the individual loss for a sample in a batch can be positive or negative depending on some conditions, how do i sum the loss over samples? it will become zero if i sum all the samples within a batch.
excuse me, have you figured this out? So it’s necessary that writing a custom backward function and then return the gradient by self?