I’ve read your loss function
but I can’t see what is the difference except you use torch and F’s operation (like torch.mean(), F.softplus()) at the end of each function

But loss value still shows constant, which means CNN is not updated like this

Total number of parameters: 224481
en_i_loss tensor(8.7888, device='cuda:0', requires_grad=True)
en_i_loss.grad tensor(1., device='cuda:0')
Saved model at end of iteration
en_i_loss tensor(8.7888, device='cuda:0', requires_grad=True)
en_i_loss.grad tensor(1., device='cuda:0')
Saved model at end of iteration
en_i_loss tensor(8.7888, device='cuda:0', requires_grad=True)
en_i_loss.grad tensor(1., device='cuda:0')
Saved model at end of iteration
Saved model at end of epoch
Train finished

And above CNN not being updated issue was my first issue.

Implementing custom loss function is attempt to resolve that first issue.

I think I’ve solved this issue.
I shouldn’t have broken backpropagation graph by newly and unnecessarily wrapping torch Variable tensor in loss_functions.py.

Newly wrapping torch tensor by Variable() deletes grad values.

In conclusion, I checked and made all torch Variable tensors print grad_fn
by using this

print(torch_Variable_tensor.grad_fn) # <MeanBackward1 object at 0x7f1ebc889da0>

If all torch tensors print grad_fn information, that means backpropagation graph is well connected
and means you can perform backpropagation by using loss.backward()