Maybe, you have to modify CustomLoss slightly. An issue due to a custom loss has been reported:
CustomLoss
He resolved this problem by cloning tensors in the custom loss.