i have exactly problem as yours description,have you solve this problem?i also try this Manually calculate loss over a number of images and then back propagate the average loss and update network weight but the net didn’t converge,i 'd be very glad for your reply