Hey. Fixed my issue. My loss was cross entropy and I was passing softmaxed outputs to it. You must pass outputs without a softmax because cross entropy calculates log-softmax internally. This could be your problem. Check out this thread : Cross Entropy loss is not decreasing