We try to solve a similar issue here, I guess. Loss per batch is not decreasing, need help! - PyTorch Forums