What optimizer and parameters did you use? Try reducing LR by a factor of 10 until you see the loss being reduced.