Unstable training in pytorch

I recently ported monodepth which is written in tensorflow to pytorch(ported code). But the training seems unstable than the tensorflow implementation(github issue).

For the first 10 - 20 epoch during training the result looks promising but its get very unstable after that. Could any one figure out the issue?