RuntimeError: cuda runtime error (6) : the launch timed out and was terminated at


I am new to Pytorch and I am trying to fit a resnet like model on CIFAR100 dataset.
but while training, I keep getting this error

This happens during training, mostly around at epoch 10~15

The code actually worked fine until yesterday.
I first thought this is some kind of driver problem so I tried reinstalling my graphics driver and cuda.

Is there anyone who had the same problem?
How do you solve it?

I am using pytorch 0.4.1
GTX 1060 with CUDA 9