Setting requires_grad_ on gpu

Have a look at this post which explains this use case quite well.