Model.cuda() is very slow

I just updated my pytorch to 1.3 and it takes a long time (5~10mins) to call cuda() on my quite large model. Before the update, its almost instantaneous. I am using a titan x pascal with cudatoolkit 10.1.168 if that helps.

Once the model is loaded, the training itself seems fine.

I am also experiencing this issue. I posted an issue to the pytorch github. The quick fix is to downgrade to CUDA 10.0

conda install -c pytorch pytorch cudatoolkit=10.0

not entirely sure why yet, but we’re looking into it.

tracking in: