Some more info which may be helpful:
I'm running a GTX 1060 (Pascal) with cuda 7.5. I've noticed that the first call to
.cuda() causes GPU memory usage to rise to about 200MB before returning. Also, python maxes out a CPU core and eats about a gig of RAM until it returns. Don't know if it makes a difference, but I installed PyTorch via conda.