I have one machine with a Titan TI 1080, and another with a couple Quadro K4200s. When I start my PyTorch code on the Quadros, the training starts immediately. When I run the same code on the Titan TI machine, it takes a minute or two for the training to start. If I check nvidia-smi during this time, I can see the memory usage of the GPU slowly increase. Once the memory is fully allocated, it runs quickly like normal. Is this because this type of GPU can only slowly allocate memory? Could it have something to do with my PyTorch installation itself? Or would it likely be the rest of the environment setup on that machine? Or might it be something else entirely? Thank you!
There is something wrong with your drivers or with you compilation if you compile Pytorch yourself. I am not sure what the real reason is, I just remember, that I had the same problem, when I messed up my installation!
Thanks! Good to know it can be fixed.