Using .cuda() is very slow on pytorch=0.2.0

#1

Due to compatibility issues, I am using pytorch=0.2.0 with python=2.7

I installed it using conda install pytorch=0.2.0 cuda80 -c soumith as it was pointed out on the forum that this will lead to reduction in lag while using .cuda() for the first time, however I do not see any improvements and loading still takes ~3mins.

(P.S I’m using tesla v100)

#2

Has been some time since PyTorch 0.2, but I can’t remember it being that slow. It should take a few seconds at most, not minutes. Not sure what’s causing the problem in your case, could be that it’s been a bug in PyTorch 0.2?

PyTorch 0.3 and 0.4 also work with Python 2.7, regardign the compatibility issues you mentioned, the Tesla V100 should work with cuda 8 & 9, so I think so if you can install it 0.3 or 0.4 somehow alongside the 0.2 version, it would help figuring out whether it’s a PyTorch 0.2-specific bug or sth else.

#3

I remember this issue occurred if a wrong CUDA version was installed and in the first run it’s recompiling pytorch for your GPU.
Unfortunately there is no CUDA9 for pytorch 0.2.0.

However, could you print torch.version.cuda after your first cuda run?

@rasbt ideas sound good to debug your issue!

1 Like
#4

I tried doing torch.version.cuda after the first .cuda() it seems like torch.version doesn’t have cuda attribute, it shows

Traceback (most recent call last):
File "<stdin>", line 1, in <module> 
AttributeError: 'module' object has no attribute 'cuda'
#5

tesla v100 needs cuda 9. cuda 9 is incompatible with pytorch 0.2.0, even if you build 0.2.0 from source.

The solution is to upgrade to 0.4.0 (0.3.0 might work, but I’m like 60% 20% sure you need 0.4.0).

You’ll notice that your ~/.nv directory is probably incrasing in size without bound right?

#6

Yeah, that was my mistake. I think the method was introduces after 0.2.0.

#7

The same lagging issue is happening on other machine as well using GTX-1060, with python 3.5.4 and
pytorch 0.3.0 py35cuda8.0cudnn6.0_0

the output of torch.version.cuda after first .cuda() is '8.0.61'

is there any way to reduce the lag here?

#8

In case the future reader is interested I solved it using the pytorch0.3.1 compiled with cuda 9 using

pip install http://download.pytorch.org/whl/cu90/torch-0.3.0.post4-cp35-cp35m-linux_x86_64.whl