Due to compatibility issues, I am using pytorch=0.2.0 with python=2.7
I installed it using
conda install pytorch=0.2.0 cuda80 -c soumith as it was pointed out on the forum that this will lead to reduction in lag while using
.cuda() for the first time, however I do not see any improvements and loading still takes ~3mins.
(P.S I’m using tesla v100)
Has been some time since PyTorch 0.2, but I can’t remember it being that slow. It should take a few seconds at most, not minutes. Not sure what’s causing the problem in your case, could be that it’s been a bug in PyTorch 0.2?
PyTorch 0.3 and 0.4 also work with Python 2.7, regardign the compatibility issues you mentioned, the Tesla V100 should work with cuda 8 & 9, so I think so if you can install it 0.3 or 0.4 somehow alongside the 0.2 version, it would help figuring out whether it’s a PyTorch 0.2-specific bug or sth else.
I remember this issue occurred if a wrong CUDA version was installed and in the first run it’s recompiling pytorch for your GPU.
Unfortunately there is no CUDA9 for pytorch
However, could you print
torch.version.cuda after your first cuda run?
@rasbt ideas sound good to debug your issue!
I tried doing
torch.version.cuda after the first
.cuda() it seems like
torch.version doesn’t have
cuda attribute, it shows
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'cuda'
tesla v100 needs cuda 9. cuda 9 is incompatible with pytorch 0.2.0, even if you build 0.2.0 from source.
The solution is to upgrade to 0.4.0 (0.3.0 might work, but I’m like
60% 20% sure you need 0.4.0).
You’ll notice that your ~/.nv directory is probably incrasing in size without bound right?
Yeah, that was my mistake. I think the method was introduces after
The same lagging issue is happening on other machine as well using GTX-1060, with python 3.5.4 and
pytorch 0.3.0 py35cuda8.0cudnn6.0_0
the output of
torch.version.cuda after first
is there any way to reduce the lag here?
In case the future reader is interested I solved it using the pytorch0.3.1 compiled with cuda 9 using
pip install http://download.pytorch.org/whl/cu90/torch-0.3.0.post4-cp35-cp35m-linux_x86_64.whl