Titan X Maxwell (old) issues

Eugenio_Culurciello · January 26, 2017, 3:57pm

Hi All, I am having a strange problem on all our Titan X Maxwell (older one) with pytorch:

gist.github.com

https://gist.github.com/culurciello/47acb81fde1d19082f2a59c73c3c2ce0

pytorch failure Titan X (Maxwell)

elab@gpu5 ~/pytorch-examples/imagenet [master*]$ python3 main.py -a alexnet /media/SuperSSD/test-dataset-train-val/
=> creating model 'alexnet'
Traceback (most recent call last):
  File "main.py", line 286, in <module>
    main()
  File "main.py", line 129, in main
    train(train_loader, model, criterion, optimizer, epoch)
  File "main.py", line 165, in train
    output = model(input_var)
  File "/usr/local/lib/python3.4/dist-packages/torch/nn/modules/module.py", line 210, in __call__

This file has been truncated. show original

We are on: Ubuntu 14.04.5 LTS

I can confirm we can run Torch7 on those machines with no issues.

Also on the new Pascal Titan X, we do not have this problem. Maybe an issues with driver?

Anyone else has the same issue?

colesbury · January 26, 2017, 4:43pm

We don’t support Python 3.4 Do you still have this issue with Python 3.5?

Eugenio_Culurciello · January 28, 2017, 3:07pm

After @smth recommendation I updated nccl and was able to make one workstation work that way.