I’m getting a Segmentation Fault when trying to run my code on an instance with 4 Tesla K80s. If I use any 2 GPUs, the model trains just fine. But when I increase the number of GPUs to 3, I get the Segmentation Fault (core dumped. I’ve tried running gdb and here’s what I see:
Thread 49 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffefaffb700 (LWP 16808)]
0x00007ffff062a8d5 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
(gdb) where
#0 0x00007ffff062a8d5 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#1 0x00007ffff077a914 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2 0x00007ffff0716e80 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3 0x00007ffff77e46ba in start_thread (arg=0x7ffefaffb700) at pthread_create.c:333
#4 0x00007ffff6e0a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)
I’m running within a conda environment on Pytorch 0.3.1 in python 2 and am using Cuda 8.0. I’m also using OpenCV.
I can actually reproduce this issue with this simple script:
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 53 * 53, 120)
self.fc2 = nn.Linear(120, 84)
self.fc3 = nn.Linear(84, 10)
def forward(self, x):
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(-1, 16 * 53 * 53)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x)
return x
if __name__=="__main__":
net = Net()
net.cuda()
net = nn.DataParallel(net)
X = torch.rand(2,3,224,224).cuda()
X = Variable(X)
y = net.forward(X)
If I comment out net = nn.DataParallel(net), it only runs when I specify CUDA_VISIBLE_DEVICES=3. Anything else produces a Segmentation Fault with the same gdb output as above.