Let’s say I have the following example (modified from [Data parallel tutorial])(https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#simple-model )
class Model(nn.Module):
# Our model
def __init__(self, input_size, output_size):
super(Model, self).__init__()
self.fc = nn.Linear(input_size, output_size)
self.f = torch.ones(1)
def forward(self, input):
output = self.fc(input) * self.f
print(self.f)
print("\tIn Model: input size", input.size(),
"output size", output.size())
return output
wrapped in a nn.DataParallel
.
This results in this error when doing a forward pass, because of
RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.FloatTensor for argument #2 'other'
If I try to call .cuda()
on the f
field, it goes on the first cuda device, and then the forward pass does not work because they are on different devices:
class Model(nn.Module):
# Our model
def __init__(self, input_size, output_size):
super(Model, self).__init__()
self.fc = nn.Linear(input_size, output_size)
self.f = torch.ones(1).cuda()
def forward(self, input):
output = self.fc(input) * self.f
print(self.f)
print("\tIn Model: input size", input.size(),
"output size", output.size())
return output
Log:
RuntimeError: arguments are located on different GPUs at
So how do I use data parallel with the functional API?