Expected object of backend CUDA but got backend CPU for argument #2 'mat2' err

1

I received this error
Expected object of backend CUDA but got backend CPU for argument #2 'mat2'
while trying to run my model. Here is what I have.

torch.cuda.current_device()

device = torch.device("cuda:0" if (torch.cuda.is_available() and ngpu > 0) else "cpu")

class Flatten(nn.Module):
    def forward(self, x):        
        x = x.view(-1)
        return x

class Discriminator(nn.Module):
    def __init__(self,ngpu):
        super().__init__()
        self.ngpu=ngpu
        self.main=nn.Sequential(
            nn.Conv2d(in_channels=nc, out_channels=ndf, kernel_size=(4,4), stride=2, padding=1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            nn.BatchNorm2d(ndf),
            nn.Conv2d(ndf, ndf * 2, kernel_size=(4,4), stride=2, padding=1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(ndf*2, ndf * 4, kernel_size=(4,4), stride=2, padding=1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
        )
    def forward(self, inputs):
        x= self.main(inputs)
        x=Flatten()(x)
        x=nn.Linear(x.size(-1),128)(x)
        x=nn.Sigmoid()(x)
        return x

model=Discriminator(ngpu).to(device)
a=torch.randn(128,3,128,128,device=device)
model(a)

I was trying to play around with the model in this tutorial and all hyperparameters were taken from there. I even tried adding .to(device) to almost every single line of the code but it did not resolve the problem.

Any help is much appreciated.

2

Currently you are re-creating a nn.Linear layer in your forward method in each call.
This will reinitialize its parameters in each forward pass, so that this layer won’t be able to learn anything.
That being said, the to() operator also cannot push this module to the device, so that you would have to do it manually in the forward method.

However, since I assume you would like to train this layer, just define it inside __init__:

class Discriminator(nn.Module):
    def __init__(self,ngpu):
        super().__init__()
        self.ngpu=ngpu
        self.main=nn.Sequential(
            nn.Conv2d(in_channels=nc, out_channels=ndf, kernel_size=(4,4), stride=2, padding=1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            nn.BatchNorm2d(ndf),
            nn.Conv2d(ndf, ndf * 2, kernel_size=(4,4), stride=2, padding=1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(ndf*2, ndf * 4, kernel_size=(4,4), stride=2, padding=1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
        )
        self.fc = nn.Linear(INPUT_FEATURES, 128)

    def forward(self, inputs):
        x = self.main(inputs)
        x = x.view(x.size(0), -1)
        x = self.fc(x)
        x = torch.sigmoid(x)
        return x

I’ve also removed the Flatten module, since it flattens the complete tensor to a single dimension, which is probably also not desired.

1 Like
3

Thank you very much for your answer. This does indeed solve my problem. Just out of curiosity, can you clarify a bit on how to push the module to cuda “manually” in the forward method?

4

Just for the sake of completeness, as I don’t recommend doing it for the aforementioned issues:

def forward(self, inputs):
        x = self.main(inputs)
        x = Flatten()(x)
        lin = nn.Linear(x.size(-1),128).to(device)
        x = lin(x)
        x = nn.Sigmoid()(x)
        return x

This will now create a new linear module in each forward pass, push the parameters to the device, and feed x to it. However, please don’t use this code, except you really want to create new linear layers in each forward pass.

1 Like