Type of Tensor in forward() is CPU tensor even though using cuda?

When I do print(type(x)) of a tensor in forward(), the type is <class 'torch.Tensor'>, shouldn’t it be .cuda.Tensor ?

cuda.is_available() is True, next(model.parameters()).is_cuda is also True, and I can tell my GPU is being used. So I’m sure it’s running on GPU, but I’m confused by the type.

Here’s the model

class Net(nn.Module):
    def __init__(self, vocab_sz, emb_sz, input_sz, out_sz):
        super(Net, self).__init__()
        self.embed = nn.Embedding(vocab_sz, emb_sz)
        self.embed_pos = nn.Parameter(torch.randn(input_sz, 32))
        self.conv1 = nn.Conv2d(1, 32, (3, emb_sz + 32,))
        self.conv2 = nn.Conv2d(32, 64, (2, 1,))
        self.conv3 = nn.Conv2d(64, 64, (2, 1,))
        self.fc = nn.Linear(64, out_sz)

    def forward(self, x):
        bs = x.size(0)

        emb_pos = self.embed_pos.unsqueeze(0).repeat(bs, 1, 1)
        emb = torch.cat((self.embed(x), emb_pos,), dim=2)

        x = emb.unsqueeze(1)
        # print(x.size())
        x = F.relu(self.conv1(x))
        print(type(x))
        # print(x.size())
        x = F.relu(self.conv2(x))
        # print(x.size())
        x = F.relu(self.conv3(x))
        # print(x.size())
        x = x.view(bs, -1)
        y = self.fc(x)
        return F.log_softmax(y, dim=1)

Also: I’m also worried that the line self.embed_pos.unsqueeze(0).repeat(bs, 1, 1) is wasteful, but unsure if it is, and if it is what one could do about it (what I’ve tried so far has resulted in weird errors).

That’s not the piece of code you need to work in GPU.
Can you show how do you set your model / input up?

you should have something like model=model.cuda() and to allocate inputs in gpu at some point

Obviously it is not. And of course I have model = model.to(DEVICE). Did you not read my post completely?

cuda.is_available() is True, next(model.parameters()).is_cuda is also True, and I can tell my GPU is being used. So I’m sure it’s running on GPU, but I’m confused by the type.

But since you asked, the relevant part is

model = model.to(DEVICE)
    modelparams = model.parameters()
    initial_lr = 3e-4
    optim = optims.Adam(modelparams, initial_lr, weight_decay=1e-5)
    num_iters = len(trainset) / BATCH_SIZE
    print('Running training.')
    for epoch in range(1, 1 + 5):
        model.train()
        if epoch == 4:
            for g in optim.param_groups:
                g['lr'] = initial_lr / 10.
        startime = time()
        for i, (feats, targs) in enumerate(trainloader):
            feats = feats.to(DEVICE)
            targs = targs.to(DEVICE)
            optim.zero_grad()
            output = model(feats)
            loss = F.nll_loss(output, targs, reduction='sum')
            loss.backward()
            optim.step()
            if i % int(num_iters // 4) == 0:
                print(loss.item() / len(targs))
        epochtime = time() - startime
        print(f'Epoch done in {epochtime}')
        evaltest(model, testloader)
    print('Done')

I think there are some layers of code which makes it to output tensor type.

You can try to print x.type() instead of type(x) or x.device as it will tell you if it’s allocated on gpu or cpu.

import torch

a=torch.rand(5).cuda()
type(a)
Out[4]: torch.Tensor
a.type()
Out[6]: 'torch.cuda.FloatTensor'
a.device
Out[7]: device(type='cuda', index=0)
1 Like

Thank you! That does show it’s on cuda.