RuntimeError: expected type torch.FloatTensor but got torch.cuda.FloatTensor

vision
1

After loading the model from check point . I tried to train the model again. While training it shows error on optimizer.step()

def load_checkpoint(checkpoint_path):
    
    model = models.vgg11(pretrained=True)
    for param in model.parameters():
        param.requires_grad = False
    
    checkpoint = torch.load(path)
    
    classifier = nn.Sequential(OrderedDict([('fc1', nn.Linear(checkpoint['input_size'], 6272)),
                                        ('relu', nn.ReLU()),
                                        ('drop1',nn.Dropout(p=0.4)),
                                        ('fc2', nn.Linear(6272, 512)),
                                        ('relu', nn.ReLU()),
                                        ('drop2',nn.Dropout(p=0.2)),
                                        ('fc3', nn.Linear(512, checkpoint['out_size'])),
                                        ('output', nn.LogSoftmax(dim=1))]))
    
    model.classifier = classifier
    model.classifier.load_state_dict(checkpoint['state_dict'])
    #model.classifier.fc1.in_features = checkpoint['input_size']
    #model.classifier.fc3.out_features = checkpoint['out_size']
    class_to_idx = checkpoint['class_to_idx']
    
    optimizer = optim.Adam(model.classifier.parameters(), lr = 0.001)
    optimizer.load_state_dict(checkpoint['optim_state'])
    
    return model, optimizer, class_to_idx

model, optimizer, class_to_idx = load_checkpoint(path)

idx_to_class = { v : k for k,v in class_to_idx.items()}

Screenshot%20from%202019-02-16%2021-41-37
Screenshot%20from%202019-02-16%2021-41-371920×1080 216 KB

RuntimeError                              Traceback (most recent call last)
<ipython-input-7-5e78ed847bd0> in <module>()
     16         loss = criterion(logps, labels)
     17         loss.backward()
---> 18         optimizer.step()
     19 
     20         train_loss += loss.item()

/usr/local/lib/python3.6/dist-packages/torch/optim/adam.py in step(self, closure)
     91 
     92                 # Decay the first and second moment running average coefficient
---> 93                 exp_avg.mul_(beta1).add_(1 - beta1, grad)
     94                 exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
     95                 if amsgrad:

RuntimeError: expected type torch.FloatTensor but got torch.cuda.FloatTensor```
2

Make sure all the relevant variables are in GPU.

Can you try to push the optimizer to gpu as well, before training loop?

optimizer = optimizer.cuda()
3

@rahulthen how did you fix this?

4

I had a similar issue and I think it is related to your optimizer referencing model parameters that are not yet loaded on GPU. In my case I fixed the issue by changing the following:

optimizer = torch.optim.Adam(model.parameters(), lr=params.lr)
try:
       checkpoint = torch.load("checkpoints/"+model_folder+"checkpoint.pth.tar")
       start_epoch = checkpoint['epoch']
       scheduler.load_state_dict(checkpoint['scheduler'])
       model.load_state_dict(checkpoint['state_dict'])
       model.to(device) # moving this line here fixed the issue
       optimizer.load_state_dict(checkpoint['optimizer'])
       …
       # it used to be somewhere around here

So it seems you need to send de model to GPU BEFORE loading the optimizer dictionary, else it will be loaded to the usual memory instead of GPU memory, which caused your issue (I think).

1 Like
5

thanks! that does the job!