RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.HalfTensor) should be the same

dulanga_heshan · November 27, 2020, 8:53am

I created a custom object detection model using the yoloV5 PyTorch version, so I tried to expose it as flask rest API, by referring to this pytorch guide
I trained and developing flask applications in the same python environment which installed PyTorch 1.7 cuda version

def transform_image(image_bytes):
    my_transforms = transforms.Compose([transforms.Resize(255),
                                        transforms.CenterCrop(224),
                                        transforms.ToTensor(),
                                        transforms.Normalize(
                                            [0.485, 0.456, 0.406],
                                            [0.229, 0.224, 0.225])])
    image = Image.open(io.BytesIO(image_bytes)).convert('RGB')
    return my_transforms(image).unsqueeze(0)

model = torch.load("weights/best.pt", map_location="cuda")['model']
model.eval()

with open("BikesHelmets128.png", 'rb') as f:
    image_bytes = f.read()
    tensor = transform_image(image_bytes=image_bytes)
    print(tensor)

outputs = model.forward(tensor)
_, y_hat = outputs.max(1)
predicted_idx = str(y_hat.item())
 print(predicted_idx)

Error stack

Traceback (most recent call last):
  File "app.py", line 46, in <module>
    outputs = model.forward(tensor)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 121, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 137, in forward_once
    x = m(x)  # run
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 93, in forward
    return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 34, in forward
    return self.act(self.bn(self.conv(x)))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

I tried several ways by changing transforms, changing device type, and read images using open cv, etc.
thank you

Alexey_Demyanchuk · November 27, 2020, 9:01am

Hi. It would be better if you provide error stack, to check from what point the problem is coming. Also, the general advice is to explicitly check if both your model and your input are on the same device.

dulanga_heshan · November 27, 2020, 10:13am

@Alexey_Demyanchuk thanks for your quick response, I added my error stack, and both modal and image are in the same device

Alexey_Demyanchuk · November 27, 2020, 10:55am

It seems like the error is a bit different from you have in the topic title. In your code you explicitly put the model on the cuda device. But I never see you put your tensor on the same device as well. So, something like tensor = tensor.cuda() should work.

dulanga_heshan · November 27, 2020, 11:53am

Yes i tried both ways when i applied tensor = tensor.cuda() it throws error

Traceback (most recent call last):
  File "app.py", line 46, in <module>
    outputs = model.forward(tensor)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 121, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 137, in forward_once
    x = m(x)  # run
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 93, in forward
    return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 34, in forward
    return self.act(self.bn(self.conv(x)))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

and if i removed tensor = tensor.cuda()

Traceback (most recent call last):
  File "app.py", line 46, in <module>
    outputs = model.forward(tensor)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 121, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 137, in forward_once
    x = m(x)  # run
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 93, in forward
    return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 34, in forward
    return self.act(self.bn(self.conv(x)))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

Sorry I updated the question with correct error stack

Alexey_Demyanchuk · November 27, 2020, 12:08pm

From the first error you can see, that now both tensors are on the same device, but model want a tensor in half precision (HalfTensor). You should cast input to this type, I believe. Look here for some reasoning.
In short (on modern version of pytorch) you need:

from  torch.cuda.amp import autocast
with autocast():
    outputs = model.forward(tensor)

Hope this helps.

dulanga_heshan · November 27, 2020, 4:31pm

yeah it works i used model.forward(tensor.half()) and now it works fine, thank you

jinfagang · June 22, 2021, 7:46am

Your works is not autocast(), it’s make your input to half match your moded cause your model is half. autocast() doesn;t work.

Owais_Ansari · June 9, 2023, 4:30am

You can also convert the input tensors into HalfTensor before passing them into a model

tensor = tensor.type(torch.cuda.HalfTensor)
outputs = model.forward(tensor)