Autocast error: expected scalar type Half but found Float

1

Hello all,

I’m trying to add AMP to the source code from a paper I read recently, but it crashes in the forward pass with the error message “RuntimeError: expected scalar type Half but found Float”. And I hope you can help me with this. :slight_smile:

The specs I’m using:
Ubuntu 20.4
Nvidia driver: 450.36.06
Cuda: 10.2
cudnn: 7.6.5
PyTorch-Nighlty : 1.7.0.dev20200721

The network is fairly simple:

scaler = GradScaler()
for epoch in trange(epochs):
    for (dataE, dataM, dataC, target) in trainLoader:
        with autocast():
            zex = AutoencoderE(dataE)
            zmx = AutoencoderM(dataM)
            zcx = AutoencoderC(dataC)
        
            ...
        
        scaler.scale(loss).backward()
        
        for optim in (optim_clas, optimE, optimM, optimC):
            scaler.step(optim)

        scaler.update()

The network classes:

class Encoder(nn.Module):
    def __init__(self, input_size, output_size, dropout_rate):
        super(Encoder, self).__init__()
        self.En = torch.nn.Sequential(
            nn.Linear(input_size, output_size),
            nn.BatchNorm1d(output_size),
            nn.ReLU(),
            nn.Dropout(dropout_rate))

    def forward(self, x):
        output = self.En(x)
        return output


class Classifier(nn.Module):
    def __init__(self, input_size, rate):
        super(Classifier, self).__init__()
        self.FC = torch.nn.Sequential(
            nn.Linear(input_size, 1),
            nn.Sigmoid())

    def forward(self, x):
        return self.FC(x)

Stacktrace:

Traceback (most recent call last):
  File "/media/tony/Volume/work/multi-omics/multi-omics_analysis/molti-omics_analysis_cetuximab.py", line 325, in <module>
    main()
  File "/media/tony/Volume/work/multi-omics/multi-omics_analysis/molti-omics_analysis_cetuximab.py", line 240, in main
    zex = AutoencoderE(dataE)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/tony/Volume/work/multi-omics/multi-omics_analysis/encoder.py", line 14, in forward
    output = self.En(x)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 91, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/functional.py", line 1682, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: expected scalar type Half but found Float

Process finished with exit code 1
2

Data Type Inconsistency
scalar type is not Half (torch.float16), but float (torch.float32)
You should convert scalar to Half like this:
scalar = scalar.to(torch.float16)

2 Likes
3

Thanks. I changed it, but now I get “RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED”

4

This Error can be caused by various factors, which may be solved according to solutions returned by Search engines like Google or Baidu.

5

I found my error. I was thinking that tensor.to(device) works inplace. Now it works without an explicit conversion

1 Like