Input type (c10::Half) and bias type (float) should be the same

[fibikong (경민 유)]

ValueError : Attemting to unscale fp16 Gradients

Hello all, I am trying to train an LSTM in the half-precision setting. The LSTM takes an encoded input from a pre-trained autoencoder(Not trained in fp16). I am using torch.amp instead of apex and scaling the losses as suggested in the documentation. Here is my training loop - def train_model(self, model, dataloader, num_epochs): model.cuda() least_loss = 5 model.train() optimizer = torch.optim.Adam(model.parameters(), lr =1e-5) scaler = amp.GradSca…

I had same problem above because of model.to(torch.float16).
But if I remove that part, I cannot match the data type of inner layer’s input and weights.

RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
Input type (c10::Half) and bias type (float) should be the same
  File "/home/cvnar1/anaconda3/envs/tgve/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
  File "/home/cvnar1/anaconda3/envs/tgve/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/cvnar1/Desktop/official_code/loveu-tgve-2023/tuneavideo/models/resnet.py", line 15, in forward
    x = super().forward(x)
  File "/home/cvnar1/anaconda3/envs/tgve/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/cvnar1/anaconda3/envs/tgve/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/cvnar1/Desktop/official_code/loveu-tgve-2023/tuneavideo/models/unet.py", line 381, in forward
    sample = self.conv_in(sample)
  File "/home/cvnar1/anaconda3/envs/tgve/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/cvnar1/anaconda3/envs/tgve/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/cvnar1/Desktop/official_code/loveu-tgve-2023/train_tuneavideo.py", line 348, in main
    unet_output= accelerator.unwrap_model(model).unet(noisy_latents, timesteps, encoder_hidden_states)
  File "/home/cvnar1/Desktop/official_code/loveu-tgve-2023/train_tuneavideo.py", line 451, in <module>
    main(**OmegaConf.load(args.config))
  File "/home/cvnar1/anaconda3/envs/tgve/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/cvnar1/anaconda3/envs/tgve/lib/python3.10/runpy.py", line 196, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
RuntimeError: Input type (c10::Half) and bias type (float) should be the same

please tell me how to fix this.

[ptrblck]

Could you post a minimal and executable code snippet raising the issue without your manual casting?