It appears LSTMCell and GRUCell have dedicated autograd operations that type-check inputs. I should add those to the FP16 cast list.
Filed issue https://github.com/pytorch/pytorch/issues/42605. Should be a straightforward fix, thanks for reporting this. I thought the cells were implemented in terms of autograd-exposed primitive ops (eg. gemms) that autocast already covers.