How to set dtype for NN layers?

1

I have training data available as dtype = torch.float64 and I want to use this data type to train my model. Now it seems that the nn.Layer (e.g. nn.Linear) interface doesn’t offer the possibility of specifying a data type. Right now I get RuntimeError: Expected object of type torch.DoubleTensor but found type torch.FloatTensor because the dtypes are incompatible. How can I set the dtype of my NN layers to torch.float64? Do I have to assign the weights manually?

1 Like
2

You can cast the whole layer to a new dtype using:

lin = nn.Linear(10, 10).double()
6 Likes
3

Thanks, that works! Is there a way to set it as the default for all layers or do I have to cast each layer manually? Something like a context manager would be handy; does it exist?

4

You can call it on the whole model which will cast all layers and parameters to the type.
Alternatively, you could set the default type at the beginning of your script using torch.set_default_dtype().

5 Likes
5

How can this be done in C++?

Also, in general, how can the .double() be achieved for a torch::Tensor object in C++?

Thank you.

1 Like
6

You could use tensor = tensor.to(at::kDouble); to transform the tensor to an FP64 one and

AutoDefaultDtypeMode dtype_mode(default_dtype);

should work to define the default type.

3 Likes
7

How can the dtype of a layer be changed in c++?

ex: auto layer = torch::nn::Linear(4, 5); // creates a linear layer that works on float32 tensors.

How can I make a Linear layer with works on int8 tensors?

8

looks like torch.float16 doesn’t work?

9

It seems to work for me as seen here:

lin = nn.Linear(10, 10)
print(lin.weight.dtype)
# torch.float32

torch.set_default_dtype(torch.float16)
lin = nn.Linear(10, 10)
print(lin.weight.dtype)
# torch.float16

However, I don’t know if all operations support a change in the default dtype as I think it can be risky if e.g. integer types are expected. Could you post more information about your errors, please?