I’m new to pytorch. I wanna create an uninitialized tensor as one of parameters in my model. I haven’t gone to the process of training or backpropagation yet. I just simply created a tensor with torch.empty() but got nan values by chance.
Here is the code and result. I sometimes got nan values inside the tensor sometimes not. It seems that the higher dimensions of size, the more likely to get nan.
u = torch.empty(size=(15,15,15))
torch.sum(u.isnan()==True)
Out[243]: tensor(6)
u = torch.empty(size=(15,15,15))
torch.sum(u.isnan()==True)
Out[245]: tensor(0)
Use torch.sum(u.isnan()==True) to check the tensor, you can see that the first time u has 6 nan values, the second time it doesn’t have nan values at all.
I wonder what is going on there. how can I completely avoid getting nan by creating an uninitialized tensor. If I replace those nan values with a specific value (e.g. zero), will it change the property of “uninitialized tensor” so to affect my model’s training process?
This is expected since you are using uninitialized memory for the tensor which may contain any value including NaN.
Initialize tensors via e.g. torch.randn to draw samples from a Normal distribution.
Thank you so much!
My purpose of creating this tensor data is to train it as a parameter (like a weight matrix) in my neural network, I’m still not sure if using torch.randn to initialize the tensor would be OK to train the parameter. (since i saw other people noramlly do torch.empty).
I’ve read the other comments from you about the difference between initialized and unintialized tensor, but still I can’t figure out which function or type of data is proper for my situation.
sorry if it sounds like a stupid question, as I’m new to pytorch and neural network modeling…
You should never use uninitialized memory as it can contain invalid data as already seen in your example. torch.empty is a fast way to initialize memory but requires you to fill it afterwards. E.g. it can be useful to preallocate a tensor which the result will be written to later. You should however never use it without a proper initialization.
A trainable parameter is initialized randomly before it gets trained. torch.randn would be one proper way but you can also use any other random initialization method from e.g. torch.nn.init. Note the difference between random initialization and uninitialized memory.
you’re right. A trainable parameter is initialized randomly before it gets trained. In that sense torch.randn would be a proper way to do the work.
Truly appreciate your help!