Hi, all
How to set ‘nan’ in Tensor to 0? Now I have a extremely inefficient method:
my_tensor_np = my_tensor.cpu().numpy()
my_tensor_np [np.isnan(my_tensor_np )] = 0
my_tensor.copy_(torch.from_numpy(my_tensor_np ).cuda())
But copy tensor between gpu and cpu takes lots of time, so I need a more efficient way.
Anyone can help me? Thanks a lot!
2 Likes
iamalbert
(Albert Zhuang)
June 11, 2017, 6:47am
2
NaN means a value which is undefined or unrepresentable. In most cases it makes no sense to simply set NaNs to zero.
Thank you, iamalbert
A paper I recently read use this trick but implemented in Theano. I want to re-implement their algorithm in PyTorch.
I think np.isnan is a useful function, but torch doesn’t implement it, is there a efficient solution?
Thank you!
iamalbert
(Albert Zhuang)
June 11, 2017, 7:09am
4
It’s simple, a != a
will give you a ByteTensor, indicating the positions of NaNs
>>> b
nan nan -0.8395
nan nan nan
-1.7921 nan 0.1864
[torch.FloatTensor of size 3x3]
>>> b != b
1 1 0
1 1 1
0 1 0
[torch.ByteTensor of size 3x3]
you can use b[b != b] = 0
to set all NaNs to zero.
41 Likes
Ben
September 2, 2017, 3:09am
5
Seems not work now.
I tried below code in v0.20:
a = torch.Tensor([1,2,3,4,5])
b = 0.0
c = a / b
c != c
got all 0s.
Is there any function like np.nan_to_num?
1 Like
colesbury
(Sam Gross)
September 2, 2017, 5:25am
6
@Ben , that’s because c
in your example is all Inf
, not NaN
. Albert’s suggestion works:
a = torch.Tensor([float('NaN'), 1, float('NaN'), 2, 3])
print(a)
a[a != a] = 0
print(a)
5 Likes
Hi @Chen-Wei_Xie ,
torch.isnan() was implemented earlier this year. So now you can set the NaN’s to 0’s using my_tensor[torch.isnan(my_tensor)] = 0
Cheers
11 Likes
See discussion in ReLU turns nan into zeros . As of pytorch 4.1 ReLU can’t be used for this anymore though. I’ve asked about it there.
For completness I’ll copy my answer:
Thank you, I’ll continue the discussion here. This works both forwards/backwards on CPU:
import torch
model = torch.nn.Linear(10,10)
x = torch.ones(10,10).detach()
x[0,0] = x[0,0]+float('NaN')
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
model.zero_grad()
y = model(x)
y[y!=y] = 0
loss = y.sum()
loss.backward()
optimizer.step()
Could someone verify if it works on GPU?
Also, anyone has insights on the computational cost vs the ol’ ReLU-hack?
1 Like
wonderful!thanks very much.
Modifying tensors in-place can cause issues with backprop. Here is my solution, since this is still ranked highly on google:
safe_tensor = torch.where(torch.isnan(my_tensor), torch.zeros_like(my_tensor), my_tensor)
4 Likes
Nikronic
(Nikan Doosti)
June 11, 2021, 5:48am
13
Hi,
From version 1.8.1, torch.nan_to_num — PyTorch 1.8.1 documentation is now available. It replaces NaN
, positive infinity, and negative infinity values in input
with the values specified by nan
, posinf
, and neginf
, respectively.
1 Like