# Well-formed input into a simple linear layer, output Nan

**URL:** <https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720>\
**Category:** Uncategorized\
**Created:** [March 29, 2020, 5:40pm UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720 "2020-03-29T17:40:38Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![wu\_hr](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wu_hr/32/13058_2.png) [@wu\_hr](https://discuss.pytorch.org/u/wu_hr)\
**Post date:** [March 29, 2020, 5:40pm UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/1 "2020-03-29T17:40:38Z")

</div>

I input well-formed data into a simple linear layer with normal weights and bias, the output has some ‘nan’ in it. This only happens on Ubuntu18 + PyTorch1.4.0, but on Win10 + PyTorch1.4.0 or Colab, the linear layer works well.  
On Ubuntu:

```auto
import torch
import torch.nn as nn
model = nn.Linear(6, 8)
x = torch.randn((4, 6))
y = model(x)
print(x)
print(y)
print(model.weight)
print(model.bias)

```

I get:

```auto
tensor([[0.4134, -1.0348, -0.7237, 0.6970, 0.0487, 0.1217],
        [1.2217, 0.2465, -0.0832, 0.5530, -0.5716, -0.4916],
        [-0.3497, -0.1285, -1.3026, -0.0845, -0.5295, -0.7776],
        [1.2792, 0.1692, -1.3023, -1.9925, -0.6869, 0.8478]])
tensor([[ 3.0334e-01, 1.9168e+05, -4.3615e+31, 3.7255e+04, -1.5095e+23,
         -6.4485e+34, 1.0371e+34, nan],
        [ 3.0324e-01, -7.7407e+05, 1.0390e+31, 4.2809e+03, 6.0958e+23,
         -7.4157e+33, -4.1881e+34, nan],
        [ 3.0343e-01, -1.2243e+06, -5.4147e+30, 6.7047e+04, 9.6410e+23,
         -1.1607e+35, -6.6239e+34, nan],
        [ 3.0323e-01, 1.3349e+06, 7.1316e+30, 6.7047e+04, -1.0513e+24,
         -1.1605e+35, 7.2227e+34, nan]], grad_fn=<AddmmBackward>)
tensor([[-0.3934, 0.2317, -0.3121, -0.1111, -0.2597, -0.2182],
        [-0.2294, 0.3308, -0.1488, -0.1515, 0.1625, 0.0586],
        [-0.3984, 0.1251, 0.0899, 0.0807, 0.0110, 0.1535],
        [-0.2065, -0.3367, 0.1329, 0.4042, -0.0387, -0.3794],
        [0.2129, 0.1050, -0.2263, -0.3991, 0.3312, 0.3797],
        [0.3508, 0.1268, -0.2395, 0.0672, 0.3916, 0.0131],
        [-0.1437, -0.4070, -0.0663, -0.1292, -0.0127, -0.0040],
        [0.0037, -0.0610, -0.0200, 0.2865, -0.0804, 0.1235]],
       requires_grad=True)
tensor([0.3034, -0.0240, 0.1232, 0.3308, 0.1517, -0.3978, -0.2769, 0.0284],
       requires_grad=True)

```

When I test on Colab or Win10, it returns y like:

```auto
tensor([[0.8947, 0.7973, 0.0000, 0.4691, 0.8554, 0.4177, 1.5419, 0.0241, 0.7379,
         0.0000, 1.6729, 0.0000, 0.0000, 1.1157, 0.0000, 0.7253],
        [0.4217, 0.1511, 0.0000, 0.9463, 0.0000, 0.0000, 0.0000, 1.0647, 0.0000,
         0.9710, 0.0000, 1.5806, 0.0000, 0.0000, 1.3374, 0.0059],
        [0.0000, 0.7281, 1.6656, 0.0000, 0.0000, 0.6633, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0893, 1.3113, 0.0000, 0.5653, 0.0000],
        [0.3799, 0.0000, 0.0000, 0.2625, 0.9877, 0.6430, 0.1361, 0.5216, 0.9120,
         1.0127, 0.0000, 0.0000, 0.6246, 0.7964, 0.0000, 0.9006]],
       grad_fn=<ReluBackward0>)

```

Could someone help?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 30, 2020, 12:52am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/2 "2020-03-30T00:52:55Z")

</div>

How did you define the `y` tensor, which contains the NaN values?

---

<div class="post-metadata">

**Author:** ![wu\_hr](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wu_hr/32/13058_2.png) [@wu\_hr](https://discuss.pytorch.org/u/wu_hr)\
**Post date:** [March 30, 2020, 4:01am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/3 "2020-03-30T04:01:42Z")

</div>

Sorry, I forgot to type it:

```auto
y = model(x)

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 30, 2020, 4:16am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/4 "2020-03-30T04:16:05Z")

</div>

Based on the values, it looks as if `y` is using uninitialized values.  
Do you get the NaN (and high value) outputs using this code?

```python
x = torch.tensor([[0.4134, -1.0348, -0.7237, 0.6970, 0.0487, 0.1217],
        [1.2217, 0.2465, -0.0832, 0.5530, -0.5716, -0.4916],
        [-0.3497, -0.1285, -1.3026, -0.0845, -0.5295, -0.7776],
        [1.2792, 0.1692, -1.3023, -1.9925, -0.6869, 0.8478]])

model = nn.Linear(6, 8)

with torch.no_grad():
    model.weight = nn.Parameter(torch.tensor([[-0.3934, 0.2317, -0.3121, -0.1111, -0.2597, -0.2182],
        [-0.2294, 0.3308, -0.1488, -0.1515, 0.1625, 0.0586],
        [-0.3984, 0.1251, 0.0899, 0.0807, 0.0110, 0.1535],
        [-0.2065, -0.3367, 0.1329, 0.4042, -0.0387, -0.3794],
        [0.2129, 0.1050, -0.2263, -0.3991, 0.3312, 0.3797],
        [0.3508, 0.1268, -0.2395, 0.0672, 0.3916, 0.0131],
        [-0.1437, -0.4070, -0.0663, -0.1292, -0.0127, -0.0040],
        [0.0037, -0.0610, -0.0200, 0.2865, -0.0804, 0.1235]]))
    model.bias = nn.Parameter(
        torch.tensor([0.3034, -0.0240, 0.1232, 0.3308, 0.1517, -0.3978, -0.2769, 0.0284]))

y = model(x)
print(y)

```

---

<div class="post-metadata">

**Author:** ![wu\_hr](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wu_hr/32/13058_2.png) [@wu\_hr](https://discuss.pytorch.org/u/wu_hr)\
**Post date:** [March 30, 2020, 4:43am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/5 "2020-03-30T04:43:48Z")

</div>

The same condition. On Ubuntu, I got:

```auto
tensor([[ 3.0335e-01, 1.9162e+05, -4.3614e+31, 3.7255e+04, -1.5090e+23,
         -6.4485e+34, 1.0367e+34, nan],
        [ 3.0325e-01, -7.7402e+05, 1.0389e+31, 4.2792e+03, 6.0954e+23,
         -7.4128e+33, -4.1879e+34, nan],
        [ 3.0344e-01, -1.2243e+06, -5.4160e+30, 6.7047e+04, 9.6416e+23,
         -1.1607e+35, -6.6243e+34, nan],
        [ 3.0325e-01, 1.3349e+06, 7.1314e+30, 6.7045e+04, -1.0512e+24,
         -1.1604e+35, 7.2223e+34, nan]], grad_fn=<AddmmBackward>)

```

On Colab:

```auto
tensor([[0.0102, -0.4440, -0.1605, 0.7313, 0.0790, -0.1432, 0.0417, 0.3183],
        [0.1001, -0.4158, -0.3773, 0.4166, -0.1402, -0.1112, -0.6095, 0.1632],
        [1.1343, 0.0887, -0.0027, 0.5545, -0.0784, -0.4480, -0.0672, -0.0167],
        [0.4606, 0.1722, -0.5206, -1.2638, 1.6261, -0.0075, -0.1805, -0.3621]],
       grad_fn=<AddmmBackward>)

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 30, 2020, 4:46am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/6 "2020-03-30T04:46:06Z")

</div>

Could you run this script to collect your setup information (from the GitHub issue template):

```python
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

```

---

<div class="post-metadata">

**Author:** ![wu\_hr](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wu_hr/32/13058_2.png) [@wu\_hr](https://discuss.pytorch.org/u/wu_hr)\
**Post date:** [March 30, 2020, 4:51am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/7 "2020-03-30T04:51:58Z")

</div>

The envs are listed here:

```auto
Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce MX150
Nvidia driver version: 440.59
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.14.3
[pip3] numpydoc==0.8.0
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas 1.0 mkl  
[conda] mkl 2018.0.2 1  
[conda] mkl-service 1.1.2 py36h17a0993_4  
[conda] mkl_fft 1.0.1 py36h3010b51_0  
[conda] mkl_random 1.0.1 py36h629b387_0

```

---

<div class="post-metadata">

**Author:** ![wu\_hr](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wu_hr/32/13058_2.png) [@wu\_hr](https://discuss.pytorch.org/u/wu_hr)\
**Post date:** [March 30, 2020, 5:45am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/8 "2020-03-30T05:45:54Z")

</div>

Sovled it by upgrading numpy. Thanks for your help.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 30, 2020, 6:24pm UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/9 "2020-03-30T18:24:53Z")

</div>

Oh wow, how did you figure it out?  
I’m not sure how numpy might interact with the linear layer and create NaNs. ☹

---

<div class="post-metadata">

**Author:** ![wu\_hr](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wu_hr/32/13058_2.png) [@wu\_hr](https://discuss.pytorch.org/u/wu_hr)\
**Post date:** [March 31, 2020, 1:16am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/10 "2020-03-31T01:16:55Z")

</div>

Pytorch’s computation may depend on numpy and I found my numpy version had not been updated for a long time. Just gave it a try. 🤣

---

<div class="post-metadata">

**Author:** ![SIQI\_ZENG](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siqi_zeng/32/29820_2.png) [@SIQI\_ZENG](https://discuss.pytorch.org/u/SIQI_ZENG)\
**Post date:** [October 15, 2020, 6:16am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/11 "2020-10-15T06:16:04Z")

</div>

It’s very strange, I met the same situation like you.I trian VGG16 with datasets CIFAR-10 in ubuntu16.04 and after first backward and step, I get a loss NAN and grad is also NAN. Then I try to change VGG16 layer like modify the parameters in nn.Linear(in,out), but I found when I change original FC layer from nn.Linear(4096,num\_classes =10) to nn.Linear(4096,293),it works and no NAN. And delete all FC layer of VGG16, it still works well. After checking my numpy version(1.18.1), I update numpy to the latest version 1.19.2.Wow !!! no more Nan exsit in my experiment !!! I still don’t understand why this problem happens in ubuntu not Windows

---

<div class="post-metadata">

**Author:** ![shihanmax](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shihanmax/32/34238_2.png) [@shihanmax](https://discuss.pytorch.org/u/shihanmax)\
**Post date:** [November 11, 2021, 11:12am UTC](https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/12 "2021-11-11T11:12:58Z")

</div>

Upgrading numpy from 1.18.5 to 1.21.4 not working for me, torch 1.6.0 on Linux running a transformer model.