# Neural Network cannot approximate sin function

**URL:** <https://discuss.pytorch.org/t/neural-network-cannot-approximate-sin-function/41306>\
**Category:** Uncategorized\
**Created:** [March 31, 2019, 8:29am UTC](https://discuss.pytorch.org/t/neural-network-cannot-approximate-sin-function/41306 "2019-03-31T08:29:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cero](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ce73a5/32.png) [@Cero](https://discuss.pytorch.org/u/Cero)\
**Post date:** [March 31, 2019, 8:29am UTC](https://discuss.pytorch.org/t/neural-network-cannot-approximate-sin-function/41306/1 "2019-03-31T08:29:05Z")

</div>

I’m trying to learn how to use Pytorch and wanted to start with a really simple test.  
So I’ve created a function f(x) = np.sin(x \* 50) \* 500 + x \* 2000  
and I want a Feedforward Neural Network to approximate this in the range [0,1].

I’ve created 5000 equally distributed samples and just trained the network on all samples. I do not care if it overfits, its just for testing.  
From my tests it seems like my network approximates just a linear function or the mean of my samples…  
I’ve tried to play with the amount of layers but even a network with 3 layers with around 100 neurons wasn’t able to find a good solution.

So I’m pretty sure I’m using Pytorch incorrectly but I cannot find my error.  
Below you can find my SourceCode, I think the most important part are the two for loops which contain the training procedure.

> **Code**
>
> ```auto
> import torch
> import torch.nn as nn
> import torch.optim as optim
> import numpy as np
> import matplotlib.pyplot as plt
> 
> # the target function
> def func(x):
> return np.sin(x * 50) * 500 + x * 2000
> 
> # plot the target function
> x_steps = np.linspace(0, 1, 5000)
> y_steps = func(x_steps)
> plt.plot(x_steps, y_steps)
> 
> # Create the model, optimizer and the loss function
> model = nn.Sequential(
> nn.Linear(1, 64),
> nn.ReLU(),
> nn.Linear(64, 32),
> nn.ReLU(),
> nn.Linear(32, 1))
> optimizer = optim.Adam(model.parameters(), 0.1)
> loss_func = nn.MSELoss()
> 
> # Create random index "batches"
> indexes = np.array(list(range(len(x_steps))), dtype=np.int)
> np.random.shuffle(indexes)
> batches = torch.split(torch.from_numpy(indexes), 64)
> 
> # Training
> for epoch in range(100):
> for batch in batches:
> x_batch = x_steps[batch]
> y_batch = y_steps[batch]
>         
> prediction = model(torch.Tensor(x_batch.reshape((-1,1))))
> loss = loss_func(prediction, torch.Tensor(y_batch.reshape((-1,1))))
>     
> optimizer.zero_grad()
> loss.backward()
> optimizer.step()
>         
> print(loss.detach().numpy())
> 
> # Test if the function was approximated
> predictions = model(torch.Tensor(np.reshape(x_steps, (-1,1))))
> plt.plot(x_steps, y_steps)
> plt.show()
> plt.plot(x_steps, predictions.detach().numpy())
> plt.show()
> 
> ```

---

<div class="post-metadata">

**Author:** ![Tony-Y](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tony-y/32/8373_2.png) [@Tony-Y](https://discuss.pytorch.org/u/Tony-Y)\
**Post date:** [March 31, 2019, 9:25am UTC](https://discuss.pytorch.org/t/neural-network-cannot-approximate-sin-function/41306/2 "2019-03-31T09:25:09Z")

</div>

My PyTorch example could help you.

> **[Tony-Y/PotFit](https://github.com/Tony-Y/PotFit)**
>
> A PyTorch Example of Potential Fitting . Contribute to Tony-Y/PotFit development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![justusschock](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/justusschock/32/4637_2.png) [@justusschock](https://discuss.pytorch.org/u/justusschock)\
**Post date:** [March 31, 2019, 10:14am UTC](https://discuss.pytorch.org/t/neural-network-cannot-approximate-sin-function/41306/3 "2019-03-31T10:14:51Z")

</div>

How long did you train the network? Maybe the L1 loss would be a better oder in this case?

---

<div class="post-metadata">

**Author:** ![Cero](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ce73a5/32.png) [@Cero](https://discuss.pytorch.org/u/Cero)\
**Post date:** [March 31, 2019, 10:53am UTC](https://discuss.pytorch.org/t/neural-network-cannot-approximate-sin-function/41306/4 "2019-03-31T10:53:38Z")

</div>

I’m currently training the network with a training set of 5000 samples and a batch size of 64 for around 200-500 epochs.  
Additionally the training data has no noise.

I thought it would be simple to approximate the data. I’ve tried it with L1Loss and MSELoss also I switched between SGD and Adam but all of them either converge to an linear function or the mean.  
Even when I increase the training data and epochs.

@Tony-Y I appreciate your example but it seems like you additionally use the derivative?  
After changing the target function to x^2 my code seems to work fine. Even some other functions are perfectly approximated. Does that mean that neural networks are not able to easily approximate periodic functions?

---

<div class="post-metadata">

**Author:** ![justusschock](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/justusschock/32/4637_2.png) [@justusschock](https://discuss.pytorch.org/u/justusschock)\
**Post date:** [March 31, 2019, 11:11am UTC](https://discuss.pytorch.org/t/neural-network-cannot-approximate-sin-function/41306/5 "2019-03-31T11:11:40Z")

</div>

This should usually be sufficient.

Theoretically, they should be able to approximate any function according to the [universal approximation theorem](https://en.m.wikipedia.org/wiki/Universal_approximation_theorem?wprov=sfla1). If I have some time later this day, I’ll try myself to get it working.

---

<div class="post-metadata">

**Author:** ![Tony-Y](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tony-y/32/8373_2.png) [@Tony-Y](https://discuss.pytorch.org/u/Tony-Y)\
**Post date:** [March 31, 2019, 11:33am UTC](https://discuss.pytorch.org/t/neural-network-cannot-approximate-sin-function/41306/6 "2019-03-31T11:33:42Z")

</div>

Why my example uses the derivative is because I considered a problem with force data in addition to potential data. My example uses only 100 sample points because of the use of both potential and force data. If you use 5 layers and 500 hidden units for your model like my code, and train it for several thousand steps (20,000 steps for my code), can your modified model learn the sine function?

---

<div class="post-metadata">

**Author:** ![Cero](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ce73a5/32.png) [@Cero](https://discuss.pytorch.org/u/Cero)\
**Post date:** [April 1, 2019, 9:37am UTC](https://discuss.pytorch.org/t/neural-network-cannot-approximate-sin-function/41306/7 "2019-04-01T09:37:42Z")

</div>

After some testing I’ve realized that my learning rate was always too small.  
When setting the learning rate to 0.1 the AdamOptimizer was able to approximate a good function.

The small learning rate was propably responsible for the optimizer getting stuck in an local optima and from which he could not escape.