Regression with nn.Sequential network fail

autograd
1

Hi there I was trying to use nn.Sequential module in a simple regression problem, just a line. The thing is that I print MSELoss every iteration and it is trying to converge at 1, intead of the values I am wondering.

Here is the code:

NN = nn.Sequential(
    nn.Linear(1,3),
    nn.Sigmoid(),
    nn.Linear(3,4),
    nn.Sigmoid(),
    nn.Linear(4,7),
    nn.Sigmoid(),
    nn.Linear(7,1),
    nn.Sigmoid()
)

criterion = nn.MSELoss()
optimizer = optim.Adam(NN.parameters(), lr=0.01)

X = torch.rand(200,1)
y = 2*X + 3

print(X.shape,y.shape)
for i in range(1000):  
    NN.train()
    optimizer.zero_grad()
    out = NN(X)
    #print(out,y)
    loss = criterion(y,out)
    loss.backward()
    optimizer.step()

    with torch.no_grad():
        print ("#" + str(i) + " Loss: " + str(loss.item()))

If I try this to check the results:

newX = []
newY = []
for i in range(5):
    r = random.uniform(0, 1)
    newX.append(r)
    newY.append(NN(torch.tensor([r])).item())

I get [0.9996629953384399, 0.9996600151062012, 0.9996635913848877, 0.9996637105941772, 0.9996615648269653].

I have tryed lot of things changing every on the network but don’t know what is happening.

2

Hi,

There is an issue with your model architecture, your target values are all greater than one (because of 2x+3 operation) and last operation in your network is nn.Sigmoid() which will return values in the range 0 to 1 so your model would never be able to converge.

3

Oh,I feel so silly its true. And what should I do, change the structure of the network or normalize inputs and outputs values?

4

Just remove last nn.Sigmoid() operation from your model.

5

That works. Thank you really much!!