Simple Linear/Sigmoid Network not learning

[KFrank]

Hello Samy!

@SamyGarib

train a network as an XOR gate.

I think it is rather difficult to train a small network to reproduce an XOR
gate.

XOR has cross-terms that are, in some sense, highly non-linear. This
is an imperfect fit for a “narrow,” “shallow” network.

Please take a look at this thread about a nearly identical problem:

Simple pattern is not detected by simple linear model

I have a very simple example of 6 training data with 6 labels. I am trying to fit a simple logistic regression to it, and hoping it should over-fit with 0 loss, but the model does not converge. Here is the rule, i am hoping the model to learn: When input is [1., 0., 0., 1.] , the label is 1 When input is [0., 1., 1., 0.] , the label is 1 When input is [1., 0., 1., 0.] , the label is 0 When input is [0., 1., 0., 1.] , the label is 0 is such pattern something a non-linear model cannot capt…

Some further comments in line:

I have tried with many learning rates and weights initialization.

Continue to experiment with learning rates and many random
initializations, and continue to train for many iterations. It looks
like it is easy for the training of this kind of network to get stuck
away from a good solution.

        def __init__(self):
            super(Network, self).__init__()
            self.x1 = nn.Linear(2,4)
            self.s1 = nn.Sigmoid()
            self.x2 = nn.Linear(4,1)

Play around with making your network deeper by adding one or
more hidden layers, e.g., something like:

            self.x1 = nn.Linear(2,4)
            self.x2 = nn.Linear(4,2)
            self.x3 = nn.Linear(2,1)

or:

            self.x1 = nn.Linear(2,4)
            self.x2 = nn.Linear(4,4)
            self.x3 = nn.Linear(4,2)
            self.x4 = nn.Linear(2,1)

Try making your network wider:

            self.x1 = nn.Linear(2,16)
            self.x2 = nn.Linear(16,1)

            self.s2 = nn.Sigmoid()

I would suggest getting rid of the Sigmoid. My intuition is that it will
actually hurt here – you have to saturate the Sigmoid (large network
parameters) to match closely your val_expected. Return the result
of your last Linear layer as the output of your network.

            criterion = nn.MSELoss()

Conventional wisdom suggests that BCEWithLogitsLoss would be
a more natural loss function for your problem. You should be able
to get your network to train with MSELoss, but maybe not as easily.

            optimizer = torch.optim.SGD(self.parameters(), lr=0.01)

Consider using momentum with SGD (as well as various learning
rates). You might also play around with other optimizers, such as
Adam.

I would first remove the final Sigmoid, use BCEWithLogitsLoss,
and make the network deeper and/or wider.

Good luck.

K. Frank