Network not converging

1

I used various structerees for my network but none of it converged better than 40% so I decided to check if it converges to a constant, it does.
Then I checked if it converges in a “do nothing” case, the net takes 27 inputs and returns a single float, I set all input values to the same as the expected output value and even in this case nothing converges.

a view tested structures :

self.conv = nn.Sequential(
                nn.Conv1d(1, 1, kernel_size=3, padding=1),
                nn.AvgPool1d(3, padding=1),
                nn.Linear(9, 9),
                nn.Conv1d(1, 1, kernel_size=3, padding=1),
                nn.AvgPool1d(3, padding=1),
                nn.Linear(3, 3),
                nn.Conv1d(1, 1, kernel_size=3, padding=1),
                nn.AvgPool1d(3, padding=1)
            )

self.dumb = nn.Sequential(
                nn.Linear(27, 27),
                nn.Linear(27, 1),
            )

self.dumb = nn.Sequential(
                nn.Linear(27, 1),
            )

I used optim.Adam with learning rates from 0.1 to 0.001 and a batch size of 16 and 64 sets in total to cause overfitting
What could solve this issue ?

2

Could you please share a small script with your dummy data and the model that does not converge?

3

you don’t have nonlinearities in your network. in fact all three versions are equivalent.