I am trying to implement and train a sparse network that looks like the following:
My understanding was that it is very similar to a 1D convolutional network with a single channel. So this is how I implemented it:
nn.Sequential( nn.Conv1d(in_channels=1, out_channels=1, kernel_size=3, bias=False), nn.Sigmoid(), nn.Linear(6, 1, bias=False), nn.Sigmoid() )
Data looks like the following:
X = [ [1, 0, 1, 1, 0, 1, 0, 0], [1, 1, 1, 1, 0, 0, 1, 1], ... ] y = [ , , ... ]
However, when I train it with simple inputs (like shown above) and examine the weights, they don’t look like I understood/expected them to be. So I reckoned perhaps that my architecture was incorrect. Can someone please let me know if this looks right?
PS: What I actually have is a hidden layer with the same number of dimensions as the input layer, so I have set
padding=1 in my actual code. The
padding=0 version was easier to show in a diagram.