Feedback on my 1D CNN needed, dimension and no convergence issue

Hello Torchers,

I have an issue with the data input from the dataloader wrapper and its dimension not adapted to my CNN inputs.

I am using a 1D CNN for an interpolation problem right now. For debugging purpose I have set the input size to 1 node and the CNN must learn the identity function. (is that meaningful btw?, target/label = input for the identity function). I am also training by batches.

Let me tell you first what I am doing before I ask my questions -

The data is constructed with the following custom dataset constructor and dataloader wrapper:

class custom_dataset(torch.utils.data.Dataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, examples, labels):
        'Initialization'
        
        self.labels = labels
        self.examples = examples

  def __len__(self):
        'Denotes the total number of samples'
        return len(self.examples)

  def __getitem__(self, index):
        'Generates one sample of data'
        # Select sample

        # Load data and get label
        X = self.examples [index]
        y = self.labels [index]

        return X, y

the wrapping happens here, in comments my data shape:

    #with train_example.shape = torch.Size([57600])
    #and train_label.shape = torch.Size([57600])
    train_example = np.asarray(train_example)
    train_label = np.asarray(train_label)
    trainset = custom_dataset(train_example,train_label)

the CNN look like this:

class ConvNet1D(nn.Module):
    def __init__(self,winlength_size, dropout=0.2):
        super(ConvNet1D, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv1d(1, 16, kernel_size=1, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=1))
        self.layer2 = nn.Sequential(
            nn.Conv1d(16, 32, kernel_size=2, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=1))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(64, 20)
        self.fc2 = nn.Linear(20, 1)



    def forward(self, x):

        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out

finally the training happens in the following snippet:

'''optimizer is adam'''
'''criterion is MSE loss, reduce = True'''

   for batch_idx, data in enumerate(train_loader):
        inputs = data[0]
        labels = data[1]
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion (outputs, labels)
        loss.backward()
        optimizer.step()

first issue that came, the dimension : run with those snippets, with a batch_size of 16, the code would give me the following error:

expected 3-dimensional input for 3-dimensional weights 16, but got 1 dimensional input of size 16

by adding new dimensions on my input and label data, I could solve the 1D ConvNet input structure issue which must be 3 dimensional, for channel numbers and lengths.
It is done as following:

train_example = train_example.reshape(train_example[0].shape, 1,1)
train_labels = train_labels.reshape(train_labels[0].shape,1,1)

now I have the following mistake:

Using a target size: torch.Size([16,1,1]) that is different to the input size : torch.Size([16,1]). This will likely lead to incorrect results due to error.

The input dimension somehow reduced to 2, but I cannot change the input dimension because of the conditions on my inputs given by the convnet, so I need to work with the label dimensions:

train_labels = train_labels.reshape(train_labels[0].shape, 1)

here I would have a running code, but I don’t have a clear convergence:

#3 firsts loss by batch:
3_losses= [0.31,0.181,0.143]

#!PSEUDO CODE! The rest:
the_rest = span(random between 0.005 and 0.09)

so I changed the CNN to

class ConvNet1D(nn.Module):
    def __init__(self,winlength_size, dropout=0.2):
        super(ConvNet1D, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv1d(1, 16, kernel_size=1, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=1))
        self.layer2 = nn.Sequential(
            nn.Conv1d(16, 32, kernel_size=2, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=1))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(64, 3000)
        self.fc2 = nn.Linear(3000, 150)
        self.fc3 = nn.Linear(150,1)



    def forward(self, x):

        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)
        return out

to get better results.

now my questions:

  • Is it okay for the CNN to have so many nodes in the fully connected layouts?
  • Is it okay for the CNN to have more than 2 layers in the fully connected part?
  • Is it okay to use relus functions on the fully connected part of the CNN?
  • Can we expect better losses for the 1 node input identity CNN ? With the later CNN architecture I get values ranging from 0.0002 and 0.001, but should I not get better losses for my trivial problem?
  • Is my identity function really that trivial?
  • why do I have to change the dims of my labels? actually, the real question is, why did the dimension of inputs reduced to 2?

It is a long post it will interest someone, thanks in advance for reading me anyway.

Cheers
P.