Hello Torchers,
I have an issue with the data input from the dataloader wrapper and its dimension not adapted to my CNN inputs.
I am using a 1D CNN for an interpolation problem right now. For debugging purpose I have set the input size to 1 node and the CNN must learn the identity function. (is that meaningful btw?, target/label = input for the identity function). I am also training by batches.
Let me tell you first what I am doing before I ask my questions -
The data is constructed with the following custom dataset constructor and dataloader wrapper:
class custom_dataset(torch.utils.data.Dataset):
'Characterizes a dataset for PyTorch'
def __init__(self, examples, labels):
'Initialization'
self.labels = labels
self.examples = examples
def __len__(self):
'Denotes the total number of samples'
return len(self.examples)
def __getitem__(self, index):
'Generates one sample of data'
# Select sample
# Load data and get label
X = self.examples [index]
y = self.labels [index]
return X, y
the wrapping happens here, in comments my data shape:
#with train_example.shape = torch.Size([57600])
#and train_label.shape = torch.Size([57600])
train_example = np.asarray(train_example)
train_label = np.asarray(train_label)
trainset = custom_dataset(train_example,train_label)
the CNN look like this:
class ConvNet1D(nn.Module):
def __init__(self,winlength_size, dropout=0.2):
super(ConvNet1D, self).__init__()
self.layer1 = nn.Sequential(
nn.Conv1d(1, 16, kernel_size=1, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool1d(kernel_size=2, stride=1))
self.layer2 = nn.Sequential(
nn.Conv1d(16, 32, kernel_size=2, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool1d(kernel_size=2, stride=1))
self.drop_out = nn.Dropout()
self.fc1 = nn.Linear(64, 20)
self.fc2 = nn.Linear(20, 1)
def forward(self, x):
out = self.layer1(x)
out = self.layer2(out)
out = out.reshape(out.size(0), -1)
out = self.drop_out(out)
out = self.fc1(out)
out = self.fc2(out)
return out
finally the training happens in the following snippet:
'''optimizer is adam'''
'''criterion is MSE loss, reduce = True'''
for batch_idx, data in enumerate(train_loader):
inputs = data[0]
labels = data[1]
optimizer.zero_grad()
outputs = model(inputs)
loss = criterion (outputs, labels)
loss.backward()
optimizer.step()
first issue that came, the dimension : run with those snippets, with a batch_size of 16, the code would give me the following error:
expected 3-dimensional input for 3-dimensional weights 16, but got 1 dimensional input of size 16
by adding new dimensions on my input and label data, I could solve the 1D ConvNet input structure issue which must be 3 dimensional, for channel numbers and lengths.
It is done as following:
train_example = train_example.reshape(train_example[0].shape, 1,1)
train_labels = train_labels.reshape(train_labels[0].shape,1,1)
now I have the following mistake:
Using a target size: torch.Size([16,1,1]) that is different to the input size : torch.Size([16,1]). This will likely lead to incorrect results due to error.
The input dimension somehow reduced to 2, but I cannot change the input dimension because of the conditions on my inputs given by the convnet, so I need to work with the label dimensions:
train_labels = train_labels.reshape(train_labels[0].shape, 1)
here I would have a running code, but I don’t have a clear convergence:
#3 firsts loss by batch:
3_losses= [0.31,0.181,0.143]
#!PSEUDO CODE! The rest:
the_rest = span(random between 0.005 and 0.09)
so I changed the CNN to
class ConvNet1D(nn.Module):
def __init__(self,winlength_size, dropout=0.2):
super(ConvNet1D, self).__init__()
self.layer1 = nn.Sequential(
nn.Conv1d(1, 16, kernel_size=1, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool1d(kernel_size=2, stride=1))
self.layer2 = nn.Sequential(
nn.Conv1d(16, 32, kernel_size=2, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool1d(kernel_size=2, stride=1))
self.drop_out = nn.Dropout()
self.fc1 = nn.Linear(64, 3000)
self.fc2 = nn.Linear(3000, 150)
self.fc3 = nn.Linear(150,1)
def forward(self, x):
out = self.layer1(x)
out = self.layer2(out)
out = out.reshape(out.size(0), -1)
out = self.drop_out(out)
out = F.relu(self.fc1(out))
out = F.relu(self.fc2(out))
out = self.fc3(out)
return out
to get better results.
now my questions:
- Is it okay for the CNN to have so many nodes in the fully connected layouts?
- Is it okay for the CNN to have more than 2 layers in the fully connected part?
- Is it okay to use relus functions on the fully connected part of the CNN?
- Can we expect better losses for the 1 node input identity CNN ? With the later CNN architecture I get values ranging from 0.0002 and 0.001, but should I not get better losses for my trivial problem?
- Is my identity function really that trivial?
- why do I have to change the dims of my labels? actually, the real question is, why did the dimension of inputs reduced to 2?
It is a long post it will interest someone, thanks in advance for reading me anyway.
Cheers
P.