CNN architecture for short time series data

1

I want to build a CNN for classifying time series data with high accuracy. The data has been windowed into chunks of 12 time steps, it looks like this:

       [0.        , 0.00167084, 0.00568087, ..., 0.60016708, 0.55238095,
        0.68421053],
       [0.00167084, 0.00568087, 0.146533  , ..., 0.55238095, 0.68421053,
        0.6566416 ],

Consider my use of CNN for such data an experiment.

Here are 2 architectures that I have tried. But they have poor accuracy.

#Architecture 1:

init:

    self.conv1 = nn.Conv1d(1,3, kernel_size = 2, stride = 1, padding =1)
    self.bn1 = nn.BatchNorm1d(3)
    self.maxpool1 = nn.MaxPool1d(kernel_size=2, stride=2, padding=1)
    self.dropout1 = nn.Dropout(0.3)
    
    self.conv2 = nn.Conv1d(3, 5, kernel_size=2, stride=2, padding=1)
    self.bn2 = nn.BatchNorm1d(5)
    self.dropout2 = nn.Dropout(0.3)
    
    self.conv3 = nn.Conv1d(5, 5, kernel_size=2, stride=1, padding=1)
    self.bn3 = nn.BatchNorm1d(5)
    self.dropout3 = nn.Dropout(0.3)
    
    self.fc = nn.Linear(5, num_classes)

forward:

    x = self.conv1(x)
    x = self.bn1(x)
    x = F.relu(x)

    x = self.maxpool1(x)
    x = self.dropout1(x)

    x = self.conv2(x)
    x = self.bn2(x)
    x = F.relu(x)

    x = self.dropout2(x)

    x = self.conv3(x)
    x = self.bn3(x)
    x = F.relu(x)

    x = self.dropout3(x)
    x = torch.mean(x, dim = 2)
    x = self.fc(x)

#Architecture 2: similar to PyTorch MNIST example

init:

    self.conv1 = nn.Conv1d(1,3, kernel_size = 2, stride = 1, padding =1)
    self.conv2 = nn.Conv1d(3, 5, kernel_size=2, stride=2, padding=1)
    self.maxpool1 = nn.MaxPool1d(kernel_size=2, stride=2, padding=1)
    self.dropout1 = nn.Dropout(0.3)
    self.dropout2 = nn.Dropout(0.3)
    
    self.fc1= nn.Linear(20, 12)
    self.fc2= nn.Linear(12, num_classes)

forward:

    x = self.conv1(x)
    x = F.relu(x)
    x = self.conv2(x)
    x = F.relu(x)
    x = self.maxpool1(x)
    x = self.dropout1(x)
    x = torch.flatten(x, 1)
    x = self.fc1(x)
    x = F.relu(x)
    x = self.dropout2(x)
    x = self.fc2(x)

Could anyone suggest what I am doing fundamentally wrong here?
P.S. I want to classify them into 3 classes

2

I can’t see any fundamental flaws in your models.
You could e.g. apply the relu before the batchnorm layers in your first model, but this might not change too much.

I would recommend to try to overfit a small dataset, e.g. just 10 samples, and make sure your model is able to do so by playing around with the hyperparameters.
If that’s not working, there might be another issue in your training routine, such as forgetting to zero out the gradients.

3

I will give that a try

@ptrblck What are your thoughts on using batch norm as well as dropout? I read that batch norm acts like a regularizer so that we dont need dropout.

4

I’ve read the same a couple of times, but would recommend to run a few experiments to make sure this claim isn’t only valid for some mentioned use cases.

1 Like