Hello I’m quite new to PyTorch. I’m trying to implement a Neural Net originally designed with Keras.
In particular, I have an issue with the Add module of Keras:
# C is also a Conv1D layer
C12 = Conv1D(filters=32, kernel_size=5, strides=1, padding='same')(A11)
S11 = Add()([C12, C])
I don’t find any equivalent in PyTorch. In particular, I’d like to implement it using the nn.Sequential design.
Any help is welcome.
Many thanks in advance
Best,
PAB
Try nn.Sequential(C12, C) Hope this helps 
I thought that nn.Sequential would pass my input sequentially into C12 and C. I think that S11 adds the ouput of C12 and C, doesn’t it ?
torch.add might be the solution but I don’t know how it can fit into nn.Sequential since I don’t specify which layers to add.
U r correct about nn.Sequential. I don’t get what r u trying to achieve, so can u explain more about what r u trying to do since I’m not familiar with Keras?
I am trying to implement a model from an Arxiv paper. The author used Keras to implement it. I thought that a relevant way to implement it using PyTorch is by using nn.Sequential . Here is the architecture of the model.
I understand the concept of nn.Sequential but I don’t know how to include torch.add() in a nn.Sequential block.
I don’t think there is something in Pytorch that does it; u probably need to split the Sequential and add them manually.
If you want to insert some modules dynamically, try nn.ModuleList instead of nn.Sequential. Some operators such as nn.ModuleList.append, nn.ModuleList.extend, nn.ModuleList.insert may helpful.
In my understanding, do you want to dynamically choose whether to use skip connection(ADD)? I have an idea
self.door = 1 # when you define your model
S11 = C12 + C * self.door # in forward function
net.door = 0 # if you want to drop the skip connection
net.door = 1 # if you want to keep the skip connection
I am trying to implement the same NN. This is the way I build the residual blocks:
class ResidualBlock(nn.Module):
def __init__(self, in_channels, out_channels, kernel_size, stride):
super(ResidualBlock, self).__init__()
self.conv1 = nn.Conv1d(in_channels, out_channels, kernel_size)
self.conv2 = nn.Conv1d(in_channels, out_channels, kernel_size)
self.max_pool = nn.MaxPool1d(kernel_size, stride)
def forward(self, x):
residual = x
out = F.relu(self.conv1(x))
out = F.relu(self.conv2(out))
out += residual
out = self.max_pool(F.relu(out))
return out
However, I am having problems because the residual’s shape doesn’t match the output’s shape of the second conv layer. Any idea to solve the problem?
You could add padding to the conv layers and make sure the spatial size isn’t decreased or alternatively, you could downsample the residual as shown in the resnet implementation.
Thank you! It works.
Some advices for those who are trying to reproduce https://arxiv.org/abs/1805.00794:
This is the code to build the NN as described in the paper. Hope this helps!
import torch
import torch.nn as nn
import torch.nn.functional as F
class ECG_CNN(nn.Module):
def __init__(self, num_classes=5):
super(ECG_CNN, self).__init__()
self.conv1 = nn.Conv1d(1, 32, kernel_size=5)
self.res1 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)
self.res2 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)
self.res3 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)
self.res4 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)
self.res5 = ResidualBlock(32, 32, kernel_size=5, stride=2, padding=2)
self.fc1 = nn.Linear(32*2, 32)
self.fc2 = nn.Linear(32, num_classes)
def forward(self, x):
out = self.conv1(x)
out = self.res1(out)
out = self.res2(out)
out = self.res3(out)
out = self.res4(out)
out = self.res5(out)
out = torch.flatten(out, 1)
out = F.relu(self.fc1(out))
out = self.fc2(out)
return out
class ResidualBlock(nn.Module):
def __init__(self, in_channels, out_channels, kernel_size, stride, padding):
super(ResidualBlock, self).__init__()
self.conv1 = nn.Conv1d(in_channels, out_channels, kernel_size, padding=padding)
self.conv2 = nn.Conv1d(in_channels, out_channels, kernel_size, padding=padding)
self.max_pool = nn.MaxPool1d(kernel_size, stride=stride)
def forward(self, x):
residual = x
out = F.relu(self.conv1(x))
out = self.conv2(out)
out += residual
out = self.max_pool(F.relu(out))
return out