Hi,
I am new to using PyTorch and really like the Pythonic approach it offers. Currently I’m trying to do a Neural Net performance comaprison between a simple Depthwise Separable Convolutional Neural Net and a standard Convolutional Neural net. The model architecture of both is simple:
The following are done 3 times
Convolution-> Batch Normalization-> ReLU -> Pool
Followed by flattening it to 3 outputs in two steps.
The code is as following:
Standard Conv. Net architecture:
in_features = 304 #in_features for Flatten(linear) layer
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(1, 4, kernel_size=2, padding=1)
self.bn1 = nn.BatchNorm2d(num_features=4)
self.relu1 = nn.ReLU()
self.pool1 = nn.MaxPool2d(2)
self.conv2 = nn.Conv2d(4, 8, kernel_size=2, padding=1)
self.bn2 = nn.BatchNorm2d(num_features=8)
self.relu2 = nn.ReLU()
self.pool2 = nn.MaxPool2d(2)
self.conv3 = nn.Conv2d(8, 16, kernel_size=2, padding=1)
self.bn3 = nn.BatchNorm2d(num_features=16)
self.relu3 = nn.ReLU()
self.pool3 = nn.MaxPool2d(2)
self.fc1 = nn.Linear(in_features, 36)
self.relu4 = nn.ReLU()
self.fc2 = nn.Linear(36, 3)
def forward(self, x):
out = self.pool1(self.relu1(self.bn1(self.conv1(x))))
out = self.pool2(self.relu2(self.bn2(self.conv2(out))))
out = self.pool3(self.relu3(self.bn3(self.conv3(out))))
out = out.view(out.size(0), -1)
out = self.relu4(self.fc1(out))
out = self.fc2(out)
return out
standardCNN = Net() # defining an instance of our network
Depthwise Separable Conv. Net architecture:
class depthwise_separable_conv(nn.Module):
def __init__(self):
super(depthwise_separable_conv, self).__init__()
self.depthwise1 = nn.Conv2d(1, 1, kernel_size=2, padding=1, groups=1)
self.pointwise1 = nn.Conv2d(1, 4, kernel_size=1)
self.bn1 = nn.BatchNorm2d(num_features=4)
self.relu1 = nn.ReLU()
self.pool1 = nn.MaxPool2d(2)
self.depthwise2 = nn.Conv2d(4, 4, kernel_size=2, padding=1, groups=4)
self.pointwise2 = nn.Conv2d(4, 8, kernel_size=1)
self.bn2 = nn.BatchNorm2d(num_features=8)
self.relu2 = nn.ReLU()
self.pool2 = nn.MaxPool2d(2)
self.depthwise3 = nn.Conv2d(8, 8, kernel_size=2, padding=1, groups=8)
self.pointwise3 = nn.Conv2d(8, 16, kernel_size=1)
self.bn3 = nn.BatchNorm2d(num_features=16)
self.relu3 = nn.ReLU()
self.pool3 = nn.MaxPool2d(2)
self.dense1 = nn.Linear(304, 36)
self.relu4 = nn.ReLU()
self.dense2 = nn.Linear(36,3)
def forward(self, x):
out = self.pool1(self.relu1(self.bn1(self.pointwise1(self.depthwise1(x)))))
out = self.pool2(self.relu2(self.bn2(self.pointwise2(self.depthwise2(out)))))
out = self.pool3(self.relu3(self.bn3(self.pointwise3(self.depthwise3(out)))))
out = out.view(out.size(0), -1)
out = self.relu4(self.dense1(out))
out = self.dense2(out)
return out
dsCNN_model = depthwise_separable_conv()
Now, here’s the important part. As the Standard Convolutions require a large number of parameters as compared to depthwise, I was hoping to get reduced number of parameters for my dsCNN_model. But to my surprise, I got these results.
Parameters:
total_params = 0
for parameter in standardCNN.parameters():
if parameter.requires_grad:
total_params += np.prod(parameter.size())
print(total_params)
>>11831
total_params = 0
for parameter in dsCNN_model.parameters():
blah blah...
print(total_params)
>>11404
Both having almost the same with a mere reduction of 400 parameters (from ~11800 to ~11400).
I wanted to ask, is it that this is normal behavior of Depthwise model and we do not see that much of a differnce in smaller models? Or is there something wrong with my approach, or in the model architecture that is causing this? I’m actually trying to get a model with least parameters so the inference is possible on a hardware with less computational resources. Also, what other approaches are there in PyTorch’s arsenal that can be used for this purpose.
Any help would be really appreciated… Thanks!
