Rewrite Keras code to PyTorch

1

Greetings,
What I want to do is rewriting the Keras code into PyTorch.
The only problem is how to implement Depthwise and Separable 2D convolution in PyTorch.
Here’s the original code from https://github.com/vlawhern/arl-eegmodels/blob/master/EEGModels.py

input1   = Input(shape = (1, Chans, Samples))

##################################################################
block1       = Conv2D(F1, (1, kernLength), padding = 'same',
                               input_shape = (1, Chans, Samples),
                               use_bias = False)(input1)
block1       = BatchNormalization(axis = 1)(block1)
block1       = DepthwiseConv2D((Chans, 1), use_bias = False, 
                               depth_multiplier = D,
                               depthwise_constraint = max_norm(1.))(block1)
block1       = BatchNormalization(axis = 1)(block1)
block1       = Activation('elu')(block1)
block1       = AveragePooling2D((1, 4))(block1)
block1       = dropoutType(dropoutRate)(block1)

block2       = SeparableConv2D(F2, (1, 16),
                               use_bias = False, padding = 'same')(block1)
block2       = BatchNormalization(axis = 1)(block2)
block2       = Activation('elu')(block2)
block2       = AveragePooling2D((1, 8))(block2)
block2       = dropoutType(dropoutRate)(block2)
    
flatten      = Flatten(name = 'flatten')(block2)

dense        = Dense(nb_classes, name = 'dense', 
                     kernel_constraint = max_norm(norm_rate))(flatten)
softmax      = Activation('softmax', name = 'softmax')(dense)

return Model(inputs=input1, outputs=softmax)
2

Found on this forum:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 1, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
        )

    def forward(self, x, y):
        x1 = self.features(x)
        x2 = self.features(y)
        x = torch.cat((x1, x2), 1)
        return x

By using x = torch.cat((x1, x2), 1) one can concatenate multiple towers/blocks. The forward function can take multiple input data. Want to learn more, search for “Pytorch multi input network” or something similar.

3

You can use the groups argument of nn.Conv2d followed by a pointwise convolution to get SeparableConv2d. Have a look at the docs for more information of groups.

4

Thanks for your response @ptrblck, but how should we define the depthwise_constraint = max_norm(1.) in pytorch as well as other features in keras documentation such as:

  • depthwise_regularizer: Optional regularizer for the convolution kernel.
  • bias_regularizer: Optional regularizer for the bias vector.
  • activity_regularizer: Optional regularizer function for the output.
  • depthwise_constraint: Optional projection function to be applied to the kernel after being updated by an Optimizer (e.g. used to implement norm constraints or value constraints for layer weights). The function must take as input the unprojected variable and must return the projected variable (which must have the same shape). Constraints are not safe to use when doing asynchronous distributed training.
  • bias_constraint: Optional projection function to be applied to the bias after being updated by an Optimizer.

Thank you.