Thanks for the code.
It looks like to padding of your second max pooling layer is wrong, since you are using the same argument in Keras.
Try this definition self.maxpool2 = nn.MaxPool2d(7,2,padding=3) and your output will be [batch_size, 96, 4, 4] for both branches.
4 Likes