and then backpropagation to update the weights of the classifier, and the feature extraction layers.
What is the best way to write this in Pytorch? (I thought of writing a new class named “VGG_extended” that do the forward pass as required, but I am not exactly sure how the backprop will work in this case… Is there any simple solution?)
I think you can define a new class , and define the vgg model and an extension linear layer separately inside that class. Then, in the forward() function, you get features from both sources (vgg features and hand-crafted features), and concatenate them and send them to the classifier. This way, the backpropagation will work on both parts.
def handcrafted_features(images):
# Your hand-crafted features here (I am just using random numbers for now)
hand_feat = np.random.normal(size=(len(images), 100))
hand_feat = np.array(hand_feat, dtype='float32')
return torch.tensor(hand_feat)
class VGGXtend(nn.Module):
def __init__(self, num_hand_features, num_classes):
super(VGGXtend, self).__init__()
# define the vgg model:
self.vgg = models.vgg16(pretrained=True)
# change the last FC layer:
self.vgg.classifier[6] = nn.Linear(4096, 1000, bias=True)
# define the new classifier:
self.classifier = nn.Linear(1000 + num_hand_features, num_classes)
def forward(self, images):
vgg_feat = self.vgg(images)
hand_feat = handcrafted_features(images)
features = torch.cat([vgg_feat, hand_feat], dim=1)
print(features.shape)
return self.classifier(features)
Dear Vahid Mirjalili, Thanks a lot! That’s exactly what I was thinking of. The thing that I am not sure about is the backprop part. Since there is no learning on the hand-crafted features (since they are not “extracted”), what exactly would
loss.backward()
do? (since it also depends on the hand-crafted features)
Yes, it depends how the hand-crafted features are computed. If you use some functions and arrays outside PyTorch, like what I did in NumPy, and then convert them to PyTorch tensors, they will not be in the « computation-graph ». So, in this setting, the actual features will not be back-propped, but only their final weights in the vggx.classifier will be back-proped.
However, if you want the hand-crafted features to also be included in the back-propagation, then you should define them in tensors and all the operations applied to them should use pytorch modules and learnable parameters. Then, those will also be automatically backpropped.
@vmirly1, Hi, may I know how can we adapt this if we want to include these handcrafted_features or lets say any matrix operation somewhere between convolutional layers, lets say after 2nd conv in vgg?
