Manually inserting hand-crafted features to a pretrained model

I am using a pretrained VGG model,

model = models.vgg19(pretrained=True)

and interested in adding hand-crafted features before the model.classifier forward pass.
So, I am actually interested in doing:

feat = model.features( image )

and then backpropagation to update the weights of the classifier, and the feature extraction layers.

What is the best way to write this in Pytorch? (I thought of writing a new class named “VGG_extended” that do the forward pass as required, but I am not exactly sure how the backprop will work in this case… Is there any simple solution?)

Thanks in advance!

I think you can define a new class , and define the vgg model and an extension linear layer separately inside that class. Then, in the forward() function, you get features from both sources (vgg features and hand-crafted features), and concatenate them and send them to the classifier. This way, the backpropagation will work on both parts.

def handcrafted_features(images):
    # Your hand-crafted features here (I am just using random numbers for now)
    hand_feat = np.random.normal(size=(len(images), 100))
    hand_feat = np.array(hand_feat, dtype='float32')
    return torch.tensor(hand_feat)

class VGGXtend(nn.Module):
    def __init__(self, num_hand_features, num_classes):
        super(VGGXtend, self).__init__()

        # define the vgg model:
        self.vgg = models.vgg16(pretrained=True)
        # change the last FC layer:
        self.vgg.classifier[6] = nn.Linear(4096, 1000, bias=True)

        # define the new classifier:
        self.classifier = nn.Linear(1000 + num_hand_features, num_classes)

    def forward(self, images):
        vgg_feat = self.vgg(images)
        hand_feat = handcrafted_features(images)
        features =[vgg_feat, hand_feat], dim=1)
        return self.classifier(features)

and a test example:

vggx = VGGXtend(100, 100)
images = np.random.normal(size=(16, 3, 224, 224))
images = np.array(images, dtype='float32')

Dear Vahid Mirjalili, Thanks a lot! That’s exactly what I was thinking of. The thing that I am not sure about is the backprop part. Since there is no learning on the hand-crafted features (since they are not “extracted”), what exactly would


do? (since it also depends on the hand-crafted features)

Hope I made my question clear enough

Yes, it depends how the hand-crafted features are computed. If you use some functions and arrays outside PyTorch, like what I did in NumPy, and then convert them to PyTorch tensors, they will not be in the « computation-graph ». So, in this setting, the actual features will not be back-propped, but only their final weights in the vggx.classifier will be back-proped.

However, if you want the hand-crafted features to also be included in the back-propagation, then you should define them in tensors and all the operations applied to them should use pytorch modules and learnable parameters. Then, those will also be automatically backpropped.

1 Like

Thanks a lot for your explanation. I will give it a try :slight_smile: