How to extract features of an image from a trained model


(Fanta) #85

What if the model was not build with nn.Sequential? How can I extract a part of it? For instance, I have derived a class from nn.Module and implemented its forward() method as

    def forward(self, x):
        x = F.conv2d(input=x, weight=self.conv1_w, bias=self.conv1_b, padding=self.conv1_pad)
        x = F.max_pool2d(F.relu(x), 2)
        x = F.conv2d(input=x, weight=self.conv2_w, bias=self.conv2_b, padding=self.conv2_pad)
        x = F.max_pool2d(F.relu(x), 2)
        x = x.view(x.size(0), -1)  # Flatten but keep the same batch size
        x = F.linear(input=x, weight=self.fc1_w, bias=self.fc1_b)
        x = F.relu(x)
        x = F.linear(input=x, weight=self.fc2_w, bias=self.fc2_b)
        return x

Can I extract the model until the second convolutional layer, to see its output? That is, without doing copy-and-paste of the code.


(Pan He) #86

Is there any update for iterating over modules with names?


(Woaichipingguo Tsh) #87

I have an error . My code is :

class FeatureExtractor(nn.Module):
    def __init__(self, submodule, extracted_layers):
        super(FeatureExtractor,self).__init__()
        self.submodule = submodule
        self.extracted_layers= extracted_layers

    def forward(self, x):
        outputs = []
        for name, module in self.submodule._modules.items():
            x = module(x)
            print(name)
            if name in self.extracted_layers:
                outputs.append(x)
        return outputs
myresnet=resnet18(pretrained=False)
num_ftrs=myresnet.fc.in_features
myresnet.fc=nn.Linear(num_ftrs,10)
exact_list=["conv1","layer1","avgpool"]
myexactor=FeatureExtractor(myresnet,exact_list)
a=torch.randn(1,3,32,32)
a=Variable(a)
x=myexactor(a)
```RuntimeError: size mismatch, m1: [512 x 1], m2: [512 x 10] at c:\miniconda2\conda-bld\pytorch-cpu_1519449358620\work\torch\lib\th\generic/THTensorMath.c:1434

type or paste code here


(Woaichipingguo Tsh) #88

sorry. I forgot x.view().


#89

can anyone please tell me what happens in the line above?


#91

Is there anyone who could figure it out and show it with examples? any blog or website or tutorial? anything?


#92

can someone please explain to me what is happening here and what this class Net is donig ??

If we are interested in using the feature part, why cannot we just keep the features and get rid of classifier?


(Ganesh Venkatesh) #93

I am not sure I understand why the FeatureExtractor function by @woaichipingguo_TSH works for Resnet network that has residual connections and not just a sequential list of convolutions. It is not always the case the output of current layer is the input to the next one because of the residual connections. How does it handle that?


(Ravi Kumar) #94

Hi, could you kindly point out, where to put x.view() in the above FeatureExtractor to avoid the mismatch error and the dimensions?
I am still a beginner to pytorch and I am trying to learn to perform feature extraction using pretrained models.


(highhig) #95

I’d like to know the result of calling forward (x) in FeatureExtractor (submodule, extracted_layers)


(highhig) #96

I want to know that Net().forward(x) is called, where x is the image, and then what is returned is the feature result that we want to extract from the layer, and this result exists as a list?So what do we say if we extract the results of each layer separately?Could you please write out the detailed process for me to see?thanks


(Lucy Gan) #97

This one is awesome :slight_smile: Exactly what I’m looking for when trying to get feature from certain intermediate layer.


(Neta Zmora) #98

Here’s how we iterate over module names: https://github.com/NervanaSystems/distiller/blob/master/distiller/model_summaries.py#L61:L71

Name                          Type
----------------------------  -----------
module.conv1                  Conv2d
module.bn1                    BatchNorm2d
module.relu                   ReLU
module.maxpool                MaxPool2d
module.layer1.0.conv1         Conv2d
module.layer1.0.bn1           BatchNorm2d
module.layer1.0.relu          ReLU
module.layer1.0.conv2         Conv2d
module.layer1.0.bn2           BatchNorm2d
module.layer1.1.conv1         Conv2d
module.layer1.1.bn1           BatchNorm2d
module.layer1.1.relu          ReLU
module.layer1.1.conv2         Conv2d
module.layer1.1.bn2           BatchNorm2d
module.layer2.0.conv1         Conv2d
module.layer2.0.bn1           BatchNorm2d
module.layer2.0.relu          ReLU
module.layer2.0.conv2         Conv2d
module.layer2.0.bn2           BatchNorm2d
module.layer2.0.downsample.0  Conv2d
module.layer2.0.downsample.1  BatchNorm2d
module.layer2.1.conv1         Conv2d
module.layer2.1.bn1           BatchNorm2d
module.layer2.1.relu          ReLU
module.layer2.1.conv2         Conv2d
module.layer2.1.bn2           BatchNorm2d
module.layer3.0.conv1         Conv2d
module.layer3.0.bn1           BatchNorm2d
module.layer3.0.relu          ReLU
module.layer3.0.conv2         Conv2d
module.layer3.0.bn2           BatchNorm2d
module.layer3.0.downsample.0  Conv2d
module.layer3.0.downsample.1  BatchNorm2d
module.layer3.1.conv1         Conv2d
module.layer3.1.bn1           BatchNorm2d
module.layer3.1.relu          ReLU
module.layer3.1.conv2         Conv2d
module.layer3.1.bn2           BatchNorm2d
module.layer4.0.conv1         Conv2d
module.layer4.0.bn1           BatchNorm2d
module.layer4.0.relu          ReLU
module.layer4.0.conv2         Conv2d
module.layer4.0.bn2           BatchNorm2d
module.layer4.0.downsample.0  Conv2d
module.layer4.0.downsample.1  BatchNorm2d
module.layer4.1.conv1         Conv2d
module.layer4.1.bn1           BatchNorm2d
module.layer4.1.relu          ReLU
module.layer4.1.conv2         Conv2d
module.layer4.1.bn2           BatchNorm2d
module.avgpool                AvgPool2d
module.fc                     Linear

If you want to play with this, clone Distiller and execute as follows:

$ python3 compress_classifier.py -a=resnet18 --pretrained ../../../data.imagenet --summary=modules

../../../data.imagenet is the path to the ImageNet dataset. When the application is invoked this way, passing the dataset path doesn’t make sense, but due to development priority it is currently mandatory - sorry about that.


(Neta Zmora) #100

In case anyone is still interested in this topic: In Distiller we’ve implemented a couple of classes that record activation statistics when you forward input images in TorchVision’s image classifiers.

The code is here, and it is long - sorry about that: it’s part of a larger library. The complexity is also because we wanted a solution that can work with all of TorchVision’s image classifier models (present, and hopefully, future).

In short:
1.First, we assign each layer a fully-qualified name which we extract by traversing the graph using DFS.
2. We register a callback on the forward_hooks
3. We use the forward_hooks to access the activations and store them in the module itself as a new member variable.
4. Finally, we create a dictionary of {module_name: module_activation_stats}

We collect both detailed records of activation statistics (e.g. min, max, mean, std for each activation), or summary data (e.g. average L1-norm of activations). We also collect statistics over sub-structures of activations (e.g. average L1-norm of activation channels), but that’s a post for a different time.
We also dump stats to Excel worksheets.

Earlier in this thread, I posted a pointer to some code which extracts fully-qualified layer (module) names, to enable use to create a dictionary of {module_name: module_activation_stats}.

Hope that helps someone, even though this is not a self-standing example :frowning:


(Jay Chuk) #101

when running the code in network_bisection.ipynb

I found an error in the code:

my_embedding = torch.zeros(512)
def fun(m, i, o): my_embedding.copy_(o.data)
h = avgpool_layer.register_forward_hook(fun)
h_x = resnet_18(x)
h.remove()

RuntimeError: expand(torch.FloatTensor{[1, 512, 1, 1]}, size=[512]): the number of sizes provided (1) must be greater or equal to the number of dimensions in the tensor (4)

can you help resolve it


(Hawk Wang) #102

just modify my_embedding = torch.zeros(1, 512, 1, 1)


(Praveen Kumar) #103

from torchvision import models
from torchsummary import summary
import torch.nn as nn
#feature = models.alexnet().features
feature = models.resnet18()
all=[]
count=1
for name,module in feature._modules.items():
for n,m in module._modules.items():
for nb,mb in m._modules.items():
if(nb==‘relu’ and count<=3):
all.append(nb)
count+=1
print(all," : ",count)


(Praveen Kumar) #104

Extract only Relu Layer Feature in ResNet18
feature = models.resnet18()
all=[]
count=1
for name,module in feature._modules.items():
for n,m in module._modules.items():
for nb,mb in m._modules.items():
if(nb==‘relu’ and count<=3):
all.append(nb)
count+=1
print(all," : ",count)


(Tanm Dl) #105

I have a similar question on feature extraction from the pretrained model. My model structure is given below,
Demo(nn.Module):
def init(self):
super(Demo,self).init()
self.conv1d = nn.Conv2d( 200, 128, 3, 1, 0, bias=False)
self.conv2d = nn.Conv2d( 128, 256, 3, 1, 0, bias=False)
self.conv3d = nn.Conv2d( 256, 512, 3, 1, 0, bias=False)
self.bn1d = nn.BatchNorm2d(128)
self.bn2d = nn.BatchNorm2d(256)
self.bn3d = nn.BatchNorm2d(512)
self.fc = nn.Linear(4608,1)
self.relud = nn.LeakyReLU(0.2, inplace=True)
def forward(self,x):
batch_size = x.size()[0]
x = self.relud(self.bn1d(self.conv1d(x)))
x = self.relud(self.bn2d(self.conv2d(x)))
x = self.relud(self.bn3d(self.conv3d(x)))
x = x.view(batch_size,-1)
x = self.fc(x)
return x
Demo model is used for training.
Feature_Net(nn.Module):
def init(self,pretrained):
super(Feature_Net,self).init()
self.pretrained = pretrained
for para in self.pretrained.parameters():
para.requires_grad =False
self.network = []
self.model_layer = list(self.pretrained.children())
for i in range(len(self.model_layer)):
self.network.append(nn.Sequential(*self.model_layer[i]))
self.max = nn.MaxPool(5)
self.max = nn.MaxPool(3)
def forward(self,x):
batch_size = x.size()[0]
x = self.network7
x = self.network7
f_1 = self.max1(x)
x = self.network7
f_2 = self.max2(x)
x = torch.cat((f_1,f_2),1)
return x

demo =Demo()
feature = Feature_Net(demo)
input = torch.ones(1,200,9,9)
output= feature(input) ###(1,768)

Should I consider only conv2d for learning features vector or the whole branch (conv2d+ bn1d+relu) ?