How can I extract intermediate layer output from loaded CNN model?

vision
#17

Hi @ptrblck, Thanks, I was able to get the proper activations, however, I am not completely convinced about the use of detach or clone.

Please reaffirm if my assumption is correct: detach() is used to remove the hook when the forward_hook() is done for an intermediate layer? I did see that when I iterated to get the next layer activation function, I also got the output from the first hook when detach() was not done.

Secondly, clone() is used to just clone the entire model as is. I tried with both output and output. detach() in the hook function and both returned after applying in-place operation.

activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.clone().detach()
    return hook

This is the function that I used and If my understanding is correct this should do the trick.

#18

No, detach() is not used for hooks, but is a tensor operation, which detaches the tensor from the computation graph. I.e. Autograd will stop during the backpropagation at this detached point.

No, clone() is also a tensor operation and will create a new tensor with the same data. The returned tensor will not be changed by inplace operations.

# without clone
x = torch.zeros(10)
y = x
x[0] = 1.
print(x)
> tensor([1., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
print(y)
> tensor([1., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

# with clone
x = torch.zeros(10)
y = x.clone()
x[0] = 1.
print(x)
> > tensor([1., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
print(y)
> tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
4 Likes
#19

Also is it possible to load in the pre-trained weights given by torchvision in the above modified ResNet model. Share. Share a link to this question.

#20

Thanks for the information

#21

@ptrblck Can we move a little further from this topic, say what if I want to get all intermediate results of all the operations in the graph during forward propagation. Because this is quite simple to do in tensorflow, you can

  1. Load the graph;
  2. Get all tensors from the graph;
    e.g.
    all_ops = graph.get_operations()
    all_op_names = [op.name for op in all_ops]
    all_tensors = [graph.get_tensor_by_name(’{}:0’.format(op_name)) for op_name in all_op_names]
  3. For tf 1.15 which I currently use a lot, use output = session.run(all_tensors, feed_dict={d_input:input_images}) to get all output. Can we do the same thing here in PyTorch. The reason why I asked this, is I want to check all operations’ results running on two different platform, and check that the results are all the same. If some intermmediate result differs, I can spot that.
    Another question is can we get all operations in a training graph including all the backprop related operations?
    Much appreciated.
#22

Yes, you could iterate the .named_modules() (or a combination of .named_children() etc.) and register a forward hook on each module, which would then store the outputs in the dict, if you are using the previously posted approach.

1 Like
#23

Thanks. I’m wondering if we could dump out model structures into a file. Because I want to write a separate script to directly read any model structure files together with the weights in pth, load data, register forward hooks to get output of all layers. So I can read e.g. YOLOV3, Mask-RCNN, or maybe resnet50 model files without having to modify my YOLOV3 / Mask-RCNN/ resnet50 project source files to register a forward hook every time. If you know what I mean, is this possible?

#24

I think the best approach would be to script the model via torch.jit.script and export this structure, as this graph would contain the execution workflow of the model, which would be missing if e.g. you only export the initialized modules.

#25

Hi @ptrblck, I’d like to know how to extract intermediate layer output when I have multi-inputs. For example

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cl1 = nn.Linear(25, 60)
        self.cl2 = nn.Linear(40, 16)
        self.fc1 = nn.Linear(76, 120)
        
    def forward(self, x1, x2):
        out1 = F.relu(self.cl1(x1))
        out2 = F.relu(self.cl2(x2))
        x = torch.cat((out1,out2), dim=1)
        x = F.log_softmax(self.fc1(x), dim=1)
        return x

And I want to get the out1. Could you please help me ? Thanks a lot.

#26

You could replace the F.relu operations with their nn.ReLU modules and register a forward hook to the first nn.ReLU module as described before.

1 Like
#27

Hello,

I have created a “recorder” for such goals part of this package:

It is very simple to record from multiple layers of PyTorch models, including CNNs.

An example to record output from all conv layers of VGG16:


model = torch.hub.load('pytorch/vision:v0.10.0', 'vgg16', pretrained = True)
# Only conv layers
layer_nr = [0, 2, 5, 7, 10, 12, 14, 17, 19, 21, 24, 26, 28]

# Get layers from model
layers = [list(model.features.named_children())[lnr][1] for lnr in layer_nr]

# Assign a recorder to each layer from modelutils in torchknickknacks
from torchknickknacks import modelutils
recorders = [modelutils.Recorder(layer, record_output = True, backward = False) for layer in layers]

# Pass some data through the model
X = torch.rand(32,3,224,224)
out = model(X)

# Get the output of each recorder (output of each layer)
rec = [r.recording.detach().clone() for r in recorders]

The Recorder class is based on Understanding Pytorch hooks | Kaggle

Get intermediate layer outputs
#28

Hi @ptrblck , I tried to follow this method as suggested for my network.
I want to extract all 4 layer features in a single go: I am unsure if they are overwritten as the layer name is same in SSL. Can you please suggest if my method is correct, if not please suggest me a better method

Thanks:

def forward(self, x, type='SL'):
    if type == 'SL':
      h0 = self.conv1(x) #Image(x) with transforms1
      h0 = relu(self.bn1(h0))
      h0 = self.maxpool(h0)
      h1 = self.layer1(h0)
  ### 1. extract and save features here ###	
      h2 = self.layer2(h1)
      h3 = self.layer3(h2)
      h4 = self.layer4(h3)

      m1_ = self.f_conv1(x) #Image(x) with transforms1
      m1 = m1_ * h1
  ### 2. extract and save features here ###
      m2_ = self.f_conv2(m1)
      m2 = m2_ * h2
      m3_ = self.f_conv3(m2)
      m3 = m3_ * h3
      m4_ = self.f_conv4(m3)
      m4 = m4_ * h4
      out = self.avgpool(m4)
      out = out.view(out.size(0), -1)
      y = self.linear(out)
      return y

    if type=='SSL':
      h0 = self.conv1(x1) #Image(x) with transforms2
      h0 = relu(self.bn1(h0))
      h0 = self.maxpool(h0)
      h1 = self.layer1(h0)
      ### 3. extract and save features here ###
      h2 = self.layer2(h1)
      h3 = self.layer3(h2)
      h4 = self.layer4(h3)
      feat = self.avgpool(h4)  
 
      h01 = self.conv1(x2) #Image(x) with transforms3
      h01 = relu(self.bn1(h01))
      h01 = self.maxpool(h01)
      h11 = self.layer1(h01)
      ### 4.  extract and save features here ###
      h21 = self.layer2(h11)
      h31 = self.layer3(h21)
      h41 = self.layer4(h31)
      feat1 = self.avgpool(h41)

I have registered hook and extracted as as follows

activation = {}
def get_activation(name):
  def hook(model, input, output):
      activation[name] = output.detach()
  return hook

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

model = Net(device, 10)
model = model.to(device)
model.layer1.register_forward_hook(get_activation('layer1'))
model.f_conv1.register_forward_hook(get_activation('f_conv1'))
x = torch.randn(1, 3, 128, 128)
output = model(x, type='SL')
print(activation['layer1'])
print(activation['f_conv1'])

model.layer1.register_forward_hook(get_activation('layer1'))
model.layer1.register_forward_hook(get_activation('layer1'))
output = model(x1, type='SSL')
print(activation['layer1'])

output = model(x2, type='SSL')
print(activation['layer1'])
#29

Yes, reusing the same key in the dict would replace the old activations.
A better approach would be to use different names in case you want to keep all intermediate activations.

#30

Do you know the module torchvision.models.feature_extraction?

That is what I usually use

#31

Thanks @ptrblck for the confirmation.

In my case, key (layer name) is the same layer from which I am trying to extract the representations, so how do I change the key name, if I want to register layer1, would this work if I change the key inside the get_activation(‘key name’)

 model.layer1.register_forward_hook(get_activation('layer-h11'))
 model.layer1.register_forward_hook(get_activation('layer-h41'))

What is the difference if I return the layers in the forward function from the example network vs using hooks…to save and access them later

def forward(self, x, type='SL'):
  if type == 'SL':
    h0 = self.conv1(x) 
    h0 = relu(self.bn1(h0))
    h1 = self.layer1(h0)
   ..#more code
    m1_ = self.f_conv1(x)
    m1 = m1_ * h1
    out = self.avgpool(m1)
    out = out.view(out.size(0), -1)
    y = self.linear(out)

   return y, h1, m1

Many Thanks

#32

Thanks @Miguel_Campos , I have never used it, I will look into it.

1 Like
#33

Yes, you can change the name in get_activation since it’s used as the key for the activation dict.

You can do either and pick the approach which fits your use case better.

#35

Thank you so much @ptrblck

#36

Hi @ptrblck , just a quick follow-up question: just trying to confirm, can I return multiple outputs without using detach(), would that have any effect on back propagation or anything else?

   def forward(self, x):
      h0 = self.conv1(x) 
      h0 = relu(self.bn1(h0))
      h1 = self.layer1(h0)
      out = self.avgpool(h1)
      out = out.view(out.size(0), -1)
      y = self.linear(out)   
      return  h1, out, y

Many Thanks !!

#37

Yes, you can return multiple tensors.
As long as you don’t call .backward on these additional tensors or any tensors calculated from them, they won’t influence the original gradient calculation.