How can I get output of intermediate hidden layers in a Neural Net to be passed as input explicitly to the hidden layer in a pretrained model to get the final layer output?

Let us assume I have a trained model saved with 5 hidden layers (fc1,fc2,fc3,fc4,fc5,fc6).

Suppose I need to get output of Fc3 layer from the existing model,
BY defining

def get_activation(name):
def hook(model, input, output):
activation[name] = output.detach()
return hook

hidden_fc3_output = model.fc3.register_forward_hook(get_activation(‘fc3’)) …

How can I use this hidden_fc3_output to pass it explicitly from fc4 layer to get the final fc6 output ?

hidden_fc3_output will be the handle to the hook and the activation will be stored in activation['fc3'].
I’m not sure to understand the use case completely, but if you would like to pass this stored activation to fc4 and all following layers, you could create a switch in your forward method and pass it to the model. This would split the original forward path into two steps.
You could of course still use the forward hook if needed.
Have a look at this code snippet and let me know, if something would be useful or if I misunderstood your use case:

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        self.fc2 = nn.Linear(1, 1)
        self.fc3 = nn.Linear(1, 1)
        self.fc4 = nn.Linear(1, 1)
        self.fc5 = nn.Linear(1, 1)
        self.fc6 = nn.Linear(1, 1)
        self.act = nn.ReLU()

    def forward_half1(self, x):
        x = self.act(self.fc1(x))
        x = self.act(self.fc2(x))
        x = self.act(self.fc3(x))
        return x
    def forward_half2(self, x):
        x = self.act(self.fc4(x))
        x = self.act(self.fc5(x))
        x = self.fc6(x)
        return x
    def forward(self, x, path='all'):
        if path=='all':
            x = self.forward_half1(x)
            x = self.forward_half2(x)
        elif path=='half1':
            x = self.forward_half1(x)
        elif path=='half2':
            x = self.forward_half2(x)
            raise NotImplementedError
        return x

model = MyModel()        
x = torch.randn(1, 1)        
out1 = model(x, path='half1')
out2 = model(out1, path='half2')
out = model(x)
print(out2 == out)

out1 = model(x, path='half1')
print(out1 == activation['fc3'])
out2 = model(activation['fc3'], path='half2')
print(out2 == out)
1 Like

Thanks ptrblck for the reply :slight_smile:

I was trying to understand the difference between defining (midoutput) architecture in forward definition and calling register_forward_hook function.
Why is it that I get a difference in predicting output if I use register_forward_hook.
It seems that model(x,path) is more accurate.

Example -

Actual Output - [8.1009,7.4015,7.7013]
Predicted Output
By model(model(x,path=‘half1’),path=‘half2’) – [8.1009,7.4015,7.7013]
By register_forward_hook in fc3 layer and then calling model(activation[‘fc3’],path=‘half2’) – [8.3453,7.6397,7.9421]

There is a huge difference in the predictions.

The reason is activation [’‘fc3’’] and model(x,path=‘half1’) are not the same.
Any specific reason for this?


In my example the hooked activation as well as the output using path='half1' yield the same result.
Are you using any batchnorm layers or other layers with running stats?
If so, note that the running estimates will be changes in each forward pass, so that the next one will yield different results.
For the sake of debugging you could set the model to eval mode (model.eval()) and test it again.