Getting output of intermediate layers for a test input

[amjass]

Hi all,

I have trained a graph attention network using Pytorch_geometric (although, I am pretty sure this question is Pytorch specific) - apologies if it is not.

I want to get node embeddings now from the first GATConv layer on the original but also test graphs that have been slightly modified. I have read about hooks etc, but am struggling to implement these as they do not make perfect sense to me (new to Pytorch framework)

given a model that looks like this:

GATmodel(
  (gat_conv1): GATConv(3517, 8, heads=8)
  (gat_conv2): GATConv(64, 4, heads=1)
)

can someone assist me in getting the outputs specifically for the first GATConv layer (gat_conv1)

thank you very much

[tataganesh]

This answer from the forums provides a simple implementation of a forward hook to obtain the output of a particular model layer.

Forward hook activations for loss computation

@Alexander_Riedel a very interesting use case. I would definitely like to know the reasons why you want the same feature map for two successive conv layers You will have to make some tweaks to the code For L1 Loss, both the outputs need to be the same, so you need to ensure that the number of channels are the same You need to resize the smaller width, height to the larger width, height so that you can pass that to the L1 Loss. You can leverage torch resize for this In your hook functio…

[amjass]

Hi @tataganesh,

thank you very much for your response - this works… however, The hook is no different to doing the following:

model.gat_conv1(data) - the output is identical to above and its slightly faster.

Also, the embeddings do not seem to be normalised in both instances

Am I missing something?

[tataganesh]

• Generally, forward hooks are used to obtain output from intermediate layer(s) when a forward pass is performed through the whole network. For instance, you can register a forward hook for all your conv layers. Then, if you perform a forward pass with model(input), the hook will store the outputs of your conv layers. In your case model.gat_conv1(data) works because it is the first layer of the network. However, you would have to perform another forward pass to get model(data). On the other hand, a forward hook will fetch you the output of both gat_conv1 and the model with one single forward pass. You can choose to use the forward hook depending on your use case.

• I have not used Pytorch Geometric, so I do not know about why the embeddings are not normalized. Does GatConv output a normalized tensor?

[amjass]

Hi @tataganesh ,

got it! this makes perfect sense - I will stick to using model.gat_conv1(data) as this is all i require for a certain task i am doing!

on the second point I need to check (i don’t know, the short answer to your question) - but when using Tensorflow, taking the output of the first GATConv layer normally returns normalised constrained values. I will do some digging!

thank you so much for your help!