Getting output of intermediate layers for a test input

Hi all,

I have trained a graph attention network using Pytorch_geometric (although, I am pretty sure this question is Pytorch specific) - apologies if it is not.

I want to get node embeddings now from the first GATConv layer on the original but also test graphs that have been slightly modified. I have read about hooks etc, but am struggling to implement these as they do not make perfect sense to me (new to Pytorch framework)

given a model that looks like this:

  (gat_conv1): GATConv(3517, 8, heads=8)
  (gat_conv2): GATConv(64, 4, heads=1)

can someone assist me in getting the outputs specifically for the first GATConv layer (gat_conv1)

thank you very much

This answer from the forums provides a simple implementation of a forward hook to obtain the output of a particular model layer.

Hi @tataganesh,

thank you very much for your response - this works… however, The hook is no different to doing the following:

model.gat_conv1(data) - the output is identical to above and its slightly faster.

Also, the embeddings do not seem to be normalised in both instances

Am I missing something?

  • Generally, forward hooks are used to obtain output from intermediate layer(s) when a forward pass is performed through the whole network. For instance, you can register a forward hook for all your conv layers. Then, if you perform a forward pass with model(input), the hook will store the outputs of your conv layers. In your case model.gat_conv1(data) works because it is the first layer of the network. However, you would have to perform another forward pass to get model(data). On the other hand, a forward hook will fetch you the output of both gat_conv1 and the model with one single forward pass. You can choose to use the forward hook depending on your use case.

  • I have not used Pytorch Geometric, so I do not know about why the embeddings are not normalized. Does GatConv output a normalized tensor?

Hi @tataganesh ,

got it! this makes perfect sense - I will stick to using model.gat_conv1(data) as this is all i require for a certain task i am doing!

on the second point I need to check (i don’t know, the short answer to your question) - but when using Tensorflow, taking the output of the first GATConv layer normally returns normalised constrained values. I will do some digging!

thank you so much for your help!

1 Like