Obtaining outputs and attention weights from intermediate Transformer layers

Hi,

I am starting to use nn.TransformerEncoder for some experiments and was wondering if there was a way to obtain the outputs and attention weights from intermediate layers?

For example, Table 7 in the BERT paper, studies the feature extraction capabilities of BERT and utilizes outputs from intermediate layers. The attention weights would generally help with analyzing the results.

Hence, I was wondering if there was a way to obtain them easily.

Thank you.

1 Like

I’m not sure what you mean by obtain is, but you can definitely inspect, dump the weights for any layer in the transformer. If you set up for transformer model something like this:
model = Transformer(args), you can print/access he intermediate weights like this model.encoder.layers[n].self_attn.W_Q.
Of course the exact keys(encoder, self_attn, etc) would change based on how you set up your model but it’s pretty straightforward.