Apply multiple heads to single tensor

1

Hi,

say I have an input tensor x and my model contains multiple heads where each head processes x separately. Afterwards, the outputs shall be concatenated. If the heads were linear layers this would not be a problem as I could model this as a single linear layer. However, the heads are convolutional networks. So far I have tried putting all the heads in a ModuleList and then iterating over the ModuleList, each time appending the new result to some existing tensor. However, this is really slow. Are there better approaches to do this?

Thanks!

2

Hi Cherubim!

I saw a suggestion in an answer to a related question: Consider adding
each result to a regular python list, and then run torch.cat() on that
list. This way you don’t incur the expense of a tensor-append for every
loop iteration.

Best.

K. Frank

3

Thank you I will try that!