I have a model that looks like this:
RegressionRNN(
(ms_group): MorphosynGroup(
(rlayer): GRU(147, 256, batch_first=True, bidirectional=True)
)
(ft_group): FastTextGroup(
(rlayer): GRU(147, 256, batch_first=True, bidirectional=True)
)
(concat): Concatenator()
(linear_layer): Linear(in_features=512, out_features=1, bias=True)
)
This is a model with two different kind of inputs. One goes into ms_group, the other goes into ft_group. The results of these groups are then concatenated, and on the concatenation the linear layer is then executed. I feel, though, that the model as represented above is not semantically clear about what is being concatenated. The concatenator class looks like this
class Concatenator(nn.Module):
def __init__(self):
super(Concatenator, self).__init__()
def forward(self, inputs):
if len(inputs) > 1:
catted = torch.cat(inputs, dim=1)
else:
catted = inputs[0]
return catted
It is simple enough, but I wanted to put this in its separate group so that you can see from the model representation that somewhere a concatenation is taking place. I feel, though, that improvements are possible but I’m not sure how. Should I place the ms_group, ft_group, and concat in their own group (e.g. ‘encoder’), and the linear layer as the decoder layer?
The code is written in a highly modular way so that you can choose to only use ms_group, or only ft_group, and based on the used number of groups the results are concatenated or not, and so on. So I wish to make it more visibly clear what is going on but I’m not sure what the conventions in PyTorch are to do this.