Hello everyone,
I have a tensor of size (batch_size, seq_length, embed_dim) and after passing through a linear layer I would like to get an input of (batch_size, latent_dim). The part of the code is shown below
class TransformerEncoder(nn.Module):
def __init__(self, head, embedding_size, pos_embed, latent_dim = 512):
super(TransformerEncoder, self).__init__()
self.head = head
self.embedding_size = embedding_size
self.pos_embed = pos_embed
self.self_attention = SelfAttention(head, embedding_size)
self.fc1 = nn.Linear(embedding_size, latent_dim)
self.fc2 = nn.Linear(embedding_size, latent_dim)
self.dropout = nn.Dropout(0.1)
def forward(self, x):
batch_size, max_words = x.size()[:2]
x = self.self_attention(x)
x = self.dropout(x)
pos_x = self.pos_embed.get_postional_embeddings()
x = (x + pos_x)
x = x.view(batch_size, -1)
mean, logvar = self.fc1(x), self.fc2(x)
return mean, logvar
the variable x
is of 3 dimensions, I don’t know the value of batch_size and seq_length until the forward propagation starts. So there will be a size mismatch in the operation self.fc1(x)
and in self.fc2(x)
. One solution I can think of is to assign a large value to seq_length and pad all the inputs to that length. But this would slow down the learning process. Is there anything else that can be done?
Thanks