The error is: RuntimeError: Mask shape should match input shape; transformer_mask is not supported in the fallback case.
I am using TransformerEncoder for a very very simple task – encoding a batch of sequences only 6 tokens long.
Some of the tokens are noninformative, so I am masking them out with src_key_padding. The layers are defined as simple as:
encoder_layer = nn.TransformerEncoderLayer(d_model=self.latent_dim, nhead=transformer_att_heads, dim_feedforward=transformer_ffn_dim, dropout=dropout, activation=transformer_actn, norm_first=norm_first, batch_first=batch_first) # We set batch_first to False to disable "fast path" (which leads to NestedTensor that can't be hooked) and enable the use of hook
self.transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=transformer_num_layers)
and in forward(), it’s also as simple as:
out = self.transformer_encoder(src = sequence_embedded, src_key_padding_mask = mask)
where sequence_embedded is of [1300, 6, 256] and mask is of [1300, 6] (a bool matrix, I make sure no rows sum to 6, so that each sample has at least one token to be passed into Transformer).
And then the error comes:
RuntimeError: Mask shape should match input shape; transformer_mask is not supported in the fallback case.
The weird thing is that when I use VSCode debug mode and run this command again when the bug pops, it works perfectly!! But I just couldn’t make it work natively. Any thoughts?