I have a multi-class classification dataset like the below. The number of class is five.
Input: i like summer. the weather is nice.
Output: 3 // WEATHER class index
Then, I created a model. This is working without bugs. But I am not sure that the accuracy is only high when I set batch-size 1. If I set a little larger number (such as 3, 5, 10), the accuracy drops more than 30%.
When I use mini-batch, I padded 0 to fix the length of sequences. Then, make it reversed. For example, the following minibatch is 53x3 (Length x batch_size).
I assume that padding makes something wrong in my model. Or, is it common phenomenon in sequence training?
0 17 0
0 484 0
0 481 0
0 605 0
0 539 0
0 675 0
0 640 0
539 334 0
126 44 0
699 216 0
256 570 0
334 688 539
578 251 126
525 3 563
295 8 256
27 525 334
578 131 701
87 63 578
235 457 71
334 205 525
119 386 457
444 95 35
class Simple(nn.Module):
def __init__(self, vocab_size, embd_size, hidden_size, class_size):
super(Simple, self).__init__()
self.embd = nn.Embedding(vocab_size, embd_size, padding_idx=0)
self.ctx_encoder = nn.GRU(embd_size, hidden_size, bidirectional=True)
self.decoder = nn.Linear(hidden_size*2, hidden_size)
self.last_layer = nn.Linear(hidden_size, class_size)
def forward(self, x):
'''
x: (L, bs) batch first is False
'''
batch_size = x.size(1)
x = self.embd(x) # (bs, L, E)
_, h = self.ctx_encoder(x) # (L, bs, 2H), (2, bs, H)
h = h.view(batch_size, -1) # (bs, 2H)
out = self.decoder(h) # (bs, H)
out = self.last_layer(out) # (bs, class_size)
return F.log_softmax(out, -1)