LSTM accuracy doesn't increase

#1

Hello,

I’m implementing RNN for video classification, but the accuracy doesn’t increase. I use pretrained resnet-50 as features extractor to feed LSTM. In particular I’m worried that I’m messing up the forward step of the network, here is my code

    def forward(self, x, seq):
        state = self._init_state(b_size=len(seq))

        y = []
        for i in range(len(seq)):
            y.append(self.resnet(x[i]))

        y = torch.nn.utils.rnn.pad_sequence(y)
        pack = torch.nn.utils.rnn.pack_padded_sequence(y, seq, batch_first=False)
        z, _ = self.lstm(pack, state)
        z = nn.utils.rnn.pad_packed_sequence(z, batch_first=False)

        t = []
        for i in range(len(seq)):
            t.append( z[0][seq[i]-1,i,:] )

        t = torch.stack(t,0)
        out = self.classifier(t)
        out = self.out(out)

        return out

I attach also the modified collate_fn function of the dataloader

def my_collate(batch):
    frames, l = zip(*batch)
    lengths = []
    for i in range(len(frames)):
        lengths.append(frames[i].shape[0])

    perm_idx = sorted(range(len(lengths)), key=lengths.__getitem__, reverse=True)

    frames_out = [frames[i] for i in perm_idx]
    l_out = [l[i] for i in perm_idx]
    lengths_out = [lengths[i] for i in perm_idx]


    return frames_out, torch.LongTensor(l_out), lengths_out

Do you think there is a bug in my implementation? Thanks