Loss dosn't decrase at all when using lstm

1

Hi,
I implemented a classification model to fix a Five-class problem, but I got a problem and the work stopped, the output of net, everything in batch is 1(class No2), while the real label is 0,1,2,3,4.
why this happen? and how? Thank u, everyone!!
Here is my code:

def forward(self, input):

    if input is None or input.size()[-1] != self.input_size:
        return None
    out1,_= self.lstm(input,self.lstm_states)  
    lstmoutsize = self.hidden_size * 1 if self.bidirectional == 0 else 2
    lstmout = out1[:, -1, :]  
    lstmout = lstmout.view([-1,lstmoutsize]) # lstmout = batchsize * lstmoutsize
    outFcLstm = self.fcLstm(lstmout)             
    self.reset_hidden_states()

    return outFcLstm

main:

    criterion = T.nn.CrossEntropyLoss()
    lr = opt.lr
    optimizer = T.optim.Adam(rnn.parameters(), lr=lr, weight_decay=opt.weight_decay)

    while True:
        batch = self.queBatchSample.get()
        if batch is None:  
            break
        optimizer.zero_grad()
        output= rnn(batchData)
        if output is None:
            continue
        loss = criterion(output, batchLabel)
        loss.backward(retain_graph=True)
        optimizer.step()**