Using LSTM stateful for passing context b/w batches; may be some error in context passing, not getting good results?

I have checked the data before giving it to network. Data is correct.

I have multi-class classification problem based upon an imbalanced dataset

Dataset_type: CSV

Dataset_size: 20000

Based upon CSV data of sensors

X = 0.6986111111111111,0,0,1,0,1,0,0,0,1,0,0,0,0,1,0,0,0,1,1,0,0,0

Y = leaveHouse

Per class accuracy:
{‘leaveHouse’: 0.34932855, ‘getDressed’: 1.0, ‘idle’: 0.8074534, ‘prepareBreakfast’: 0.8, ‘goToBed’: 0.35583413, ‘getDrink’: 0.0, ‘takeShower’: 1.0, ‘useToilet’: 0.0, ‘eatBreakfast’: 0.8857143}


# Using loss weights, the inverse of class frequency

criterion = nn.CrossEntropyLoss(weight = class_weights)

 hn, cn = model.init_hidden(batch_size)
            for i, (input, label) in enumerate(trainLoader):
                input = input.view(-1, seq_dim, input_dim)

                if torch.cuda.is_available():
                    input = input.float().cuda()
                    label = label.cuda()
                    input = input.float()
                    label = label

                # Forward pass to get output/logits
                output, (hn, cn) = model((input, (hn, cn)))

                # Calculate Loss: softmax --> cross entropy loss
                loss = criterion(output, label)#weig pram
                running_loss += loss
                loss.backward()  # Backward pass
                optimizer.step()  # Now we can do an optimizer step
                optimizer.zero_grad()  # Reset gradients tensors


class LSTMModel(nn.Module):
    def init_hidden(self, batch_size):
        self.batch_size = batch_size
        if torch.cuda.is_available():
            hn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim).cuda()
            # Initialize cell state
            cn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim).cuda()
            hn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim)
            # Initialize cell state
            cn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim)
        return hn, cn

    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, seq_dim):
        super(LSTMModel, self).__init__()
        # Hidden dimensions
        self.hidden_dim = hidden_dim

        # Number of hidden layers
        self.layer_dim = layer_dim

        self.input_dim = input_dim
        # Building your LSTM
        # batch_first=True causes input/output tensors to be of shape
        # (batch_dim, seq_dim, feature_dim)
        self.lstm = nn.LSTM(self.input_dim, hidden_dim, layer_dim, batch_first=True)

        # Readout layer
        self.fc = nn.Linear(hidden_dim, output_dim)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)
        self.seq_dim = seq_dim

I solved it, error was that Pytorch implicitly applies softmax when you use cross entropy and I was applying it again and also the dim =1 part was wrong, since it was a seq output.