Loss not decreasingn LSTM classification

hi guys, I’ve finally been able to shape my data an start training LSTM network but the Loss doesn’t seem to drop.

My data is 1000 records of ECG signal each belong to one of 17 classes, each of length [3600], I’ve reshaped it to [180,20]. the Adam optimizer keeps giving me a loss that sometimes increase instead of decreasing, I tried SGD optimizer and the best result I had so far with (learning rate=0.01, momentum=0.5, weight decay=0.1,neterov=True) are shown in the pic, the network has 7 layers with 0.1 dropout and hidden size of 200.

If anyone has any Idea about what’s going on please let me know.
P.S: I’s a newbie and this is my first project so please bare with me.2