RNN's and imbalanced data

[Sai1 (Sai)]

Hi all, I have signal data on which I’m doing binary classification. I trained a Resnet 50 with 1D convs and achieved 85% accuracy. In the process of improving the performance, I’m currently training a Bidirectional RNN with LSTM. I heard a couple of times that RNNs perform really well for sequential data. But, I was able to only achieve 75% with RNN model. I tried different settings for hidden layers and num_layers in LSTM, but nothing seems to have worked. Any suggestions that could help me? The data is imbalanced with 80-20 distribution. Accuracy is not my primary metric. AUC was 81 for RNN and 93 for RESNET.
Thanks in Advance.

[J_Johnson]

Standard accuracy scores mean very little with a highly unbalanced dataset.

If you have an 80:20 distribution, we can get 80% accuracy with standard scoring methods by just writing a few lines of code to always return the 80% class.

Here is an accuracy method which allows you to get class specific accuracy, or a mean of each class accuracy:

How do I get the accuracy of an unbalanced dataset or segmentation task?

This post is to define a Class Weighted Accuracy function(WCA). As questions related to this get asked often, I thought it might help people to post a tool torchers can use and reference here. Why? Take, for example, a classification dataset of kittens and puppies with a ratio of 0.2:0.8 kittens to puppies. A model trained on this dataset might show an overall accuracy of 80% by just predicting “puppies” every time. A WCA function resolves this problem by checking the accuracy of each class s…

Regarding training on an unbalanced dataset, loss functions allow you to pass in a weight(for binary classes) or pos_weight(for multi-class) argument. These will in turn increase the loss ratio for the minority classes and/or decrease the loss ratio for the majority classes.

[Sai1 (Sai)]

Thanks Johnson. I’ve tried weighted loss function quite a few times in the past but never had any success with it. But RNN seems to be doing a lot better with weighted loss function than plain loss function.