I am trying to solve a classic unbalanced classification problem where I have class 1 and class 0 with distribution ratio of 0.04/0.96. I need to correctly identify the minority class 1 (0.04% of all the data samples). For that, I created a simple feed-forward neural network using BCECriterion as a loss function. Now, I tried random undersampling and ensemble approach both giving me around +5-7% accuracy improvement. Last thing I wanted to try is cost sensitive loss function. I know that BCECriterion has weigths parameter but it is somewhat “ill defined” and difficult to understand when reading the source code. Before I assumed that naively giving a tensor [0.9, 0.1] to binary criterion would work but it doesn’t. Can somebody please explain what is the purpose of weights parameter in this criterion and given the problem above how should I apply weights to BCECriterion?
Have you tried balancing the data in sk-learn prior to using it in PyTorch?
One example is provided here:
https://www.kaggle.com/varunsharma3/credit-card-fraud-detection-using-smote-0-99-auc
I am aware of imbalance-learn lib and yes, I tried using it but unfortunately my model preprocessing is different from what they use and I have no confidence that it would dramatically improve the score because I am already using random undersampling and I also implemented ensemble learning trying 5 up to 40 models trained on different parts of the data with majority voting. It improved the f1 score only slightly. Using sklearn however does not answer the question.
So, last thing I want to try is loss function weights to be sure that feed forward network is not capable of classifying this data well enough.