Hi, I am starting working with PyTorch and learning about deep learning and neural networks.
After different lessons and tutorials, I built my very first simple model. I get a datasets of credit risk customers classification and tried to work on a classification problem. The idea is to classify into “good” and “bad” creditors based on 18 features.
I got 70.5% of accuracy, but this occurs since the first epoch and every iteration never improves. If I change some parameter like hidden neurons, learning rate, etc. the only variation is changing from 70.5% to 29.5% (just the opposite to 100%). The same if I had layers or activation such as ReLU.
Could you help me to understand how to work on it? What approach should I start to improve the model?
Here the Colab link to see the notebook: notebook link
First, I see you are using loss_fn = nn.BCEWithLogitsLoss(). BCEWithLogitLoss already implements the sigmoid function so there is no need to apply it on the result of your model before passing it into the loss function. Also, and I’m not sure here, rounding the results of the sigmoid function will make the predicted labels either 0 or 1.
For a richer training process, you might want your predicted labels to be the raw result of the sigmoid function. This makes a difference between a predicted label of 0.9 and another of 0.56. Assuming the correct label is 1 in both cases, the first predicted label should have a lower error than the second one. Rounding up both values would make the error be the same for every non correct predicted label.
I sugest changing a few things. Instead of
y_pred = torch.round(torch.sigmoid(y_logits)).float() # turn logits → pred probs → pred labls
try doing directly
loss = loss_fn(y_logits.float(), y_train.float()) # Remember BCEWithLogitsLoss already implements sigmoid.
Thanks David, it helped me a lot! First of all for the theorical explanation and then because increased by 8% accuracy (78%). 12% of wrong predictions is still bad, but in my credit risk model, what I really want to avoid are false positive for the negative impact they could have on business. So I will evaluate the percentage of false positive on the test dataset.