Hi all!

I’m trying to implement a simple neural network to address a binary classification problem on an imbalanced dataset (21% positive vs 79% negative).

I can’t find any configuration (hyper parameters or model architecture) that make the training/val loss decrease: the loss is highly unstable around the 0.6 value (I assume it’s near the log(2)) so I wondering what could be the problem. I performed a dataset cleaning process in which I reduced the features dimensionality from 45 to 8: however, the correlation of each feature to the label is between 0.13 and 0.21 and this worry me.

To solve the imbalanced problem I oversample the positive data during the training in such a way to have 50-50 pos-neg ratio in each batch.

Actually, this is my model architecture:

```
class Network(nn.Module):
def __init__(self,in_ch,out_ch):
super().__init__()
# Inputs to hidden layer linear transformation
self.fc1 = nn.Linear(in_ch, 16)
self.bn1 = nn.BatchNorm1d(16)
self.fc2 = nn.Linear(16, 32)
self.bn2 = nn.BatchNorm1d(32)
self.fc3 = nn.Linear(32, 64)
self.bn3 = nn.BatchNorm1d(64)
self.fc4 = nn.Linear(64, 32)
self.fc5 = nn.Linear(32, out_ch)
def forward(self, x):
x = F.leaky_relu(self.bn1(self.fc1(x)))
x = F.leaky_relu(self.bn2(self.fc2(x)))
x = F.leaky_relu(self.bn3(self.fc3(x)))
x = F.leaky_relu(self.fc4(x))
return self.fc5(x)
```

I’ve already tried different activations functions (relu and tanh) and also to remove batch normalization.

I use a simple SGD optimizer.

Do you have any suggestion on what could be the problem?