I have a computer vision problem, but it’s probably relevant to other deep learning models.
Sometimes my model will immediately get stuck at a high training loss and won’t be able to improve from there. If I re-train the model several times, eventually one of the training runs results in learning. I’m assuming this has to do with bad initialization of weights, but I was wondering what all I should try to address this.
I have a small dataset (only ~100 positive examples). The problem is binary classification.
For reference here is the model:
class CNN(nn.Module):
def __init__(self, input_dim):
super().__init__()
self.conv1 = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, padding=1)
self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
self.conv3 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)
self.conv4 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding=1)
self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
# Input to classifier is input_dim / 4 because 2 max pool layers with kernel_size of 2
final_dim = int(input_dim / 4)
self.classifier = nn.Sequential(
nn.Linear(in_features=64 * final_dim * final_dim, out_features=128),
nn.ReLU(),
nn.Dropout(),
nn.Linear(in_features=128, out_features=2)
)
def forward(self, x):
x = self.conv1(x)
x = F.relu(x)
x = self.conv2(x)
x = F.relu(x)
x = self.pool1(x)
x = F.dropout(x, p=.25)
x = self.conv3(x)
x = F.relu(x)
x = self.conv4(x)
x = F.relu(x)
x = self.pool2(x)
x = F.dropout(x, p=.25)
x = x.reshape(x.shape[0], -1)
x = self.classifier(x)
x = F.softmax(x, dim=-1)
return x
Using:
- NLLLoss
-Adam optimizer
-lr 1e-3
Data is normalized to .5 mean and .5 std and had range [0, 1]
Thanks!