BCELoss Problems

1

I am having trouble training a model with BECLoss. This is my model and below that is my training script.

class StartDetector(nn.Module):
    CHECKPOINT_FILENAME_PATTERN = 'StartDetector-{}.tar'

    def __init__(self):
        super(StartDetector, self).__init__()

        self.cnn1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5, stride=1, padding=1)
        self.relu1 = nn.ReLU()
        self.maxpool1 = nn.MaxPool2d(kernel_size=2, stride=1, padding=1)

        self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=1)
        self.relu2 = nn.ReLU()
        self.maxpool2 = nn.MaxPool2d(kernel_size=2, stride=2, padding=1)

        self.cnn3 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5, stride=1, padding=1)
        self.relu3 = nn.ReLU()

        self.dropout = nn.Dropout(0.3)
        self.fcl1 = nn.Linear(64*9*9, 1)
        self.sigmoid = nn.Sigmoid()


    def forward(self, x):

        out = self.cnn1(x)
        out = self.relu1(out)
        out = self.maxpool1(out)
        out = self.cnn2(out)
        out = self.relu2(out)
        out = self.maxpool2(out)
        out = self.cnn3(out)
        out = self.relu3(out)

        out = out.view(out.size(0), -1)

        out = self.dropout(out)
        out = self.fcl1(out)
        out = self.sigmoid(out)

        return out

    criterion = nn.BCELoss()
    optimizer = optim.Adam(model.parameters(), lr=learning_rate)
    evaluator = Evaluator(val_loader)

    while True:
        for batch_idx, (images, labels) in enumerate(train_loader):
            start_time = time.time()
            images, labels = (Variable(images.to(device)),
                              Variable(labels.to(device)))

            logits = model.train()(images)
            loss = criterion(logits.squeeze(1).float(), labels.float())

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            step += 1

I had to change both the logits and labels to floats in order to solve the

RuntimeError: Expected object of scalar type Float but got scalar type Long for argument #2 'target'

Now that the error is gone my model is just not learning. The model seems to do fine using CrossEntropyLoss.

Can anyone give my some guidance on how to use BCELoss for binary classification?

2

There are at least 2 issues in your code:

  1. BCELoss takes sigmoid outputs not logits, if you want to use logits (recommended) use BCEWithLogitsLoss
  2. like the error message recommends, labels should not be of type float but of type long. Hence, just drop the float from labels.float() and it should be fine (or make it long-type if it is some other type: labels.long())

I guess applying 1 & 2 should fix your issues :slight_smile:

3

My model does output from a sigmoid.

        self.cnn3 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5, stride=1, padding=1)
        self.relu3 = nn.ReLU()

        self.dropout = nn.Dropout(0.3)
        self.fcl1 = nn.Linear(64*9*9, 1)
        self.sigmoid = nn.Sigmoid()

What would the output of the model look like in order to not use logits?

4

If you give that sigmoid output to the BCELoss, then it’s correct. But don’t call it logits then, because it can be misleading, because logits is defined logits = sigmoid(z) / (1-sigmoid(z)) = z