How to interpret the output of a single neuron?

vision
1

Hi, I’m currently using CrossEntropyLoss for a model with a single neuron as ouptut.

My predictions have the following pattern:

        [ 0.0445],
        [-0.1698],
        [-0.1343],
        [-0.5633],
        [ 0.2206],

To get my classes, do I have to do:

      pred_batch = torch.sigmoid(pred_batch)
      pred_batch[pred_batch >= 0.5] = 1
      pred_batch[pred_batch < 0.5] = 0

or

      pred_batch[pred_batch >= 0] = 1
      pred_batch[pred_batch < 0] = 0

Also, I’m getting some negative loss for my training set, is that normal?

2

Firstly, nn.CrossEntropyLoss() is mostly used when you need multple predictions.
Since you have just one output neuron, you must use nn.Sigmoid() followed by nn.BCELoss() or simply nn.BCELossWithLogits().

Just need a clarification,

  1. You have single output neuron.
  2. Your target spans between 0 and 1.

Consider a batch-size of 7.

  1. pred_batch will be of shape (7,1) since u have single neuron.
  2. target shape will be (7,) with minimum value of 0 and maximum value of 1.
    Right ?

How come you are able to pass these inside CrossEntropyLoss() without getting any error.?

3

Good question, I have no idea.

When I use BCELossWithLogits() I had to do a
label_batch = label_batch.view(-1, 1)

4

Didn’t BCELosswithLogits() do the purpose then ?
If you were using this bce-with-logits, then torch.sigmoid() is the right way as you have posted in the question.
If you’ve used nn.CrossEntropyLoss(), then u apply Softmax(), and pick the class which has the highest value.
In your case, picking the class with highest value would always result in the same class ( zeroth class ), since you have just one neuron.

5

Here’s my code:

    model = models.resnet50(pretrained=pretrain_status)
    num_ftrs = model.fc.in_features
    model.fc = nn.Sequential(
      nn.Dropout(dropout_rate),
      nn.Linear(num_ftrs, 1))

  criterion = nn.BCELossWithLogits() 
  optimizer = torch.optim.SGD(model.parameters(), lr =lr)

  model.train()
  for image_batch, label_batch in tqdm(train_dl):
    
    optimizer.zero_grad()
    image_batch, label_batch = image_batch.to(DEVICE), label_batch.to(DEVICE)
    pred_batch = model(image_batch)
   
    label_batch = label_batch.view(-1, 1)

    batch_loss = criterion(pred_batch.double(), label_batch.double())
    total_train_loss += batch_loss.item()

    batch_loss.backward()
    optimizer.step()

    pred_batch[pred_batch >= 0] = 1
    pred_batch[pred_batch < 0] = 0

I know there’s a mistake in there but I can’t seem to find it

6

Hi @alx,
the code is perfectly fine upto optimizer.step()

Please add these steps after that line

probabilities  = torch.sigmoid(pred_batch)
pred_batch[probabilities>=0.5] = 1
pred_batch[probabilities<0.5]  = 0

This is the first case which you have asked in your question.!