BCELoss vs BCEWithLogitsLoss on Autoencoder

Hy all, i have a problem in the code below

import torch
import torch.nn as nn
from torch.optim import SGD, Adam
from torch.optim.lr_scheduler import StepLR
import numpy as np
from dataset import Dataset
from torch.utils.data import DataLoader
from torchnet.meter import AverageValueMeter
from torchnet.logger import VisdomPlotLogger, VisdomSaver
from nets import AutoEncoder
import torch.multiprocessing as multiprocessing



def train(model, train_loader, valid_loader, exp_name = "prova",  lr=0.000001, epochs=1000, wd = 0.000001):

  

    optimizer = Adam(params=model.parameters(),lr = lr, weight_decay=wd)
    scheduler = StepLR(optimizer, step_size=50, gamma=0.5)

    # meters
    lossAE_meter = AverageValueMeter()
    

    # device
    device = "cuda if torch.cuda.is_available() else "cpu
    model.to(device)

    loader = {"train": train_loader, "validation": valid_loader}

    loss_AE_logger = VisdomPlotLogger('line', env=exp_name, opts={'title': 'LossAE', 'legend': ['train', 'validation']})
  
    visdom_saver = VisdomSaver(envs=[exp_name])

    last_best_loss = np.inf
    for e in range(epochs):
        for mode in ["train", "validation"]:


            lossAE_meter.reset()
            
            model.train() if mode == "train" else model.eval()

            with torch.set_grad_enabled(mode == "train"):  

                for i, batch in enumerate(loader[mode]):

                    optimizer.zero_grad()
                   
                    x = batch["Input"].to(device)
                    Target = batch["Target"].to(device)

                    code, rec = model(x)
                    input = nn.Sigmoid(rec)
                    lossAE = nn.BCELoss(input, Target)
                    

                    l = lossAE
                    if mode == "train":

                        l.backward()
                        optimizer.step()

                    else:
                        if l < last_best_loss:
                            torch.save(model.state_dict(), 'Best_%s.pth' % exp_name)
                            last_best_loss = l



                    n = x.shape[0]  

                    lossAE_meter.add(lossAE.item() * n, n)
       



                    if mode == "train":
                        loss_AE_logger.log(e + (i + 1) / len(loader[mode]), lossAE_meter.value()[0]*100, name=mode)
                      



            loss_AE_logger.log(e + (i + 1) / len(loader[mode]), lossAE_meter.value()[0]*100, name=mode)
           
        scheduler.step()

        #salviamo l'ambiente su visdom
        visdom_saver.save()

        # conserviamo solo l'ultimo modello sovrascrivendo i vecchi, salviamo anche il best di volta in volta
        torch.save(model.state_dict(), '%s.pth' % exp_name)



    return model

My model is an autoencoder that try to reconstruct a binarize image, if i pass the output of my autoencdoer to nn.Sigmoid() i receive this error:

Traceback (most recent call last):
  File "/home/giuseppe/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3325, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-1c293c75e8af>", line 1, in <module>
    runfile('/home/giuseppe/Scrivania/EXP/main.py', wdir='/home/giuseppe/Scrivania/EXP')
  File "/home/giuseppe/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/giuseppe/pycharm-2019.2/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/giuseppe/Scrivania/EXP/main.py", line 13, in <module>
    train.start_all()
  File "/home/giuseppe/Scrivania/EXP/train.py", line 140, in start_all
    model_trained= train(model, train_loader, valid_loader, exp_name="Mvm_AE", epochs=400)
  File "/home/giuseppe/Scrivania/EXP/train.py", line 75, in train
    input = nn.Sigmoid(rec)
TypeError: __init__() takes 1 positional argument but 2 were given

Where am i wrong?
Furthermore i would like to use BCEWithLogitsLoss, what should I pass as pos_weight?

1 Like

Hi Giuseppe!

You are calling the constructor for Sigmoid incorrectly, namely with
an explicit argument. (There is also the hidden self argument. That
is why the error message complains about two arguments.)

In more detail, torch.nn.Sigmoid is a class, while torch.sigmoid()
is a function (as is the deprecated torch.nn.functional.sigmoid()).
So you have to construct an instance of Sigmoid before you can call
it as a so-called function object. (You can construct it on the fly, if you
like.)

Thus:

activation = nn.Sigmoid()
...
input = activation (rec)

# or

input = nn.Sigmoid() (rec)

# or

input = torch.sigmoid (rec)

(All three do the same thing.)

Class weights in loss functions are typically used to account for
unbalanced datasets where certain classes occur significantly more
frequently than others.

For a binary classification problem – including a multi-label, multi-class
problem, which is basically nClass binary problems being processed
“in parallel” – you have two classes, “negative” and “positive.”

Common practice is to use pos_weight = n_negative / n_positive,
passed into the constructor of BCEWithLogitsLoss. (In the multi-label
case, you would pass in a vector of nClass pos_weight values, one
for each of your classes.)

(For whatever reason, BCELoss doesn’t take a pos_weight
constructor argument.)

As a side note, you are correct to prefer BCEWithLogitsLoss,
without the explicit Sigmoid, over BCELoss.

Best.

K. Frank

One only answer: the best reply that I ever read.
Thank you very much :slight_smile:

Which way do i have to use the input and target if i use a BCEWithLogitLoss? I think a way like this:
criterion = torch.nn.BCEWithLogisLoss
criterion(input, target)
Why i have numbers different from 0 and 1?
I think i have to use the sigmoid function when i infer the model, but the minimum number is 0.5 for all element :pensive: Why?