How to calculate running loss/training loss while training a CNN model

1

Can anyone please tell me if there is a specific method to calculate the running loss/training loss? I came a few formulae while looking for it

  1. train_loss = train_loss + ((1 / (batch + 1)) * (loss.data - train_loss))
  2. train_loss += loss.item().

Below is the code while using it:
#train model
def train(n_epochs,model,loader,optimizer,criterion,save_path):

for epoch in range(n_epochs):
    train_loss = 0

    net.train()
    for batch, (data,target) in enumerate(loaders['train']):
        
        target = target.view(target.size(0),-1)
        target = Variable(target)

        optimizer.zero_grad()

        outputs = net(data)
        
        loss = criterion(outputs,target)
        loss.backward()
        optimizer.step()
        #calculate training loss
        train_loss = train_loss + ((1 / (batch + 1)) * (loss.data - train_loss))

        #print results
        if batch % 100 == 0:
            print("Epoch: {}, Batch: {}, Training Loss: {}".format(epoch+1, batch, train_loss/1000))

print("Finished Training")

I want to know if there is a particular way to calculate it?

2

I like the approach used in the ImageNet example using an AverageMeter and updating it on the fly.

3 Likes
3

I think its common to calculate it like this:

batch_loss += loss.item() * batch_size

Then at the end of the epoch we divide the number with the number of steps.

We multiply it with the batch size since loss.item() returns the average loss for each sample within the batch, so to get per sample overall we have to correct for that.

1 Like