Difference between loss.item() and loss.detach().cpu().numpy()

Can anyone please help me with the difference between:

loss.item() and loss.detach().cpu().numpy()

Which one should I use in the training loop ?

Well, one returns a python float and the other one a numpy array.
It really depends on what do you want to use them for.

Thanks for the reply.

If I have a batch size of 16, how will a python float (I am assuming that you meant a single number) help?

I want to collect loss over all the batches in each epoch and average them to get/ calculate “batch loss per epoch” after the end of epoch.

Would like to draw your attention, @ptrblck

loss.item() will return a float that is the average of loss for all samples in your batch in a single iteration. If you want to average loss across all iterations or epochs, you have to implement a class that averages all your losses in iterations or epochs: something like this: examples/main.py at 2639cf050493df9d3cbf065d45e6025733add0f4 · pytorch/examples · GitHub

I thought that a good measure to know that a model is actually learning something could be “average batch loss per epoch”

which could be calculated by collecting all the loss.item() in the epoch and then taking an average.

Also, seems like loss.detach().cpu().numpy() should not be used while training.