Getting individual backward() info on a loss function with (reduction='none')

[NiceDesk]

Hi all,

Apologies if this has been asked to death, but searching didn’t find anything I could use.

I can use the reduction keyword in a loss function and it will output the specific losses for each sample of my minibatch as a vector. But I’m testing a new algorithm that requires the individual gradients for each of the minibatch terms. Currently running backward() on the loss will only give me the sum over terms, and I cannot figure out how to get a tensor with all of the components in.

So for example, running for a minibatch of 1000 entries:

lf = nn.CrossEntropyLoss(reduction='none')
loss = lf( data , labels )
print loss.shape 

gives

(1000,)

as it should. I can then run e.g.

loss[0].backward()

and it will deposit the gradient d[loss_0] / dx into x[0].grad as expected (although this comes at the same cost as computing the gradient of all 1000 entries), assuming I’ve zeroed the grad before.
Is there an efficient way to recover each d[loss_i] / dx for all i in the minibatch, without getting the sum?

Thanks for reading the post - let me know if anything is unclear.

[tom (Thomas V)]

I think you want something like discussed in this thread (see also the bug report):

Efficient computation of per-sample examples

Additional discussion of [feature request] Simple and Efficient way to get gradients of each element of a sum @tom I hope I interpreted your last post correctly by posting here. The code to reproduce the numbers I see is available here. The main.py file runs different ways of computing individual gradients and checks/times them. Your recommendation seems to be along the line of Goodfellow’s derivation, implemented as goodfellow, timing output under goodf. I don’t think it is a bug per-se, …

It’s not something that you would change at the “loss end” of the computational graph, but at the parameter end.

Best regards

Thomas