# Calculate individual gradients for each example in mini-batch? **URL:** \ **Category:** autograd\ **Created:** [August 27, 2019, 2:54pm UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470 "2019-08-27T14:54:42Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![garland](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/e47774/32.png) [@garland](https://discuss.pytorch.org/u/garland)\ **Post date:** [August 27, 2019, 2:54pm UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470/1 "2019-08-27T14:54:42Z")
Say I have an the output of `y=model(x)` where `y.size()` is [batch\_dimensions, 1]. I would then like to calculate the individual gradients of each example in the output with respect to the `model.parameters()`. Naively, I could do: ```auto grads = [torch.autograd.grad(y[batch], model.parameters(), retain_graph=True) batch in range(batch_dimensions)] ``` but this is serial and slow. I know I can, however, do: ```auto grads = torch.autograd.grad(y, model.parameters(), grad_outputs=torch.ones(y.size())) ``` which is an awful lot faster. However, this seems to sum the gradients _across_ the batches, which is not quite what I want. How could I possibly achieve what I require efficiently? Cheers. ---
**Author:** ![Yaroslav\_Bulatov](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yaroslav_bulatov/32/7017_2.png) [@Yaroslav\_Bulatov](https://discuss.pytorch.org/u/Yaroslav_Bulatov)\ **Post date:** [September 6, 2019, 12:54am UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470/2 "2019-09-06T00:54:08Z")
Try this: > [GitHub - cybertronai/autograd-hacks](https://github.com/cybertronai/autograd-hacks#per-example-gradients) ---
**Author:** ![roylight](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roylight/32/19198_2.png) [@roylight](https://discuss.pytorch.org/u/roylight)\ **Post date:** [July 29, 2020, 7:21pm UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470/3 "2020-07-29T19:21:18Z")
@Yaroslav_Bulatov Hi Can this module support LSTM? Thank you! ---
**Author:** ![richard](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/32/1861_2.png) [@richard](https://discuss.pytorch.org/u/richard)\ **Post date:** [August 3, 2020, 2:19pm UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470/4 "2020-08-03T14:19:34Z")
@roylight, could I ask about your use case for individual per-example gradients? To answer your question, cybertronai/autograd-hacks don’t support PyTorch’s nn.LSTM out of the box, but if you write a custom LSTM (using nn.Linear, some activation functions, and a for-loop), cybertronai/autograd-hacks should be able to help (there is an implementation for per-sample-gradients of nn.Linear in there). ---
**Author:** ![Yaroslav\_Bulatov](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yaroslav_bulatov/32/7017_2.png) [@Yaroslav\_Bulatov](https://discuss.pytorch.org/u/Yaroslav_Bulatov)\ **Post date:** [September 9, 2020, 8:40pm UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470/5 "2020-09-09T20:40:42Z")
Indeed, it doesn’t support anything other than Conv/Linear, so would need some work. In ideal world, an autovectorization operation would be able to do this automatically, but right now it seems manual algebra is required. @richard one common demand for per-example gradient norms is from privacy people – a datapoint with a large gradient could influence the final solution strongly, so you could “leak” information about that datapoint through the final model. ---
**Author:** ![richard](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/32/1861_2.png) [@richard](https://discuss.pytorch.org/u/richard)\ **Post date:** [September 10, 2020, 7:49pm UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470/6 "2020-09-10T19:49:19Z")
@Yaroslav_Bulatov we’re working on prototyping an auto-vectorization operation like JAX’s vmap ([https://github.com/pytorch/pytorch/issues/42368](https://github.com/pytorch/pytorch/issues/42368)) in the hopes that it may be able to compute per-sample-gradients (among other use cases). There’s a slight problem where it doesn’t quite work for the per-sample-gradient use case, because of how PyTorch internals are organized, but we’re figuring that out. I’m curious if you’ve come across other demands for per-example gradients. I’ve spoken with some folks in the differential privacy space but haven’t been able to find other use cases. ---
**Author:** ![Yaroslav\_Bulatov](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yaroslav_bulatov/32/7017_2.png) [@Yaroslav\_Bulatov](https://discuss.pytorch.org/u/Yaroslav_Bulatov)\ **Post date:** [September 10, 2020, 9:19pm UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470/7 "2020-09-10T21:19:20Z")
The other place is optimization research. There are thousands of papers whose implementation would benefit from efficient access to per-example gradients. Few examples – - Optimize faster: cluster gradients, then undersample large clusters, [paper](https://arxiv.org/abs/1405.3080). - Find potentially mislabeled examples by checking if gradients change rapidly, [paper](https://arxiv.org/abs/2008.11600). - Predict generalization by measuring sharpness of feasible solution. This needs variance of per-example Hessians, which are gradients of the gradient function. [paper](https://papers.nips.cc/paper/8049-how-sgd-selects-the-global-minima-in-over-parameterized-learning-a-dynamical-stability-perspective.pdf). Popular optimizers like AdaGrad and natural gradient were formulated in terms of per-example gradients. Because frameworks didn’t provide efficient way to do this, all practical implementations substituted batch gradients as an approximation. Their derivatives (Adam, [KFAC](https://arxiv.org/abs/1503.05671), [Shampoo](https://arxiv.org/abs/1802.09568), [full AdaGrad](http://proceedings.mlr.press/v97/agarwal19b/agarwal19b.pdf)) ended up inheriting this approximation. What if this approximation is bad? Without per-example gradient support, it’s too much work to check. Basically this feature would make PyTorch a more friendly research platform. PS: efficient per-example gradients (as well as per-example Hessians, Hessian vector products, per-example gradient norms, etc) are all just special cases of einsum optimization. Computation for these quantities comes down to a series of multiplications and additions, and einsum optimizer is what tells you how to properly rearrange them. An example of using this perspective to [improve on](https://colab.research.google.com/drive/1VlirNQxEr3I36s7ra7wzYIapNpcB-l97) PyTorch’s Hessian vector product and to compute batch of Hessian [norms efficiently](https://github.com/pytorch/pytorch/issues/28341#issuecomment-545092466) ---
**Author:** ![richard](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/32/1861_2.png) [@richard](https://discuss.pytorch.org/u/richard)\ **Post date:** [September 10, 2020, 9:28pm UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470/8 "2020-09-10T21:28:49Z")
Thank you for the detailed reply! I’ll take a look through the papers. This is really helpful - we’re also exploring other mechanisms to compute per-example-gradients (that aren’t through an auto-vectorization operator) and investigating use cases will help out with the decision making. ---
**Author:** ![christianperone](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/christianperone/32/1054_2.png) [@christianperone](https://discuss.pytorch.org/u/christianperone)\ **Post date:** [October 20, 2020, 10:42am UTC](https://discuss.pytorch.org/t/calculate-individual-gradients-for-each-example-in-mini-batch/54470/9 "2020-10-20T10:42:39Z")
Another project that is worth mentioning: [https://backpack.pt](https://backpack.pt) (from "[BackPACK: Packing more into backprop](https://arxiv.org/abs/1912.10985%22)).