How to Penalize Norm of End-to-End Jacobian

[konstmish (Konstantin Mishchenko)]

I guess this post is related

How to efficiently compute gradient for each training sample?

Hi folks, There is a problem that has bothered me for quite a long time. Assume we are minimizing a loss function [WechatIMG4] parameterized by [WechatIMG3], on samples [WechatIMG5] using SGD, where M is the mini-batch size. Since the PyTorch autograd can only be implicitly created for scalar outputs, I am wondering if there is any efficient way to compute the gradient for each sample, i.e., [image], without setting the batch size equals to 1 and compute in a for loop (which is too slow)? Than…

And you might like the following two repos

GitHub - cybertronai/autograd-hacks

GitHub - cybertronai/autograd-lib

Unfortunately, it doesn’t support all types of networks (e.g. no batch norm), but convnets and all types of activations will work with just one backprop.