Thanks Alexis for the reply.
If I understand correctly, the following is not generally true,
if you want grad_input = target, then it is the same than: output = target*input + c, for any c.
that is the same than : (output-c)/input = target
provided that grad_input is the gradient of output w.r.t to the input, i.e.
grad_input = d{output}/d{input}.
Sorry for not making it clear.
Actually, If we backward (output-c)/input, we get
d{(output-c)/input}/d{weight} = [d{output}/d{weight}*input - (output-c)*d{input}/d{weight}] / (input*input).
Apparently, this is not
d(d{output}/d{input})/d{weight})
needed by the optimizer.
I don’t think there is work-around for the second derivative. So I’ll keep an eye on the topic you mentioned.