I have the following problem: I have a bunch of parameters that I need to project into a the valid parameter space. I can only efficiently compute the valid parameter-space during the forward-pass (it changes with the parameters).
So I have computations like this:
lower_bound, upper_bound = compute_bound(input) param[param < lower_bound] = lower_bound param[param > upper_bound] = upper_bound out = f(input, param)
the thing is…I wonder what that means for the gradients for param? Do they get obstructed by the assignment? Because I just want to replace every value out of bounds with the nearest bound, but still optimise it.