Unique nn.Parameter computation by shifting paramater values for gradient computation

1

Hi everyone,
So to do a particular computation in my neural network training I have a unique way I must compute gradients. This involves shifting the values of the parameters, these params are then processed by a function. Currently, i have a class implemented along these lines:

class Network(nn.Module):
    def __init__(self):
        super().__init__()

        self.params = nn.ParameterList(
            [
                nn.Parameter(nn.init.uniform_(torch.empty((bits, 2), dtype=torch.float32), a= -pi, b = pi)),
                nn.Parameter(nn.init.uniform_(torch.empty((depth, bits, 2), dtype=torch.float32), a = -pi, b = pi))
            ]
        )
    def grad_computation(self):
        <code_here>
    def forward(self):
        ...
        return ...

What i want to do is along the following:

  • Make a copy of self.params called something like copy_params
  • Shift the first value of copy_params by +1 and then -1 leaving the rest of the params untouched
  • Do my computation using shifted values
  • Repeat with the next param in self.q_params list
  • Do until at the end of the self. param list

Here is some rough seudo-code:

copy_params = self.q_params.copy() #Cant copy nn.ParameterList?

#Some sort of loop here to loop over the parameters
# Cant do copy_params[i,j]  due to the shape of self.q_params

copy_params[i,j] += 1 
forward = func(copy_params)

copy_params[i,j] -= 1
backward = func(copy_params)

Any help would be greatly appreciated!