The logic of he implementation is this: we need to inject the error signal at the output of each leayer, so we will create a custom SequentialSG model, everytime it sees a special module called ErrorFeedback copies the error signal in the grad_output of that module.
Now in each ErrorFeedback module, during the forward pass we simply copy the input to the output (i.e. we do nothing), during the backward pass instead we perform the random projection of the error on the dimension of the hidden layer. To do this, we have to define a new module but also extend torch.autograd. I followed http://pytorch.org/docs/master/notes/extending.html
So we have to define a class ErrorFeedbackFunction which inherits from torch.autograd.Function, where we define the two static methods forward and backward and a class which inherits from torch.nn.Module.
Now we just have to define a new nn.Module, which uses our new function in the forward pass.
Now if I try to use it.
I get this error:
TypeError: 'ErrorFeedbackFunction' object does not support indexing
What am I doing wrong? Did I miss some steps in my implementation?
I looked through the torch.nn source code and I saw that in the original Linear implementation apply is not used, in fact it is defined in pytorch/torch/nn/functionals.py.
Instead for example Bilinear is using it, even though an apply method is not defined anywhere in the Bilinear class in pytorch/torch/nn/_functions/linear.py.