Implementing Custom Module with custom backward pass

1

Hi everyone, I’m trying to port this Torch code (https://github.com/anokland/dfa-torch) in Pytorch. It’s the implementation of Direct Feedback Alignment (https://arxiv.org/abs/1609.01596).

The logic of he implementation is this: we need to inject the error signal at the output of each leayer, so we will create a custom SequentialSG model, everytime it sees a special module called ErrorFeedback copies the error signal in the grad_output of that module.

Now in each ErrorFeedback module, during the forward pass we simply copy the input to the output (i.e. we do nothing), during the backward pass instead we perform the random projection of the error on the dimension of the hidden layer. To do this, we have to define a new module but also extend torch.autograd. I followed http://pytorch.org/docs/master/notes/extending.html

So we have to define a class ErrorFeedbackFunction which inherits from torch.autograd.Function, where we define the two static methods forward and backward and a class which inherits from torch.nn.Module.

Now we just have to define a new nn.Module, which uses our new function in the forward pass.

Now if I try to use it.

I get this error:

TypeError: 'ErrorFeedbackFunction' object does not support indexing

What am I doing wrong? Did I miss some steps in my implementation?

2

I was reading again http://pytorch.org/docs/0.2.0/notes/extending.html and I realized I didn’t alias the apply method. The problem is that even when I try with the Linear example provided in the docs, I get:

type object 'Linear' has no attribute 'apply'

I looked through the torch.nn source code and I saw that in the original Linear implementation apply is not used, in fact it is defined in pytorch/torch/nn/functionals.py.
Instead for example Bilinear is using it, even though an apply method is not defined anywhere in the Bilinear class in pytorch/torch/nn/_functions/linear.py.

Can someone explain and walk me through this?

3

are you using atleast pytorch 0.2.0? The apply method is auto-generated via a class transformation. You just need to define forward and backward

4

Now I am. I was still using version 0.1. Now I don’t get strange errors anymore and the code runs until the end.

Still I’m suspicious that my custom backward isn’t being executed. If I put a print statement inside the backward @staticmethod I don’t see anything on my jupyter notebook.

5

oh that’s definitely suspect. it should be printing things.

6

I inserted a print statement in the code sample in the link below. It prints things fine.

7

Ok, my ErrorFeedback module is working properly. Now I need to subclass nn.Sequential and override accUpdateGradParameters, accGradParameters and backward methods.

But if I do something very simple to test, like copying the original accGradParameters method and add a print statement. Nothing is shown on the console. Why is that?

8

accUpdateGradParameters is not pytorch, it’s torch(lua)

9

I was looking at torch/**legacy**/nn/Sequential, but subclassing torch/nn/Sequential. I think it’s easier to port my code to legacy nn and then I will think about another implementation.