# Difference between apply an call for an autograd function

**URL:** <https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845>\
**Category:** Uncategorized\
**Created:** [February 20, 2018, 2:25pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845 "2018-02-20T14:25:38Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![cdancette](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cdancette/32/3531_2.png) [@cdancette](https://discuss.pytorch.org/u/cdancette)\
**Post date:** [February 20, 2018, 2:25pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/1 "2018-02-20T14:25:38Z")

</div>

I created a simple autograd function, let’s call it F (based on torch.autograd.Function).

What’s the difference between calling

```auto
a = F.apply(args)

```

and instantiating, then calling, like this :

```auto
f = F()
a = f(args)

```

The two versions seem to be used in pytorch code, and in examples

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 20, 2018, 2:27pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/2 "2018-02-20T14:27:33Z")

</div>

Hi,

This difference is that instantiating + calling the `Function` works with “old style” functions (which are going to be deprecated in the future).  
Using `.apply` is for the “new style” functions. You can differentiate the two easily: new style functions are defined with only `@staticmethod`, while old style ones have an ` __init__ `.

---

<div class="post-metadata">

**Author:** ![cdancette](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cdancette/32/3531_2.png) [@cdancette](https://discuss.pytorch.org/u/cdancette)\
**Post date:** [February 20, 2018, 2:30pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/3 "2018-02-20T14:30:32Z")

</div>

Ok, I see, thanks. Since I want to have a scalar parameter for my function, if I want to use the new style, I need to pass it as an argument to the forward then, and save it for backwards ?

I did this :

```auto
class F(torch.autograd.Function):
    def __init__ (self, gamma=0.1):
        super(). __init__ ()
        self.gamma = gamma

    def forward(self, args):
        pass
    def backward(self, args):
        pass

```

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 20, 2018, 2:33pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/4 "2018-02-20T14:33:15Z")

</div>

Hi,  
What you wrote is an old style function.

The equivalent new style would be:

```auto
class F_new(torch.autograd.Function):
    @staticmethod
    def forward(ctx, args, gamma):
        ctx.gamma = gamma
        pass

    @staticmethod
    def backward(ctx, args):
        pass

# Using your old style Function from your code sample:
F(gamma)(inp)
# Using the new style Function:
F_new.apply(inp, gamma)

```

---

<div class="post-metadata">

**Author:** ![cdancette](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cdancette/32/3531_2.png) [@cdancette](https://discuss.pytorch.org/u/cdancette)\
**Post date:** [February 20, 2018, 2:35pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/5 "2018-02-20T14:35:43Z")

</div>

Thank you very much !

And so everytime I call F.apply, a new “instance” will be created with its own context ?

Can I save intermediary results with ctx.intermediary = intermediary ?

If I call several times the function before doing a backward pass, will it override the gamma attribute each time, and the intermediary results I saved with ctx ?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 20, 2018, 2:37pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/6 "2018-02-20T14:37:59Z")

</div>

Each call to `.apply` will have a different context. So you can save everything you need in it without risk.  
Note that if you need to save input or outputs, you should use the `ctx.save_for_backward`.

---

<div class="post-metadata">

**Author:** ![cdancette](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cdancette/32/3531_2.png) [@cdancette](https://discuss.pytorch.org/u/cdancette)\
**Post date:** [February 20, 2018, 2:39pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/7 "2018-02-20T14:39:50Z")

</div>

Ok, why do we need to use `save_for_backward` for this ? Is it just a convention ? Or does it performs additional checks ?  
But intermediary tensors are fine to save, right ? I tried to save some with `save_for_backwards` but it failed, so I saved them as attributes in self (ctx now)

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 20, 2018, 2:46pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/8 "2018-02-20T14:46:09Z")

</div>

Yes `save_for_backward` is just for input and outputs, it allows to perform additional checks (make sure that you don’t create non-collectable cycles). For intermediary results, you can save them as attribute of the context yes.

---

<div class="post-metadata">

**Author:** ![cdancette](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cdancette/32/3531_2.png) [@cdancette](https://discuss.pytorch.org/u/cdancette)\
**Post date:** [February 20, 2018, 2:46pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/9 "2018-02-20T14:46:35Z")

</div>

Thanks, it much clear now !

---

<div class="post-metadata">

**Author:** ![acgtyrant](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/acgtyrant/32/4971_2.png) [@acgtyrant](https://discuss.pytorch.org/u/acgtyrant)\
**Post date:** [October 28, 2018, 12:05pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/10 "2018-10-28T12:05:19Z")

</div>

What version does the new style appear?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [October 29, 2018, 2:02pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/11 "2018-10-29T14:02:42Z")

</div>

If I remember correctly, it was for [0.2.0](https://github.com/pytorch/pytorch/releases/tag/v0.2.0) to support higher order derivatives.

---

<div class="post-metadata">

**Author:** ![cosmozhang1988](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cosmozhang1988/32/1593_2.png) [@cosmozhang1988](https://discuss.pytorch.org/u/cosmozhang1988)\
**Post date:** [January 27, 2019, 5:58pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/12 "2019-01-27T17:58:49Z")

</div>

Hi Dear AlbanD, @albanD

It seems the staticmethod is faster than the old sttyle function (ctx comparing with self).  
But what if I need to output some auxiliary output in the forward pass, which does not need to receive its gradient?  
In the old style, what I do is to create another instance method which is separate from the forward pass.  
A more concrete example:

```auto
class F(torch.autograd.Function):
    def __init__ (self):
        super(F). __init__ ()

    def some_func(self, x2):
         //self.x is also needed here
         return z2

    def forward(self, x):
         self.x = x
         z1 = self.some_func(self.x)
         return y
                 
    def backward(self, y.grad):
         return x.grad

y = F()(x)
z2 = F()(x2)
y.backward()

```

Is there a way that this old styled can be re-rewriten into the new style (using ‘ctx’)?  
Thanks a lot!

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [January 28, 2019, 10:21am UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/13 "2019-01-28T10:21:45Z")

</div>

Hi,

This is not what Function are made for. All arguments _must_ be given as input to the forward function.  
To get this behavior, you can simply create an `nn.Module` which can do whatever you want.

---

<div class="post-metadata">

**Author:** ![cosmozhang1988](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cosmozhang1988/32/1593_2.png) [@cosmozhang1988](https://discuss.pytorch.org/u/cosmozhang1988)\
**Post date:** [January 28, 2019, 5:58pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/14 "2019-01-28T17:58:48Z")

</div>

Yeah, my current solution is using `nn.module`, but comparing with `ctx`, it is much more slower…The reason I use `torch.autograd.Function` is because I need to write my own derivative.

---

<div class="post-metadata">

**Author:** ![cosmozhang1988](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cosmozhang1988/32/1593_2.png) [@cosmozhang1988](https://discuss.pytorch.org/u/cosmozhang1988)\
**Post date:** [January 28, 2019, 6:21pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/15 "2019-01-28T18:21:14Z")

</div>

@albanD  
Maybe my description was not very precise. Taking `torch.autograd.Function` as an example, what I am seeking to do is:

```auto
class F(torch.autograd.Function):

   @staticmethod
    def some_func(z1, x2):
         //x2 is also needed here
         return z2

   @staticmethod
    def forward(ctx, x, x2):
         z1 = F.some_func3(x)
         y = z1**2
         z2 = F.some_func(z1, x)

         u = F.some_func2(y) \\derivative wrt x
         ctx.save_for_backward(u)
         return y, z2
                 
    @staticmethod
    def backward(ctx, y.grad, dummygrad):

         u, = ctx.saved_tensors
         return u, None

```

In this case, I do not need x2’s derivative, but I need to output z2 in the forward pass to save some computation cost in the intermediate values.

---

<div class="post-metadata">

**Author:** ![cosmozhang1988](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cosmozhang1988/32/1593_2.png) [@cosmozhang1988](https://discuss.pytorch.org/u/cosmozhang1988)\
**Post date:** [January 28, 2019, 8:44pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/16 "2019-01-28T20:44:21Z")

</div>

Well, my way of writing actually seems working.

---

<div class="post-metadata">

**Author:** ![s\_n](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/278dde/32.png) [@s\_n](https://discuss.pytorch.org/u/s_n)\
**Post date:** [June 2, 2020, 10:41pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/17 "2020-06-02T22:41:53Z")

</div>

Hi @albanD

Inside this forward function can I use register buffers?  
I can’t use self.register\_buffer for sure. Also ctx.register\_buffer doesn’t work.  
What is correct way to define register buffer in forward?

Thanks!

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [June 2, 2020, 11:02pm UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/18 "2020-06-02T23:02:14Z")

</div>

Hi,

`.register_buffer()` is a function on nn.Modules, not autograd.Function.  
These are two completely different constructs. And you won’t need any buffer for custom Function as it is not a class instance, you should just create the Tensor you need during the forward.

---

<div class="post-metadata">

**Author:** ![s\_n](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/278dde/32.png) [@s\_n](https://discuss.pytorch.org/u/s_n)\
**Post date:** [June 3, 2020, 1:10am UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/19 "2020-06-03T01:10:14Z")

</div>

Hi @albanD ,  
If I will just create a tensor how will that tensor be moved to GPU?  
I don’t want to use .to(device) because that won’t work with DataParallel.

Are you saying that all the tensors defined under autograd.Function will be automatically taken to device when I will do model.cuda() or torch.nn.DataParallel(model)?

Example:  
class Myclass(Function):  
@staticmethod  
def forward(ctx,input):  
mytensor = torch.tensor([1])

So a model defined using above class (Myclass) when taken to device (by model.cuda() or torch.nn.DataParallel(model)) will take all the tensors in forward to the device?

---

<div class="post-metadata">

**Author:** ![s\_n](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/278dde/32.png) [@s\_n](https://discuss.pytorch.org/u/s_n)\
**Post date:** [June 3, 2020, 1:18am UTC](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845/20 "2020-06-03T01:18:18Z")

</div>

I think I might have to create a tensor using register\_buffer outside this forward function and then pass it to forward so that my tensors can be used with torch.nn.DataParallel and can be taken to device.

[Next page](https://discuss.pytorch.org/t/difference-between-apply-an-call-for-an-autograd-function/13845.md?page=2)