Is it possible to calculate the second-order derivative of a custom CUDA function?

autograd
1

Is possible to calculate the second-order derivative of a custom CUDA function?
When I try to calculate it with a twice backward technique, I got the error as follows.

RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.

I guess the reason is the computation graph is not created while backward pass in the custom CUDA function…

2

Hi,

How do you define your custom CUDA function? :slight_smile:
If you use autograd.Function in python to create new autograd Functions, you need to make sure that your backward can be auto-diffed. If it cannot, then your backward should call the apply of another Function that corresponds to your backward and double-backward.

3

Hi,

Thank you for your answer :grinning:

If it cannot, then your backward should call the apply of another Function that corresponds to your backward and double-backward.

I see! That’s what I want to know!
I’m gonna try to call the apply inside the backward function.

4

@albanD
After I implemented a double backward function following what you said, it finally started working!!
Thank you for your wonderful advise!

1 Like