Change the tanh backward code

vision
1

Hi, there.

I want to change the backward behavior of tanh.
So, i have to touch the source of torch.
But i don’t know where my downloaded torch code exist.
So,

  1. where is my torch source code exist in my computer? (I am using anaconda)
  2. Where is the directory of tanh’s backward function?
  3. do i have to do something after changing the source code? (i.e. recompile? re-source?)

here is my bashrc file.

image
image1556×585 76.1 KB

2

You won’t be able to change the pre-built binaries so would need to build PyTorch from source as described here.
The easier approach to change the backward method of a specific operation would be to implement a custom autograd.Function and write the backward method manually as described e.g. here.

1 Like
3

Oh, thank you @ptrblck.
Let me try that way.

But, before doing it, can you specify where the original tanh’s backwar function is?
I could find the forward function of it, but i cannot find the backward of tanh’s.

I want to check the exact math expression of tanh’s backward function.
How backward of tanh is coded. So, I want where is this directory is.

Thx.

4

The CPU kernel is defined here and the CUDA kernel here.

1 Like
5

Thank you, @ptrblck .
now, I know what i have to do.

6

Hi, again. @ptrblck

Thanks for the kernel code. But I want more. in the referenced page,

void tanh_backward_kernel_cuda(TensorIteratorBase& iter) {
  if(isComplexType(iter.dtype())) {
    AT_DISPATCH_COMPLEX_TYPES(iter.dtype(), "tanh_backward_complex_cuda", [&]() {
      gpu_kernel(iter, [] GPU_LAMBDA(scalar_t a, scalar_t b) -> scalar_t {
        return a * std::conj(scalar_t{1.} - b * b);
      });
    });
  } else {
    AT_DISPATCH_FLOATING_TYPES_AND2(at::ScalarType::Half, at::ScalarType::BFloat16, iter.dtype(), "tanh_backward_cuda", [&]() {
      gpu_kernel(iter, [] GPU_LAMBDA(scalar_t a, scalar_t b) -> scalar_t {
        return a * (scalar_t{1.} - b * b);
      });
    });
  }
}

In these code, I want to find the where the ‘b’ came from.
So, can you give me the “upper code” of that cuda kernel?

7

a and b would be the inputs to tanh_backward, which should be grad_output and output.

1 Like
8

Yes, @ptrblck i also presume that a and b are grad_output and output from these kind of mathematical sources (Coding Neural Network - Forward Propagation and Backpropagtion | Imad Dabbura).
But i want to know how these cuda function are connected to python code(“upper code”) just to study this.

And i have to know exactly what output is.
For example, (tanh function) y = tanh(x), output should be y, right?

Then, at the forward path, it must be something like ‘ctx.save_for_backward(output)’.
So, i presume those kind of things. But i want to see it.

I searched myself through all the directories of pytorch github. But I could not understand the structure of it. So, i lost myself and don’t know where to go.

Thank you.