Libtorch CUDA backpropagation

C++
1

Hello, I want to make custom cuda layer in my model.

I found some examples that make custom forward and backward.

but, i can’t find the example how to teach my model using my gradient on c++.

Do i need to make custom optimizer?

2

@yessyw
We have a end to end example here:
https://pytorch.org/cppdocs/frontend.html (scroll down to end to end example)

Yes, you might need optimizer, we have several optimizer available. check here:
‘https://github.com/pytorch/pytorch/issues/28440’

And yes you can add your own optimizer, they should be the same as how you add forward and backward, same python binding, only the input and output are different.

1 Like
3

Thanks to your reply.

Sorry. my question was not clear.

I want to make custom CUDA layer in my model
It means to make (.cu) file.
So, I have to use Aten tensor which means I can’t use autograd.
I have to make custom forward and backward layer for getting right gradient.
I found some example for make custom forward backwadr layer.
But, I can’t find how to update variables in c++

Can I have to make custom optimizer to update variables or Is there simple way.

4

@yessyw
Correct me if I am wrong, what you want to do is to apply your calculated gradient to the original input tensor, right? If your gradient apply logic is not trivial, you can make your own optimizer, but if you just want to do simple operation, you can do it directly.

eg. if you want to add your gradient to input tensor, you can just do input + grad where input is your input tensor and grad is the calculated gradient. Same as what you do in python.

1 Like
5

Thanks to your reply.

If the original input tensor that you mention means weight, yes that i want to do.
Thanks to your help. I will try to make optimizer.

6

I found some solution for this problem in tutorial. But, I’m not sure. I want to check my code is working right whether or not. This example code is conceptual code.

  1. kernel.cu
#include <ATen/ATen.h>
#include <ATen/cuda/CUDAContext.h>
#include <ATen/core/TensorAccessor.h>

#include "cuda_runtime.h"
#include "device_launch_parameters.h"
using at::Tensor;

Tensor CustomLossFunction(Tensor &grad, const Tensor &A, const long long N){
    auto B = at::zeros(N).to(at::kCUDA);
    LossFunction<<<nBlock, nThread>>>(grad.data<float>(), A.data<float>(), B.data<float>(), N);
   return B;
}
template<typename T>
__global__ void LossFunction(T *grad, T *B, const T *A, const long long N){
    const size_t tid = blockDim.x * gridDim.x + blockIdx.x * blockDim.x + threadIdx.x;
    if(tid < N){
         C[tid] = A[tid] * A[tid] * 2.f;
         grad[tid] = A[tid] * 4.f;
    }
}
  1. main.cpp
#include <ATen/ATen.h>
#include <torch/torch.h>
#include "main.h"
using at::Tensor;

Tensor CustomLossFunction(Tensor &grad, const Tensor &A, const long long N);
int main(){
   Net model;
   model.to(torch::kCUDA);
   torch::optim::SGD optimizer(model.parameters(), torch::optim::SGDOptions(0.001).momentum(0.5));
   auto model_in = DataLoader(fileName).to(torch::kCUDA);
   auto Exact = DataLoader(fileName).to(torch::kCUDA);
   
   optimizer.zero_grad();
   auto grad = torch::zeros(10).to(Torch::kCUDA);
   auto Y = model.forward(model_in);
   auto Loss = CustomLossFunction(grad, Y, 10).mean();
   Y.backward(grad / 10);
   optimizer.step();
}

Updating parameters using custom loss function.