Implement AGC optimizer helper class

BongHoe_Koo · August 11, 2022, 9:07am

C++ Optimizer header link
#--------------------------------------------------------------------------------------------------------------------------

github.com

pytorch/pytorch/blob/main/torch/csrc/api/include/torch/optim/optimizer.h

#pragma once

#include <ATen/Tensor.h>
#include <c10/util/Exception.h>
#include <c10/util/flat_hash_map.h>

#include <torch/arg.h>
#include <torch/csrc/Export.h>

#include <algorithm>
#include <functional>
#include <iterator>
#include <memory>
#include <string>
#include <vector>

// Forward declarations confuse Doxygen
#ifndef DOXYGEN_SHOULD_SKIP_THIS
namespace at {
class Tensor;

This file has been truncated. show original

#--------------------------------------------------------------------------------------------------------------------------

Hi pytorch team

I really want to train NFNet model(Nomalization Free series and my custom model) in C++
but the problem is that here is no official AGC optimizer in libtorch (Adaptive Gradient Clipping)

reference link : [2102.06171] High-Performance Large-Scale Image Recognition Without Normalization
reference pytorch code link : DeepLearningStudy/torch/util/nf_helper.py at main · gellston/DeepLearningStudy · GitHub (target AGC python class)

That is why i made some question list for AGC implementation
(FYI : i am c++ developer and i know basic c++ container like std::vector, std::map , blabla )

Is there simple optimizer inheritance tutorial code for libtorch C++?
how to unpack params of optimizer argument and pack again in C++? (for skipping specific parameter)

image937×392 18.4 KB

(DeepLearningStudy/torch/util/nf_helper.py at e51b11565e924becc39355bf712c398902bc323a · gellston/DeepLearningStudy · GitHub)
how can i get p, p.grad in libtorch

image817×430 15.2 KB

(DeepLearningStudy/torch/util/nf_helper.py at e51b11565e924becc39355bf712c398902bc323a · gellston/DeepLearningStudy · GitHub)
how to calculate total model loss in step function C++ (like closure)

DeepLearningStudy/torch/util/nf_helper.py at e51b11565e924becc39355bf712c398902bc323a · gellston/DeepLearningStudy · GitHub
if there is easy way to implement AGC (without inheritance optimizer) please share