Build time of release build

I am using libtorch 1.8.0 with Cuda in my C++ project. My project is build in Visual Studio 2019, C++17. The problem is build time of release build. Debug build is created under a minute, however, release build takes up to 10 minutes. I know that release builds take longer time in generall, but this seems way too much for an app with a single, small convolution network.

Is there any way how to speed up this process? I have tried to use precompiled headers, but took was even longer.