How can i inference speed up in libtorch c++

I tried to improve the inference speed of my model using the C++ programming language, but I encountered slow inference times. Here are the results of my tests:

  1. I ran the same LibTorch C++ code on both Windows and Linux, and I noticed that the inference speed was faster on Linux compared to Windows.
  • Windows environment:
    • GPU: RTX 3060
    • CUDA: 11.7
    • LibTorch version: 2.0.0
    • Resulting frame rate: 11 fps
  • Linux environment:
    • GPU: RTX 2080 Ti
    • CUDA: 12.1
    • LibTorch version: 2.0.0
    • Resulting frame rate: 19 fps
  1. I also tested my LibTorch program on both Visual Studio 2022 and WSL Ubuntu 22.04. Surprisingly, when I tested it on Visual Studio 2022, I achieved a frame rate of 11 fps, while WSL only gave me 9 fps.
  • Windows environment:
    • GPU: RTX 3060
    • CUDA: 11.7

Now, I’m looking for suggestions on how to improve my inference times. In conclusion, the inference speed on Linux is approximately twice as fast as on Windows.

Any advice on improving the inference speed would be greatly appreciated.

Thnak you