When using torch fp16 for inference, a significant decrease in the time consumption of the cuda kernel is observed through nvidia nsight profile. However, there is no change in the overall time consumption when testing end-to-end.
By observing the profile information, it can be confirmed that the computational accuracy used in the inference process is indeed fp16 precision.
Your use case might be CPU-limited and you can verify it by profiling it with a visual profiler (e.g. Nsight Systems). In this case, you could try to use CUDA Graphs as it will reduce the CPU overhead significantly.