How to call cudaProfilerStart()/cudaProfilerStop()?

There is torch.cuda.profiler.profile() - and seems there is no documentation for it (though one can easily find source code)? wonder if it’s intentionally ‘hidden’? It works fine for me but only for 1 device (GPU)

At the same time can’t make torch.autograd.profiler.profile() working (with use_cuda=True in particular) - i.e. nvprof --profile-from-start off doesn’t profile anything (“No kernels were profiled.”). Also tried using emit_nvtx as per this example:
https://pytorch.org/docs/stable/_modules/torch/autograd/profiler.html#emit_nvtx

and it gives me :

  File "/opt/anaconda3/lib/python3.7/site-packages/torch/autograd/profiler.py", line 309, in __enter__
    torch.autograd._enable_profiler(torch.autograd.ProfilerState.NVTX)
RuntimeError: can't change kind of profiling (e.g. NVTX to CPU) while profiler is running

Wonder if i’m doing smth wrong?

Separately (from user perspective) torch.cuda.profiler seem to be more natural namespace (as strictly speaking i might not have any models at all running, but rather measuring some tensor operations)

Thanks

So ended-up doing this… (but sure there is better solution)

with torch.cuda.device(d1):
    torch.cuda.profiler.cudart().cudaProfilerStart()
with torch.cuda.device(d0):
    torch.cuda.profiler.cudart().cudaProfilerStart()

x0 + 1
x1 + 1

with torch.cuda.device(d1):
    torch.cuda.profiler.cudart().cudaProfilerStop()
with torch.cuda.device(d0):
    torch.cuda.profiler.cudart().cudaProfilerStop()
1 Like