PyTorch Profiler CUPTI warning

1

Hi!
I have run into some CUPTI warning in PyTorch 1.9.0

In PyTorch 1.8.0,

with torch.autograd.profiler.profile(use_cuda=True) as prof:
 y = model(x)
prof.export_chrome_trace("trace.json")

The following code works and chrome trace shows both CPU and CUDA traces.

Whereas in PyTorch 1.9.0,

with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA]) as prof:
 y = model(x)
prof.export_chrome_trace("trace.json")

gives me a warning of /home/surya/.local/lib/python3.6/site-packages/torch/autograd/profiler.py:440: UserWarning: CUPTI tracing is not available, falling back to legacy CUDA profiling warn("CUPTI tracing is not available, falling back to legacy CUDA profiling") and shows only CPU traces.

Any idea why this warning shows up and how to fix it? Thanks in advance for your reply.

I have been working on with Jetson Xavier AGX and installed PyTorch binaries from here

2

I guess the linked binaries did not build PyTorch with CUPTI support so you could check, if you could rebuild them (or generally build from source).

3

I rebuild the wheel from source by adding the flag USE_KINETO=1.
Still facing the same issue.

>>>torch.profiler.supported_activities()
{<ProfilerActivity.CPU: 0>}

Is there any flags which I have to setup for CUPTI support?
Thanks in advance for any replies.

4

The build log should show if CUPTI was found and which support Kineto is thus using.
Since the output only mentioned the CPU support I guess that CUPTI wasn’t found (or is not installed on your system).

5

Hmm yeah. I checked up the logs and it showed CUPTI not found.

I was able to backtrack it and fix up the issue. Manually exporting USE_CUPTI_SO=1 and rebuilding from source helped me fix the issue.

The default build option is setup with CUPTI off.

option(USE_KINETO "Use Kineto profiling library" ON)
option(USE_CUPTI_SO "Use CUPTI as a shared library" OFF)

Thanks again for your help.

1 Like
6

Thank you for the follow up! :slight_smile:

1 Like