PyTorch Profiler not working with CUDA

Hi,
For me, Torch.Profiler is not working with CUDA activity only. With CPU it is working for me.

Code:

with torch.profiler.profile(
        activities=[torch.profiler.ProfilerActivity.CUDA],
        schedule=torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=2),        
        on_trace_ready=torch.profiler.tensorboard_trace_handler('./log/model'),
        profile_memory=False,
        record_shapes=True,
        with_stack=True,
        use_cuda=True
    ) as prof:

        for mini_batch_data in dataloader:
              ...

Error Msg: “AssertionError: No activities specified for Kineto profiler”

Is someone able to give me an hint?

Thanks in advance!

I cannot reproduce the issue using this simple code snippet:

import torch
from torch.profiler import profile, record_function, ProfilerActivity

with profile(
        activities=[torch.profiler.ProfilerActivity.CUDA],
        schedule=torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=2),        
        on_trace_ready=torch.profiler.tensorboard_trace_handler('./log/model'),
        profile_memory=False,
        record_shapes=True,
        with_stack=True,
    ) as prof:
        for _ in range(10):
            y = torch.randn(1).cuda() + torch.randn(1).cuda()
            prof.step()
            

print(prof.key_averages())

so could you post a minimal, executable code snippet which would reproduce the issue, please?

Hi ptrblck,
actually I can reproduce the same issue with your code snippet. Here the full error stack:

C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\autograd\profiler.py:440: UserWarning: CUPTI tracing is not available, falling back to legacy CUDA profiling
  warn("CUPTI tracing is not available, falling back to legacy CUDA profiling")
Traceback (most recent call last):
  File "test_profiler.py", line 14, in <module>
    prof.step()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\profiler\profiler.py", line 282, in step
    self._start_warmup()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\profiler\profiler.py", line 413, in _start_warmup
    self.profiler = prof.profile(
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\autograd\profiler.py", line 447, in __init__
    assert len(self.kineto_activities) > 0, \
AssertionError: No activities specified for Kineto profiler

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_profiler.py", line 14, in <module>
    prof.step()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\profiler\profiler.py", line 253, in __exit__
    self._exit_actions()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\profiler\profiler.py", line 404, in _exit_actions
    self._start_trace()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\profiler\profiler.py", line 425, in _start_trace
    assert self.profiler is not None
AssertionError

FYI:

  • Python 3.8.0
  • GPU: NVIDIA Tesla V100-PCIE-16GB
  • torch==1.9.0+cu111
  • torch-tb-profiler==0.2.0

Any tips/ideas?
Or are any additional information needed?

No, unfortunately I don’t have any debugging advice and would recommend to create an issue in GitHub, so that others could try to reproduce it as well.

Problem was issued here: PyTorch Profiler is not working with CUDA · Issue #65393 · pytorch/pytorch · GitHub

To run profiler you have do some operations, you have to input some tensor into your model.

Change your code as following.

import torch
import torchvision.models as models

model = models.densenet121(pretrained=True)
x = torch.randn((1, 3, 224, 224), requires_grad=True)

with torch.autograd.profiler.profile(use_cuda=True) as prof:
    model(x)
print(prof) 

This is the sample of the output I got.

MY code snippet is already performing operations on the device via:

y = torch.randn(1).cuda() + torch.randn(1).cuda()

and while it works for me, it seems that @marewi is still hitting the issue.
Are you seeing the same error using my code snippet?