Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler

torch.compile
1

I wanted to share a blog post that I (along with my colleagues at Hugging Face) have written on the torch.profiler. It covers profiling a matmul and addition operation. I think this could serve as a great tutorial for anyone beginning their journey into profiling, optmization, or even kernel developement.

Link: Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler

Any feedback, criticisms, or observations are very welcome!

PS: I don’t know if the category was set correctly or not, please feel free to let me know if that is not the case.

2

It’s great.

When do you think you can post Part 2? Looking forward to it.

3

Hello @ariG23498 (and fellow colleagues at Hugging Face :hugs:),

I wish I had read your guide a few months ago! Special thanks for including both the profiling code and walking us though what is happening behind the hood!

I recall seeing a larger cudaDeviceSynchronize in Perfetto for the first time a few months ago and being very, very confused.

:red_question_mark:Question: How would you produce figure to visually show the GPU speedup between 64x64 (Figure 3) and a 4096x4096 trace (e.g. to demonstrate how this speed conforms to Amdahl’s law)?