Weird `with torch.autograd.profiler.profile(use_cuda=True)` results

                                 Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls
-------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                             aten::to         0.05%      28.825us         1.57%     846.961us     846.961us      18.496us         0.03%     828.160us     828.160us             1
                  aten::empty_strided         0.01%       8.040us         0.01%       8.040us       8.040us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         1.51%     810.096us         1.51%     810.096us     810.096us     809.664us         1.46%     809.664us     809.664us             1
                             aten::to         0.02%       8.829us         1.30%     698.454us     698.454us      15.712us         0.03%     698.208us     698.208us             1
                  aten::empty_strided         0.01%       6.741us         0.01%       6.741us       6.741us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         1.27%     682.884us         1.27%     682.884us     682.884us     682.496us         1.23%     682.496us     682.496us             1
                             aten::to         0.02%       8.701us         0.10%      54.518us      54.518us      15.360us         0.03%      54.880us      54.880us             1
                  aten::empty_strided         0.01%       6.311us         0.01%       6.311us       6.311us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.07%      39.506us         0.07%      39.506us      39.506us      39.520us         0.07%      39.520us      39.520us             1
                             aten::to         0.01%       7.228us         0.06%      30.217us      30.217us      10.304us         0.02%      29.344us      29.344us             1
                  aten::empty_strided         0.01%       3.438us         0.01%       3.438us       3.438us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.04%      19.551us         0.04%      19.551us      19.551us      19.040us         0.03%      19.040us      19.040us             1
                             aten::to         0.01%       7.638us         0.13%      71.446us      71.446us      11.008us         0.02%      70.528us      70.528us             1
                  aten::empty_strided         0.01%       3.721us         0.01%       3.721us       3.721us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.11%      60.087us         0.11%      60.087us      60.087us      59.520us         0.11%      59.520us      59.520us             1
                          aten::slice         0.01%       5.624us         0.01%       7.152us       7.152us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       1.528us         0.00%       1.528us       1.528us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.264us         0.01%       2.734us       2.734us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.470us         0.00%       0.470us       0.470us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.944us         0.01%       2.801us       2.801us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.857us         0.00%       0.857us       0.857us       0.000us         0.00%       0.000us       0.000us             1
                            aten::sum         0.07%      38.494us         0.08%      41.391us      41.391us      41.344us         0.07%      41.344us      41.344us             1
                          aten::empty         0.01%       2.897us         0.01%       2.897us       2.897us       0.000us         0.00%       0.000us       0.000us             1
                        aten::type_as         0.01%       7.933us         0.08%      40.439us      40.439us       7.552us         0.01%      40.256us      40.256us             1
                             aten::to         0.01%       7.729us         0.06%      32.506us      32.506us      12.224us         0.02%      32.704us      32.704us             1
                  aten::empty_strided         0.01%       3.739us         0.01%       3.739us       3.739us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.04%      21.038us         0.04%      21.038us      21.038us      20.480us         0.04%      20.480us      20.480us             1
                           aten::view         0.01%       3.399us         0.01%       3.399us       3.399us       0.000us         0.00%       0.000us       0.000us             1
                            aten::div         0.05%      26.231us         0.05%      28.614us      28.614us      27.168us         0.05%      27.168us      27.168us             1
                          aten::empty         0.00%       2.383us         0.00%       2.383us       2.383us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.664us         0.01%       3.241us       3.241us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.577us         0.00%       0.577us       0.577us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.012us         0.00%       2.326us       2.326us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.314us         0.00%       0.314us       0.314us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.861us         0.00%       2.441us       2.441us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.580us         0.00%       0.580us       0.580us       0.000us         0.00%       0.000us       0.000us             1
                            aten::sub         0.04%      23.023us         0.05%      24.994us      24.994us      26.592us         0.05%      26.592us      26.592us             1
                          aten::empty         0.00%       1.971us         0.00%       1.971us       1.971us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.417us         0.01%       2.858us       2.858us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.441us         0.00%       0.441us       0.441us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.061us         0.00%       2.388us       2.388us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.327us         0.00%       0.327us       0.327us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.806us         0.00%       2.184us       2.184us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.378us         0.00%       0.378us       0.378us       0.000us         0.00%       0.000us       0.000us             1
                     aten::zeros_like         0.02%       9.266us         0.06%      34.875us      34.875us      14.208us         0.03%      34.816us      34.816us             1
                     aten::empty_like         0.00%       2.448us         0.01%       4.493us       4.493us       0.000us         0.00%       0.000us       0.000us             1
                          aten::empty         0.00%       2.045us         0.00%       2.045us       2.045us       0.000us         0.00%       0.000us       0.000us             1
                          aten::zero_         0.02%       8.284us         0.04%      21.116us      21.116us       7.168us         0.01%      20.608us      20.608us             1
                          aten::fill_         0.02%      12.832us         0.02%      12.832us      12.832us      13.440us         0.02%      13.440us      13.440us             1
                          aten::slice         0.00%       2.425us         0.01%       2.889us       2.889us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.464us         0.00%       0.464us       0.464us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.885us         0.00%       2.214us       2.214us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.329us         0.00%       0.329us       0.329us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      11.693us         0.02%      12.144us      12.144us      12.288us         0.02%      12.288us      12.288us             1
                     aten::as_strided         0.00%       0.451us         0.00%       0.451us       0.451us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.130us         0.00%       2.515us       2.515us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.385us         0.00%       0.385us       0.385us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      10.538us         0.02%      11.102us      11.102us      10.240us         0.02%      10.240us      10.240us             1
                     aten::as_strided         0.00%       0.564us         0.00%       0.564us       0.564us       0.000us         0.00%       0.000us       0.000us             1
                             aten::to         0.01%       7.161us         0.06%      29.683us      29.683us       9.344us         0.02%      28.768us      28.768us             1
                          aten::empty         0.01%       2.816us         0.01%       2.816us       2.816us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.04%      19.706us         0.04%      19.706us      19.706us      19.424us         0.04%      19.424us      19.424us             1
                      aten::unsqueeze         0.01%       5.503us         0.01%       6.034us       6.034us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.531us         0.00%       0.531us       0.531us       0.000us         0.00%       0.000us       0.000us             1
                            aten::mul         0.04%      23.639us         0.05%      25.936us      25.936us      24.832us         0.04%      24.832us      24.832us             1
                          aten::empty         0.00%       2.297us         0.00%       2.297us       2.297us       0.000us         0.00%       0.000us       0.000us             1
                            aten::add         0.04%      20.709us         0.04%      22.143us      22.143us      21.888us         0.04%      21.888us      21.888us             1
                          aten::empty         0.00%       1.434us         0.00%       1.434us       1.434us       0.000us         0.00%       0.000us       0.000us             1
                            aten::sub         0.03%      18.169us         0.04%      19.918us      19.918us      25.600us         0.05%      25.600us      25.600us             1
                          aten::empty         0.00%       1.749us         0.00%       1.749us       1.749us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.501us         0.01%       3.035us       3.035us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.534us         0.00%       0.534us       0.534us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.885us         0.00%       2.218us       2.218us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.333us         0.00%       0.333us       0.333us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      10.748us         0.02%      11.258us      11.258us      10.816us         0.02%      10.816us      10.816us             1
                     aten::as_strided         0.00%       0.510us         0.00%       0.510us       0.510us       0.000us         0.00%       0.000us       0.000us             1
                           aten::view         0.00%       2.158us         0.00%       2.158us       2.158us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.04%      19.246us         0.04%      19.246us      19.246us      19.648us         0.04%      19.648us      19.648us             1
                          aten::slice         0.00%       2.526us         0.01%       2.996us       2.996us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.470us         0.00%       0.470us       0.470us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       1.756us         0.00%       2.106us       2.106us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.350us         0.00%       0.350us       0.350us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      10.652us         0.02%      11.078us      11.078us      10.752us         0.02%      10.752us      10.752us             1
                     aten::as_strided         0.00%       0.426us         0.00%       0.426us       0.426us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.00%       2.116us         0.00%       2.512us       2.512us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.396us         0.00%       0.396us       0.396us       0.000us         0.00%       0.000us       0.000us             1
                         aten::select         0.02%      10.011us         0.02%      10.417us      10.417us      10.208us         0.02%      10.208us      10.208us             1
                     aten::as_strided         0.00%       0.406us         0.00%       0.406us       0.406us       0.000us         0.00%       0.000us       0.000us             1
                             aten::to         0.01%       7.377us         0.05%      27.906us      27.906us       8.320us         0.02%      26.720us      26.720us             1
                          aten::empty         0.00%       2.613us         0.00%       2.613us       2.613us       0.000us         0.00%       0.000us       0.000us             1
                          aten::copy_         0.03%      17.916us         0.03%      17.916us      17.916us      18.400us         0.03%      18.400us      18.400us             1
                      aten::unsqueeze         0.01%       3.181us         0.01%       3.668us       3.668us       0.000us         0.00%       0.000us       0.000us             1
                     aten::as_strided         0.00%       0.487us         0.00%       0.487us       0.487us       0.000us         0.00%       0.000us       0.000us             1
                            aten::mul         0.04%      19.252us         0.04%      20.895us      20.895us      20.672us         0.04%      20.672us      20.672us             1
                          aten::empty         0.00%       1.643us         0.00%       1.643us       1.643us       0.000us         0.00%       0.000us       0.000us             1
                            aten::add         0.03%      17.770us         0.04%      19.141us      19.141us      19.488us         0.04%      19.488us      19.488us             1
                          aten::empty         0.00%       1.371us         0.00%       1.371us       1.371us       0.000us         0.00%       0.000us       0.000us             1
                            aten::sub         0.03%      17.869us         0.04%      19.398us      19.398us      25.632us         0.05%      25.632us      25.632us             1
                          aten::empty         0.00%       1.529us         0.00%       1.529us       1.529us       0.000us         0.00%       0.000us       0.000us             1
                          aten::slice         0.01%       2.847us         0.01%       3.431us       3.431us       0.000us         0.00%       0.000us       0.000us             1
-------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 53.808ms
CUDA time total: 55.443ms

It seems the columns only add up to 3.84% of the total. Also, does this mean the total time is the CPU + the CUDA time, or does it run the graph in CPU and CUDA separately? Thanks in advance for your help.

  1. seems that you didn’t do profile = profile.key_averages() and are viewing ungrouped outputs
  2. there is a default value: profile.table(…, row_limit=100), this cutting explains missing percentages
  3. IIRC, total time reported there is wrong, as it doesn’t account for nested calls (or there is some similar issue)

Thanks Alex! I’ll out profile.key_averages() and row_limit.

re 3: I’ll try summing the columns without a row limit and compare.