Getting output of torch.utils.bottleneck

1

Sorry, for this very basic question.

I run utils.bottleneck as described (https://pytorch.org/docs/stable/bottleneck.html). It runs. Everything finishes. And nothing else happens. Is there supposed to be output? If so, how do I get it?

3 Likes
2

You should get an output in your terminal.
Does your terminal just return after the call without printing anything?

2 Likes
3

Yes. Nothing is printed.

4

Does your script finish its execution?

5

Wow, cool! Didn’t know this existed. Gotto try this :smile:

6

Hi @ptrblck, I am facing the same issue. I run my script with python -m torch.utils.bottleneck main.py -args, the script gets executed, and nothing gets printed. I am using Pytorch version 1.1.0, on a machine with 4 P100 GPUs. Is there something I am missing regarding the inputs?

I also did not find many resources while trying to understand what happens under the hood, could you point me to any?

7

How long would this script take to finish its execution without the bottleneck call?
Make sure it’s finishing in a reasonable time, e.g. just use some iterations instead of a lot of epochs.
Could this be an issue for your use case?

8

I ran it for just one epoch, and this time it printed the output as expected, unlike previous times where I used a manual sys.exit(0) to exit after a specific statement, and it did not return anything on the screen. So I guess that should be the issue.

2 Likes
9

I don’t have any exit calls, do just one single step and then break out of the training loop, no output.

>>> python -m torch.utils.bottleneck train.py --config-file config/config.gin
`bottleneck` is a tool that can be used as an initial step for debugging
bottlenecks in your program.

It summarizes runs of your script with the Python profiler and PyTorch's
autograd profiler. Because your script will be profiled, please ensure that it
exits in a finite amount of time.

For more complicated uses of the profilers, please see
https://docs.python.org/3/library/profile.html and
https://pytorch.org/docs/master/autograd.html#profiler for more information.
Running environment analysis...
Running your script with cProfile
python -m torch.utils.bottleneck train.py --config-file config/config.gin  24,21s user 8,77s system 170% cpu 19,396 total
>>>
2 Likes
10

I face the same issue. Did you get the solution?

11

For me, this problem was solved by running the script for one epoch instead of multiple like @ptrblck suggested.

12

I still have this problem. The tool finishes, but nothing gets printed.
I am using torch 1.7.1

Is it possible that all exceptions are caught?
Btw my network is quite deep. Torchviz recently throwed an exception because recursion depth was reached.

13

Hi, sorry to revive a pretty old topic but I also stumbled upon this issue. I am using torch 1.10.1 and ignite 0.4.7. I tried all of the suggested things above, such as explicitly terminating the training script by sys.exit(0) or reducing the num_epochs to 1. I also tried

        torch.autograd.profiler.profile(enabled=True)
        torch.autograd.profiler.emit_nvtx(enabled=True)
        torch.autograd.set_detect_anomaly(mode=True)

There was also a different issue on GitHub which suggested invoking the bottleneck module by

python -m torch.utils.bottleneck -- src/models/train_model.py data/processed config/experiment.yaml

instead of

python -m torch.utils.bottleneck src/models/train_model.py data/processed config/experiment.yaml

No matter what I do, I get the prompt

bottleneck is a tool that can be used as an initial step for debugging
bottlenecks in your program.

It summarizes runs of your script with the Python profiler and PyTorch’s
autograd profiler. Because your script will be profiled, please ensure that it
exits in a finite amount of time.

For more complicated uses of the profilers, please see
The Python Profilers — Python 3.12.0 documentation and
Automatic differentiation package - torch.autograd — PyTorch master documentation for more information.
Running environment analysis…
Running your script with cProfile

and my logs appear, after the training finishes I don’t see any performance analysis.

14

Double check there is no exit() trailing at the end of your file, or anywhere…

15

I didn’t have any obvious exit(), however I use click as the command line parser. Do you think this might be affecting?