Grad_fn get whole graph in dot?

autograd
1

It is possible to get whole graph in grad_fn for each iteration with the calculated gradients?

In dot format or other?

Also I find curios that if I print

grad_fn <AliasBackward object at 0x7fd60006ccf8>---------------------------------------------------| 0.00% [0/11 00:00<00:00]
grad_fn <AliasBackward object at 0x7fd5d025b748>---------------------------------------------------| 9.09% [1/11 00:00<00:09]
grad_fn <AliasBackward object at 0x7fd60212d2e8>---------------------------------------------------| 18.18% [2/11 00:01<00:07 0.3831]
grad_fn <AliasBackward object at 0x7fd5d0171fd0>---------------------------------------------------| 27.27% [3/11 00:02<00:05 0.3686]
grad_fn <AliasBackward object at 0x7fd5d01711d0>---------------------------------------------------| 36.36% [4/11 00:02<00:04 0.3569]
grad_fn <AliasBackward object at 0x7fd5d0171fd0>████-----------------------------------------------| 45.45% [5/11 00:03<00:03 0.3541]
grad_fn <AliasBackward object at 0x7fd5d01711d0>███████████----------------------------------------| 54.55% [6/11 00:03<00:03 0.3449]
grad_fn <AliasBackward object at 0x7fd5d0171f28>███████████████████--------------------------------| 63.64% [7/11 00:04<00:02 0.3349]
grad_fn <AliasBackward object at 0x7fd5d0171fd0>███████████████████████████------------------------| 72.73% [8/11 00:04<00:01 0.3285]
grad_fn <AliasBackward object at 0x7fd5d0171fd0>███████████████████████████████████----------------| 81.82% [9/11 00:05<00:01 0.3174]
grad_fn <AliasBackward object at 0x7fd5d0171fd0>███████████████████████████████████████████--------| 90.91% [10/11 00:06<00:00 0.3130]
3         0.296759    0.051774    0.985694  00:07                                                                                      
grad_fn <AliasBackward object at 0x7fd60006be10>---------------------------------------------------| 0.00% [0/11 00:00<00:00]
grad_fn <AliasBackward object at 0x7fd5d025b7b8>---------------------------------------------------| 9.09% [1/11 00:00<00:09]
grad_fn <AliasBackward object at 0x7fd5d025b7b8>---------------------------------------------------| 18.18% [2/11 00:01<00:06 0.2870]
grad_fn <AliasBackward object at 0x7fd5d025b7b8>---------------------------------------------------| 27.27% [3/11 00:02<00:06 0.2805]
grad_fn <AliasBackward object at 0x7fd5d025b6a0>---------------------------------------------------| 36.36% [4/11 00:02<00:05 0.2727]
grad_fn <AliasBackward object at 0x7fd602160b38>████-----------------------------------------------| 45.45% [5/11 00:03<00:04 0.2671]
grad_fn <AliasBackward object at 0x7fd5d025b7b8>███████████----------------------------------------| 54.55% [6/11 00:04<00:03 0.2676]
grad_fn <AliasBackward object at 0x7fd5d025b7b8>███████████████████--------------------------------| 63.64% [7/11 00:04<00:02 0.2604]
grad_fn <AliasBackward object at 0x7fd5d025b7b8>███████████████████████████------------------------| 72.73% [8/11 00:05<00:02 0.2553]
grad_fn <AliasBackward object at 0x7fd5d025b7b8>███████████████████████████████████----------------| 81.82% [9/11 00:06<00:01 0.2495]
grad_fn <AliasBackward object at 0x7fd5d025b7b8>███████████████████████████████████████████--------| 90.91% [10/11 00:06<00:00 0.2435]

I see sometimes the same object (reused?) and other times not, some one know why?

2

Hi,

This package will return a dot graph: https://github.com/szagoruyko/pytorchviz

The objects are re-used because the first one goes out of scope and is free. But later one, since you redo an allocation of the same size, the same memory is returned to you (many allocator do caching for allocations of the same size).

1 Like
3

I see, so that behaviour is more a python thing, for me it was weird to look at the numbers “poping randomly” like that

4

This is a behavior of the allocator as well actually. If the object gets destroyed then its memory can be re-used :slight_smile: