Hello everyone, this is my first post on the Pytorch forums - I hope you are keeping well!
I understand that what I am going to ask is detailed and/or advanced. But please bare with me - I am willing to go all the way
I want to track what (layer module) accessed my tensor and when my tensor was accessed.
For example - when training:
- A tensor is used in the forward computation of a layer.
1a) The output produced by that layer is then saved for autograd on the later backwards pass.
- Then during the backpass of the autograd, that same tensor is used to calculate gradients for the layer that produced it.
By recording when steps 1) and 2) are started, a tensor access pattern can be observed. The goal of this is to identify the time difference between the 1st and 2nd accesses. (I am a grad student - please feel free to ask more about my work)
I am posting here just to hear from some other voices who are more knowledgable and experienced in Pytorch.
Thank you for your time.