In the middle of the forward pass of a network, if I convert a feature tensor to numpy, do some operations in numpy and convert back to Tensor, how does this affect the backward pass exactly?
Your numpy conversion will break Autograd’s ability to create the backward automatically, so that you would need to write your own backward.
Here are some more information on Autograd.