Thanks hughpekins. I like your explanation. It is intuitive and straight forward.
I reckon the Network forward propagation seems asychronized too. Do you have any idea about that? Or are all pytorch codes executed just “return immediately” instead of “return after real executing” ?