Cyclic computational graph

autograd
1

Hey! I’m new to Pytorch. Sorry in advance if this is a stupid question. My general question: is it possible to build a cyclic computational graph with or without autograd? I do notice that computational graph is DAG by its mathematical origin (i.e. decomposition of mathematical expression ) and the in-place autograd is really hard in current Pytorch, but if we merely tackle this question from building a graph with tensor and autograd-like mechanism, is it possible? If possible, then what kind of Neural network structure is this computational graph representing? Maybe this question does not make sense at all and i’m going to very wrong direction, can you help me sort out the reason of it? Thanks a lot~:smiley:

2

I am not sure to understand 100%, but it seems to me that your processor must be able to travel in time if you want to make such computation.

I mean, if the computational graph is cyclic (let say the simplest case, with 2 nodes), you need the result of operation 1 in order to compute operation 2, in order to compute operation 1. So you need the result of operation 1 in order to compute its own result.

Otherwise, you think about a time decomposition (so you need the result at time 0 for the result at time 1) but then the graph is acyclic

3

Thank you for the explanation. Time decomposition is a good point.

But What defines a computational graph(node, outgoing and incoming edge)? You provided two way to define a computational graph(with “procedural” and “object-oriented”).

And let’s take say we have a leaf tensor T0 with initialized value, a math operation OP0 , a outgoing edge EG0 from T0 to OP0, a outgoing edge EG1 from OP0 to T1 and a outgoing edge EG2 from the output T1 to input T0( or directly from OP0 to T0).
Briefly,
T0 ====(EG0)=====> OP0 =====(EG1)====> T1
|<================<==(EG2)==============|

Will this graph I presented count as a valid and cyclic one?
Sorry for the bad “drawing” with these symbols.

4

Then again, with your example, if you take time into account (otherwise it’s impossible), then EG2 is not from T1 to T0 but from T1(t=0) to T0(t=1), and your computational graph is acyclic. If fact, per definition, a computational graph is acyclic.

So, in your example, the graph looks like (no need to name edges) :

T0(t=0) ==> OP0(t=0) ==> T1(t=0) 
                           ||
                           ||
                           \/
                         T0(t=1) ==> OP0(t=1) ==> T1(t=1) 
                                                   ||
                                                   ||
                                                   \/
                                                T0(t=2) ==> OP0(t=2) ==> T1(t=2) 
                                                                         (...)

(as you may notice, I’m a painter in my free time)

5

Thanks for the reply~ It seems that the nature of Comp Graph is time-decomposed…