I recently implemented a model with two branches. Though the computational graphs of the two branches have no connections, they both modify a shared GPU storage when backward. The losses of the two branches are summed and so backward together. The non-derterministic backward result makes me realize that the backward of the computation graph is not single-threaded but multi-threaded.

Multi-threaded backward depending on the autograd graph is smart. But how can I identify the gradient computation of some parts of a model is multi-threaded? Are there some clear guidelines or rules? In which scope two disconnected parts will do backward in the multi-threaded way?

e.g.

```
a = conv1(x)
b = conv2(x)
c = a + b
c.sum().backward()
```

Is it certain that the gradients of `a`

and `b`

will be computed in multiple threads?