# Reverse Fusion of Node Pairs in Scheduler

**URL:** <https://discuss.pytorch.org/t/reverse-fusion-of-node-pairs-in-scheduler/202350>\
**Category:** torch.\_inductor\
**Created:** [May 8, 2024, 10:03am UTC](https://discuss.pytorch.org/t/reverse-fusion-of-node-pairs-in-scheduler/202350 "2024-05-08T10:03:22Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![NiklasZ](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/8e8cbc/32.png) [@NiklasZ](https://discuss.pytorch.org/u/NiklasZ)\
**Post date:** [May 8, 2024, 10:03am UTC](https://discuss.pytorch.org/t/reverse-fusion-of-node-pairs-in-scheduler/202350/1 "2024-05-08T10:03:22Z")

</div>

Hi,

I’m interested in PyTorch’s operator fusion (still relatively new, so please feel free to correct me). I have been reading the Scheduler’s [source code](https://github.com/pytorch/pytorch/blob/main/torch/_inductor/scheduler.py) and noticed a strange case where it seems to consider the fusability of a pair of nodes `node1, node2` in the normal order: `self.can_fuse(node1, node2)` but also the reverse order: `self.can_fuse(node2, node1)`. Specifically, I’m referring to this code here:

```python
def check_all_pairs(nodes):
    for node1_index, node1 in enumerate(nodes):
        for node2 in nodes[node1_index + 1 :]:
            key = (node1, node2)
            if key in seen:
                continue
            seen.add(key)

            if self.can_fuse(node1, node2):
                possible_fusions.append(key)
            elif (node2.is_template() or node2.is_foreach()) and self.can_fuse(
                node2, node1
            ):
                # foreach fusions and epilogue fusions are order dependent
                possible_fusions.append((node2, node1))

```

[source](https://github.com/pytorch/pytorch/blob/main/torch/_inductor/scheduler.py#L1979C1-L1993C64)

I inspected this function on some trivial functions via `torch.compile()` and it seems that `nodes` is typically a set containing a graph node `n` and all the nodes it outputs to `m_1,...m_k` (it’s probably not that simple, so feel free to correct me there). So we have a set (`n`, `m_1`, … `m_k`). So in the `self.can_fuse(node1, node2)` case we have:

1. Check whether we can fuse `n` with some `m_i+1` (vertical fusion?)
2. Check whether we can fuse `m_i` with `m_i+1` (horizontal fusion?)  
for some `i` in `1...k`.

Now my question is, when is `self.can_fuse(node2, node1)` meaningful? From my understanding this could be wrong as fusing operations in reverse can affect the overall function’s correctness.

Would appreciate someone’s insight here. Many thanks!