I have a graph neural network which has an architure that is roughly:
Backbone → Graph network.
In order to compute an embedding for neighbor nodes, I have to pass them through the backbone, however I don’t want to update the backbone based on gradients from neighbors–only the targets. Is there a way to achieve this?
I think the most simple approach would be to run a forward on the backbone on neighbor nodes and then just pass in the embeddings, but that seems a bit clunky.