How to build a custom connections

1

I have an adjacency matrix and based on that I would like to build a costume connection between the input and the first hidden layer. I was wondering if someone can help me understand how I should go about the implementation?

2

Does your custom adjacency matrix have the same shape as the weight matrix? Would it be possible to mask the connections with a multiplication (if your matrix contains ones and zeros of course)?

3

Yes, lets say that my input has 100 features, and the first hidden layer has 50 nodes. And my adjacency matrix has the following shape: 100x50 and it’s zero or one. Would it be enough to multiply the weight matrix by the adjacency to remove someone the connections and not let the any backpropagation through the nodes that have 0 adjacency?

4

The gradients will be zero at the masked points:

x = torch.randn(1, 10)
weight = nn.Parameter(torch.randn(5, 10))
adj = torch.randint(0, 2, (5, 10)).float()

out = torch.matmul(x, (adj * weight).t())
out.mean().backward()

print(weight.grad)
> tensor([[ 0.1776, -0.1401,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,
         -0.0000, -0.1063],
        [ 0.1776, -0.0000,  0.0133, -0.0294,  0.0000,  0.3073,  0.1285, -0.0000,
         -0.0000, -0.0000],
        [ 0.1776, -0.0000,  0.0000, -0.0000,  0.0000,  0.3073,  0.1285, -0.0000,
         -0.0000, -0.0000],
        [ 0.1776, -0.1401,  0.0000, -0.0000,  0.0944,  0.3073,  0.1285, -0.1295,
         -0.0000, -0.1063],
        [ 0.1776, -0.1401,  0.0133, -0.0000,  0.0944,  0.0000,  0.0000, -0.1295,
         -0.1472, -0.1063]])
print(adj)
> tensor([[1., 1., 0., 0., 0., 0., 0., 0., 0., 1.],
        [1., 0., 1., 1., 0., 1., 1., 0., 0., 0.],
        [1., 0., 0., 0., 0., 1., 1., 0., 0., 0.],
        [1., 1., 0., 0., 1., 1., 1., 1., 0., 1.],
        [1., 1., 1., 0., 1., 0., 0., 1., 1., 1.]])

and won’t add any information to the gradient which flows to the preceding layer, if that’s what you are looking for.

2 Likes