What is the default initial weights for pytorch-geometric SAGEconv layer and other convolution layers? and how to initialize them using Xavier?

1

Hello,
what is the default initial weights for pytorch-geometric SAGEconv layer and other convolution layers? and how to initialize them using Xavier?

I need guidance in how to apply Xavier initialization in graph neural network, I am using pytorch geometry

I tried this, but it didn’t work,

def initialize_weights(self):
for m in self.modules():
if isinstance(m,SAGEConv):
nn.init.xavier_normal_(m.weight)
if m.bias is not None:
nn.init.constant_(m.bias, 0)

it neither goes to the first if statement, and gives error “AttributeError(f”‘{type(self).name}’ object has no attribute ‘{name}’“) AttributeError: ‘MeanAggregation’ object has no attribute ‘bias’”

2

The default PyTorch conv class uses Kaiming uniform for weights and a standard uniform distribution for biases.

IMG_20231221_220720
IMG_20231221_2207201240×1138 157 KB

I’m not sure for SAGEConv, but for regular PyTorch layers, you can just define a custom conv class, and reference the parent accordingly, then redefine the reset_parameters() function. Or just modify your code to something like this:

def initialize_weights(self):
    for m in self.modules():
        if isinstance(m,SAGEConv):
            nn.init.xavier_normal_(m.weight.data)

Note the .data.

3

when I tried using m.weight.data, I still get this error,
‘SAGEConv’ object has no attribute ‘weight’
do you have any idea ?

4

Had a look at the SAGEConv module docs and it appears they just create a separate linear class, also defined within that library:

IMG_20231226_072027
IMG_20231226_0720271240×2304 134 KB

So calling on the SAGEConv class won’t target the weights. Instead, you may need to change SAGEConv to Linear. But given it’s not a torch.nn.Linear, that might not work, too.

They also have a specific reset_parameters method:

IMG_20231226_072345
IMG_20231226_0723452467×382 85.3 KB

Digging further, we can find the specific reset_weight definition:

IMG_20231226_072636
IMG_20231226_0726361920×883 80.2 KB

And it appears that takes an initialization argument.

However, I do not see Xavier listed as an initialization option.

Perhaps you could check on the Slack linked on their docs homepage:
https://pytorch-geometric.readthedocs.io/en/latest/index.html

Seems to me the issue is not related to PyTorch.

5

I tried to print out the weights and I found that it is initialised by default with uniform Kaiming, and the weights are in conv1.lin_l.weight and conv1.lin_r.weight. for each SAGEconv layer. Reaching both of these weights, I could initialise them using Xavier.
thanks.

1 Like