I’m tring to initialize a sparse model (in_channels are sparsed 50%) to train it from scratch, I don’t know it is a correct way as the code show below
for module_name, module, p_name, p, mask, pruned in cls.__sparse_parameters: kaiming_normal_(p, mask, nonlinearity='relu')
p will be the weight tensor and mask is the binary-mask and I multiply it with weight then generating a sparse weight.And in the function
_calculate_fan_in_and_fan_out, I just change the
num_input_fmaps = tensor.size(1) -> num_input_fmaps = mask[0,:,0,0].sum()
What confuse me is that by this way the recptive_field_size always equals
s so the std will not grow when layer become deeper.
I’m not familer with weight inilization so really hope someone can help me out!