I have the following function I’m using to initialize weights for testing purposes. (I need a deterministic function that is something non-trivial. I.e, not all 0s or 1s).
def init_weights(m): # Avoid random initializations if isinstance(m, nn.Linear) or isinstance(m, nn.Conv3d) or isinstance(m, nn.LayerNorm): print("initializing other layers") # Choose this b/c it's deterministic normal_(m.weight) elif isinstance(m, nn.Parameter): print("initializing parameters") normal_(m) else: raise Exception("unknown layer")
The problem is, this function does not seem to initialize parameters (i.e.,
nn.Parameter). I can tell b/c
initializing parameters is never printed. Is there a good solution here?