im trying out different Hyper-parameters for my model and for that i need to cross-validate over them.
it looks some thing like this:
the issue is that after each variation of Hyper-Parameter, reset is required for the model, otherwise the ‘‘learning’’ will continue where the last one stopped (need to re-initialize the weights after each itiration).
im having problem doing that, help would be much appriciated.
If you are already using a weight init function, you could just call it again to re-initialize the parameters.
Alternatively, you could create a completely new instance of your model. Using this you would have to re-create the optimizer as well.
Another way would be iterate your layers holding parameters and call
.reset_parameters() on them.
thanks for the quick reply,
im kind of new to the Pytorch framework, still trying to get a grasp on it.
thats the model class:
how could i implement the first option you offered, i think it would be the better approach out of 3.
You can define a function to initialize the weights based on the layer and just apply your model on it:
if isinstance(m, nn.Conv2d):
elif isinstance(m, nn.Linear):
You can find the different init functions here.
unfortunately im still having problem to implement that
(not sure what initialization i need to use and how to use the init class)
is it possible some how to “save” the weights before the cross validation and
set the “saved” weights after each itiration ?
That’s also a good idea. You can save the
state_dict and reload it after your experiment. Have a look at the Serialization Semantics.
You use the
weights_init function just by calling
model.apply(weights_init). This will use every submodule to call the function.