Cross Validation, Model reset

1

im trying out different Hyper-parameters for my model and for that i need to cross-validate over them.
it looks some thing like this:

image
image1007×503 17.8 KB

the issue is that after each variation of Hyper-Parameter, reset is required for the model, otherwise the ‘‘learning’’ will continue where the last one stopped (need to re-initialize the weights after each itiration).

im having problem doing that, help would be much appriciated.

2 Likes
2

If you are already using a weight init function, you could just call it again to re-initialize the parameters.
Alternatively, you could create a completely new instance of your model. Using this you would have to re-create the optimizer as well.
Another way would be iterate your layers holding parameters and call .reset_parameters() on them.

2 Likes
3

thanks for the quick reply,
im kind of new to the Pytorch framework, still trying to get a grasp on it.

thats the model class:

image
image1008×668 37.1 KB

how could i implement the first option you offered, i think it would be the better approach out of 3.

4

You can define a function to initialize the weights based on the layer and just apply your model on it:

def weights_init(m):
    if isinstance(m, nn.Conv2d):
        torch.nn.init.xavier_uniform(m.weight.data, nn.init.calculate_gain('relu'))
        m.bias.data.zero_()
    elif isinstance(m, nn.Linear):
        ...

model.apply(weights_init)

You can find the different init functions here.

2 Likes
5

unfortunately im still having problem to implement that
(not sure what initialization i need to use and how to use the init class)

is it possible some how to “save” the weights before the cross validation and
set the “saved” weights after each itiration ?

6

That’s also a good idea. You can save the state_dict and reload it after your experiment. Have a look at the Serialization Semantics.

You use the weights_init function just by calling model.apply(weights_init). This will use every submodule to call the function.