Individual learning rate for each sample

vision
1

Hi,

I’m training CNNs on MNIST dataset and would like to prioritize learning some of the samples (they will be determined during the training) and thought a straightforward way to do this would be to set a learning rate (multiplier) for these “important” samples.

Another way to do this is maybe by duplicating these samples in training and validation sets, but, I think, with dynamic prioritization the code can get too big and complex.

What I’m looking for is basically like weighing in semantic segmentation but instead of weights being static and per class I want them to be dynamic and per sample.

2

Hi,

You can use a weighted loss to change the relative importance of your samples. Indeed, multiplying the loss for a single sample by 2 is the same as multiplying the gradients corresponding to this sample by 2.

3

Thanks, I didn’t think of that, good idea.