Online hard negative mining (OHNM) - On Background or on every class?

Hi @ptrblck, reading a lot of iterature the OHNM is being suggested as a method to be used with background class (say for object detection). Does it really work? This is my implementation of ohnm where I sort losses from each image from a batch and create a boolean mask for values >top_k:

    loss = F.cross_entropy(logits, batch_labels , weight=class_weights, reduction='none')

    batch_size = loss.shape[0]

    for i in range(batch_size):

        i_loss = loss[i]

        k_per = 0.9

        loss_sorted=torch.sort(i_loss.flatten())

        loss_len=len(loss_sorted[0])

        top_k_index = int(k_per*loss_len)

        ohnm_thresh = loss_sorted[0][top_k_index]

        loss_ohnm = i_loss>ohnm_thresh

        loss[i] = loss[i]*loss_ohnm

    loss = loss.sum()/class_weights[batch_labels].sum()

My question is, how does OHNM help and should it only be applied on the background?

Hard mining (whether on or offline) is designed to help prevent class imbalance from hindering the network’s ability to learn. Typically in object detection there are way more background pixels than object class pixels, so that’s why it is applied on background. There’s no reason you couldn’t take the same approach for an object class that was overrepresented though, it’s just not as common