Backpropogating the loss of specific classes

I think I misunderstood your question. Did you mean one of these?

  1. Ignore samples in a batch that are not from the selected classes

  2. Use all samples per batch, but backpropage only trough the dimensions out the output vector corresponding to the selected classes (similar to what some people do in deep Q-learning to backpropagate only trought a single action)

I was thinking about option 2. Btw, I think it’s not very simple to fix random seed for dataloader if you use multiple workers. This thread might have some work arounds.