I think I misunderstood your question. Did you mean one of these?
-
Ignore samples in a batch that are not from the selected classes
-
Use all samples per batch, but backpropage only trough the dimensions out the output vector corresponding to the selected classes (similar to what some people do in deep Q-learning to backpropagate only trought a single action)
I was thinking about option 2. Btw, I think it’s not very simple to fix random seed for dataloader if you use multiple workers. This thread might have some work arounds.