Which loss function is suitable?

[learner47 (learner)]

Hello @KFrank!

@KFrank

Unless z has some additional structure / meaning (that you have not
shared with us, e.g., being the binary representation of an integer), then
you should understand your problem as being a so-called multi-label,
multi-class classification problem.

z is the binary representation of the class with m bits. As an example, for m=3 z runs from 000 to 111. So z has a particular structure in that it is the exhaustive list of combinations of 1s and 0s.

@KFrank

The most common approach to such a problem is to have your
final Linear layer output m logits (one for how strongly each of
the m bits is predicted to be “on”). Then use BCEWithLogitsLoss
to compare your prediction with your ground-truth z. The output
of your model will be of type float32 and have shape [nBatch, m].
Your ground-truth target will have the same shape and also be of
type float32 and will have value 1.0 whenever the corresponding
bit of z is set.

I have two questions here. One, for this to work should the activation function at the end of my network be nn.Sigmoid (I checked the documentation and found that nn.Sigmoid is added internally in the loss function)? And two, is this loss function optimal for the setting? What I mean is does it lead to a trained network, which can perform as well as ||y-ax||^2 being the loss function? I apologize if my questions do not make sense.

@KFrank

The problem, of course, is the 2^m classes, as you lose information
about how the classes are “correlated,” and the potentially large
number of classes means you have fewer samples per class.

The relation y=ax+b can produce infinite number of training samples, theoretically. This is because the there are infinite values of a and b for a fixed distribution with fixed statistics.

Thank you so much Frank, for taking time to make me understand the working of the loss functions. I feel I am really close to the correct architecture for this problem. Appreciate it.