Hi Shakeel!
I don’t know that much about adversarial networks, and I haven’t
looked at the paper to which you linked, but let me outline some
details that might be relevant to what you are trying to do:
Your network comes in three pieces, the front end that produces your
“shared feature representation,” let’s call it FeatureNet, the one half
of your back end that predicts emotions, call it EmotionNet, and the
other half that predicts the speakers, `SpeakerNet’.
To be clear, you don’t want to train SpeakerNet to do poorly
predicting the speakers. (That would be trivial.)
The core goal of your adversarial network is to train FeatureNet
to generate features that that can be used to successfully predict
emotions, but that are not useful for predicting the speakers.
The key idea is that we actively train SpeakerNet to do the best
job it can predicting the speakers using the output of FeatureNet,
but train FeatureNet so that SpeaketNet does poorly (while
EmotionNet does well).
We introduce two loss functions, emotionLoss and speakerLoss, and
the combined loss, loss = emotionLoss + lambda * speakerLoss.
(In your case the two losses are both CrossEntropyLoss.)
We backpropagate / optimize loss normally through both EmotionNet
and SpeakerNet, so that we train the both to make their predictions as
successfully as they can, given the features produced by FeatureNet.
(Notice that emotionLoss doesn’t depend on SpeakerNet, so
backpropagating the gradient of emotionLoss through SpeakerNet
doesn’t affect how SpeakerNet’s weights are updated. And vice
versa.)
But we keep track of the emotionLoss and speakerLoss gradients
separately so that before we backpropagate through FeatureNet
we can flip the sign of the speakerLoss gradient. (This would be
your “Gradient Reversal.”)
You are now training FeatureNet to produces features that let
EmotionNet be trained to do well predicting emotions, but also
prevent SpeakerNet from doing well predicting the speakers,
even though the SpeakerNet portion of your whole network is
being trained to do as well as it can, given the features it gets
from FeatureNet.
(I’m not aware of anything built into pytorch that will do “Gradient
Reversal” for you, and I’m not really sure how best to implement
something like this. In any event, at the point where the gradients
are being backpropagated from EmotionNet and SpeakerNet back
through FeatureNet, you somehow have to get your hands on the
speakerLoss / SpeakerNet gradient and flip its sign, before passing
it on back through FeatureNet.)
Good luck.
K. Frank