I would like to add manualy an output layer to BERT in order to predict multiple features which are binary.
For example, these outputs would answer the questions:
- Is the text positive? 1 if yes, 0 otherwise.
- Is this text about sports? 1 if yes, 0 if not.
- Is this text about business? 1 if yes, 0 if not.
My first idea was to do it as a regression task, adding an output layer with 3 neurons: one for each question.
def __init__(self): self.bert = CamembertModel.from_pretrained('camembert-base') self.regressor = nn.Sequential(nn.Linear(dim_in, 3)) def forward(self, input): outputs = self.bert(input) # "Bert Layers" outputs = self.regressor(outputs) # LinearSequential with output of 3 return outputs
But I’ll get values above 1 and below 0. So, which kind of layer could I add in order to get probability (values between 0 and 1) to deal with this problem ?
I hope my question was clear, thank you for your help.