Extracting the features from the layer before the softmax for BERT

nlp
1

How can I input a sentence and extract the feature vector from the layer before the last layer from BERT? I think it should be length 768.

2

what do you mean by

“How can I input a sentence and extract the feature vector from the layer before the last layer from BERT?”

Do you want someone to give you exact code to do that?

3

I see this https://github.com/huggingface/pytorch-pretrained-BERT
but I am not sure how I can extract features with it.

For example, I can give an image to resnet50 and extract the vector of length 2048 from the layer before softmax.

I am not sure how to do this for pretrained BERT.
Also, I am not sure if the link is what I am supposed to use.

Yes, some ready code maybe online?

4

You can look at what the BertForSequenceClassification model does in it’s forward.
The pooled_output obtained from self.bert would seem to be the features you are looking for.

Best regards

Thomas

3 Likes
5

I also ran the following code https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/extract_features.py

though it does not seem very straightforward to interpret the output:

$ python extract_features.py --input_file test_bert.txt --output_file out_bert.txt --bert_model bert-base-uncased
$ cat test_bert.txt 
My name is mona jalal and I live in boston
6

As a general expectation, I think this might be a lot to expect from a fellow community member (who is spending their time doing work for you). There should be some effort from your side to understand the code / repository as well. That particular repository is in fact one of the best documented and coded one.

3 Likes