How can I input a sentence and extract the feature vector from the layer before the last layer from BERT? I think it should be length 768.
what do you mean by
“How can I input a sentence and extract the feature vector from the layer before the last layer from BERT?”
Do you want someone to give you exact code to do that?
I see this https://github.com/huggingface/pytorch-pretrained-BERT
but I am not sure how I can extract features with it.
For example, I can give an image to resnet50 and extract the vector of length 2048 from the layer before softmax.
I am not sure how to do this for pretrained BERT.
Also, I am not sure if the link is what I am supposed to use.
Yes, some ready code maybe online?
You can look at what the
BertForSequenceClassification model does in it’s forward.
pooled_output obtained from
self.bert would seem to be the features you are looking for.
I also ran the following code https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/extract_features.py
though it does not seem very straightforward to interpret the output:
$ python extract_features.py --input_file test_bert.txt --output_file out_bert.txt --bert_model bert-base-uncased $ cat test_bert.txt My name is mona jalal and I live in boston
As a general expectation, I think this might be a lot to expect from a fellow community member (who is spending their time doing work for you). There should be some effort from your side to understand the code / repository as well. That particular repository is in fact one of the best documented and coded one.