Hello friends. I want to have a representation for each sentence as sentence embedding vector + sentence position embedding vector.
I have no problem with sentence embedding vector, but I have trouble understanding sentence position embedding.
I read in an article that first set a position value for each sentence that was 0.1, or 2.
A value of zero for the first 3 sentences, a value of one for the last 3 sentences and a value of two for the other middle sentences of the document.
The authors’ idea was that they want the beginning and ending sentences of the document to be of greater importance than middle ones.
And they said the rest is like a normal embedding procedure …
Now I do not understand, for example, if I have 10 sentences in my document and the length of the embedding vectors is 4, then the sentence embedding matrix will be (10,4).
But for the position embedding matrix, will the matrix size be (10,4)? Or (3,4)? (3 for the three values 0, 1, or 2 assigned to the position).
I do not understand, where should I use those three values of position 0,1 or 2 in the embedding layer?
Thanks in advance.