Suppose I have a list of encoded tokens and an encoded string of words
tokens = [[5], [243,290], [0,2,0,4], ...]
prompt = [243, 5, 290, 5, 0, 2, 243, 290, 0, 2, 0, 4, ...]
I want to create a 0-1 mask where the 1-s correspond to where the tokens match exactly with the prompt. In the example the mask would look something like
mask = [0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, ...]
Notice that the 243 and 290 at the beginning of the prompt are 0 because the two numbers are not next to each other, and the first 0, 2 are not marked as 1 because they are only part of the 0,2,0,4 token.
I am thinking that the way to implement this would be to loop over each of the tokens, and perform a sliding window technique for each one, but I think this can be better optimised, and I am not sure how. Could I please have some suggestions for this?
Thanks in advance!