Generate text from tensor

nlp
1

Hi. Is there any proper approach to generate text from a given tensor/embedding? Let’s say I have a tensor and I want to generate a sequence of N words from it.

Thanks.

2

Can you explain more details?
What’s the relation between the tensor and the text. Is there a LUT or neural network for the mapping?

3

I have a dataset of paris of word embeddings and that word definition in natural language. During training I use a NN to map word embeddings to definition embeddings. My question is, after that, how could I generate the sequence of words from the definition given those word embeddings. A trasnformer decoder would be enough?

Thanks.

4

Correct me if I’m wrong. You want to generate sequences with a trained model. Is your model an auto-regressive one?
Usually, we use language model to generate that. A dummy code would be like

model = ...
# tokenizer, a.k.a., the vocabulary mapping text to integer
tokenizer = ...
# start flag of sequence
bos = '<s>'
seq = [bos]
# convert '<s>' to integer, usually 0
last_token_id = tokenizer.encode(bos)
hidden_state = None
for i in range(max_len):
    predict_token_id, hidden_state = model(last_token_id, hidden_state)
    last_token_id = predict_token_id
    seq.append(tokenizer.decode(predict_token_id.item()))
seq = ' '.join(seq)

Here is a GPT-2 example. gpt2 · Hugging Face

1 Like
5

Thank you much.

Sorry but: here “hidden_state” is the embedding tesnor? Else, I think my doubt is missing (how to generate from a given tensor).

6

No, hidden_states stores the historical states of model predictions. For example

if given ‘h e l l’ , the model would likely to predict the next one is ‘o’;
if given ‘h e a v e’, it would likely to predict the next one is ‘n’.

For your case, if there isn’t any historical information, you can set hidden_state as None