I have a binary input to my model that I sample from some Bernoulli distribution:
>>> batch_size = 3
>>> weights = torch.tensor([0.1111, 0.2669, 0.5122, 0.2045, 0.0224]).repeat((batch_size, 1))
>>> batch = torch.bernoulli(weights)
>>> batch
tensor([[1., 0., 0., 1., 1.],
[0., 1., 0., 0., 0.],
[0., 1., 1., 0., 0.]])
I want to get a padded sequence of embeddings for every sample in my batch so I can run it through a transformer encoder. For that, I first need to turn batch
into a tensor of indices, then pad it, and finally run it through an embedding layer.
The closest I’ve come is to use tensor.nonzero()
but I don’t know how to pad it and reshape it back to batch_size, max_seq_len
.
Desired output:
>>> func(batch, padding_value=-1)
tensor([[ 0., 3., 4.],
[ 1., -1., -1.],
[ 1., 2., -1.]])