How can I stack tensors from a mapping list?

1

Hi,

I have different different u, i c variables of shape [99] which contain:

  `In [8]: u
  Out[8]:
  tensor([820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820,
          820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820,
          820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820,
          820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820,
          820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820,
          820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820,
          820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820, 820,
          820], device='cuda:0')
  
  In [9]: i
  Out[9]:
  tensor([1280, 2561, 2562, 1029, 1019, 2058, 1550, 1807, 2319, 2067, 2324, 1304,
          2331, 1052, 1565, 1308, 2591, 1567, 1314, 1573, 1574, 1066, 2090, 1324,
          1329, 2610, 2356, 1589, 1085, 1600, 1856, 1603, 1099, 1101, 1873, 1362,
          1107, 1624, 1113, 1114, 2396, 1888, 1633, 1121, 1641, 1643, 1645, 1392,
          1906, 2164, 1910, 1403, 1405, 1920, 1921, 1410, 1160, 2187, 2448, 1173,
          2456, 2459, 1183, 1184, 1187, 2467, 1192, 1450, 1963, 2481, 2226, 2232,
          1722, 2240, 1472, 1984, 1220, 2500, 1483, 1228, 1741,  977, 2260, 2005,
          1750, 2522, 2523, 1244, 1505, 2535, 1516, 1773, 1007, 1777, 2289, 1522,
          1013, 1016, 1275], device='cuda:0')
  
  In [12]: c
  Out[12]:
  tensor([2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694,
          2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694,
          2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694,
          2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694,
          2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694,
          2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694,
          2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694,
          2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694, 2694,
          2694, 2694, 2694], device='cuda:0')`

In order to generate their embeddings I pass to the model the unique idx of all of them so that it computes the corresponding embeddings, following the example, I pass the variable (whose len(unique_uic) == 101) :

`In [32]: unique_uic  
Out[32]:
tensor([ 820,  946,  959, 1009, 1010, 1013, 1044, 1109, 1147, 1149, 1152, 1182,
        1184, 1208, 1222, 1267, 1308, 1334, 1409, 1417, 1421, 1431, 1455, 1457,
        1462, 1470, 1475, 1565, 1578, 1586, 1628, 1632, 1675, 1686, 1690, 1698,
        1723, 1733, 1737, 1745, 1775, 1802, 1817, 1835, 1854, 1860, 1864, 1868,
        1879, 1892, 1894, 1896, 1915, 1918, 1930, 1947, 1951, 1968, 1978, 1984,
        2037, 2042, 2045, 2070, 2075, 2086, 2106, 2125, 2186, 2199, 2205, 2290,
        2302, 2303, 2332, 2338, 2347, 2363, 2368, 2381, 2382, 2417, 2440, 2441,
        2449, 2464, 2466, 2471, 2500, 2512, 2519, 2548, 2549, 2557, 2568, 2578,
        2580, 2582, 2617, 2618, 2694])
`

Then, when I compute the embeddings I get embeddings variable of shape torch.Size([101, 64]) which refers to the embeddings on the variable unique_uic respectively.

However, in order to feed them into another model, I need them to be in the following shape:

`In [34]:self.embeddings(torch.stack((u, i, c), dim=1))
 Out[34]:
tensor([[ 820, 2568, 2694],
        [ 820, 1802, 2694],
        [ 820, 2578, 2694],
        [ 820, 2580, 2694],
        [ 820, 1044, 2694],
        [ 820, 2070, 2694],
        [ 820, 2582, 2694],
        [ 820, 1817, 2694],
        [ 820, 2075, 2694],  
                  .........      
        [820, 2042, 2694],
        [ 820, 2045, 2694],
        [ 820, 2302, 2694],
        [ 820, 2303, 2694]]) 
`

So, I should have the embeddings in shape [99, 3, 64] as it refers to the 99 interactions (in this example) of u-i-c. However, when I perform the embeddings I get embeddings from 0 to 101 corresponding to the unique id’s and I do not know how to perform the mapping in an efficient way.

If you could give me ideas it would be very helpful.

Thank you very much in advance!

2

I’m not sure I understand the use case completely, so feel free to correct me :wink:

Based on your description you are dealing with 3 different tensors, u, i, and c.
unique_uic contains the 101 unique values of all three tensors and is used in some way to create dense embedding tensors of the shape [101, 64], i.e. 101 tensors (one for each input index) with a feature dimension of 64.
Now you would like to concatenate the three input tensors and feed to this embedding?

If that’s the case, I assume you would like to treat each input combination as an “input index”, which would need a mapping. However, these three tensors could create more combinations than the number of unique values, so how would you deal with it?
u and c only contain a static value, but is this always the case in your workflow?

3

Hi, thanks a lot for answering!

So, the first part of your understanding is correct, u, i, and c are different embeddings and unique_uic contains the 101 unique values of all three tensors, which are used to create dense embedding tensors of the shape [101, 64].

Lets say that u always go from range 0-900 , i go from 901 - 2579, and c 2580-2697. In each iteration it is fed to the network a batch of [99,3], which are triplets of indexes which have interacted. For example:

        [ 820, 1802, 2694], --> user 820 have interacted with item 1802 under context 2694
        [ 820, 2578, 2694],

If I just get the embeddings of those 101 “nodes” in shape [101,64], what I would like to know is how to come back to the original triplet shapes but instead of stacking indexes, stacking their corresponding embeddings. Thus, just asking how to go from those [101, 64] embeddings to a shape [99, 3, 64] embeddings, taking into account that, in those 101 unique embeddings, there are 3 different nodes (for this example the first one is a user, the last one context and the 99 in the middle are items but it can be completely different – so the shape of unique_uic is completely variant).

Thank you so much again!

1 Like
4

To just verify: the [101, 64] shape is defined as [number_of_unique_triplets, embedding_dim]?

That’s a bit tricky, since your first approach would map the triplets to 101 unique indices in the range [0, 100]. However, now you would like to feed each input separately to the embedding layer, which would then need 2698 indices, since all inputs together contain valid values in [0, 2697].

Could you explain the idea a bit more? Note that an embedding layer is basically a lookup table for an integer input. The idea is that “sparse” inputs will be mapped to “dense” outputs.
However, you are currently trying to mix the triplets with the raw indices, which won’t work out of the box, so some mapping might be needed.

1 Like