Index adding over multiple dimensions

1

Is there a way to use index_add with the index argument being more that 1-dimensional ?
More especially, if I have a 2d-array that I want to fill not row by row or column by column, but element by element by specifying the 2d coordinates in which to add the desired amount in the 2d-array.

Example:

>>> to_be_filled = torch.zeros((2, 7))
>>> to_be_filled
torch.tensor([
    [0, 0, 0, 0, 0, 0, 0], 
    [0, 0, 0, 0, 0, 0, 0]
])
>>> index = torch.tensor([
    [0, 1], # adding 0.4
    [0, 2], # adding -1.4
    [0, 3], # adding -1.13
    [0, 6], # adding -1
    [1, 2], # adding 3
    [0, 1]  # adding 2
])
>>> values = torch.tensor([
    0.4,    # at coordinates (0, 1)
    -1.4,   # at coordinates (0, 2)
    -1.13,  # at coordinates (0, 3)
    -1,     # at coordinates (0, 6)
    3,      # at coordinates (1, 2)
    2       # at coordinates (0, 1)
])

I would like to do this like the following, but I have an error:

>>> to_be_filled.index_add_(0, index, values)
IndexError: index_add_(): Index is supposed to be a vector

Expected result:

torch.tensor([
    [0, 2.4, -1.4, -1.13, 0, 0, -1], 
    [0, 0, 3, 0, 0, 0, 0]
])

Is there a way to do this using pytorch operations?

Note: doing to_be_filled[index[:, 0], index[:, 1]] += values yields the following result:

torch.tensor([
    [0, 2, -1.4, -1.13, 0, 0, -1], 
    [0, 0, 3, 0, 0, 0, 0]
])

This approach does not accumulate when an index appear twice or more (notice that to_be_filled[0, 1] == 2 instead of 2.4)

2

but why would you not directly specify 2.4 to add, instead of 0.4, and then 2, that is accumulate the same coordinates values in the values tensor itself

another way to do this is through,

values = torch.tensor([
    [0.4],    # at coordinates (0, 1)
    [-1.4],   # at coordinates (0, 2)
    [-1.13],  # at coordinates (0, 3)
    [-1],     # at coordinates (0, 6)
    [3],      # at coordinates (1, 2)
    [2]       # at coordinates (0, 1)
])
to_be_filled[index.split(1, 1)] += values

but it would also consider the last addition for each index, and give 2, not 2.4

3

Firstly, thanks for the answer!

As of why I would not directly specify the sum for each index, is because I have no choice: my data have this shape, with some indexes appearing twice or more, and I have to work with it. I don’t decide of the shape of it, it comes like that.

Grouping by index and summing as you say would also answer my needs (like a groupby operation on a pandas.DataFrame and then aggregate with a sum, and then doing to_be_filled[index[:, 0], index[:, 1]] += values or to_be_filled[index.split(1, 1)] += values as you said), but I am convinced that there is a way to do it the way I said. I just find to find how.

4

Hi Lapertor!

If by “the way I said” means to perform your computation with a single
pytorch tensor operation in the presence of duplicate indices, then I
don’t think it’s possible.

It’s difficult to prove a negative, but I think any kind of pytorch "advanced
indexing* or index_add-style approach will give the “wrong” answer
(and potentially become non-deterministic) when confronted with
duplicate indices.

My assumption is that pytorch, for efficiency reasons, does the indexed
operation, figuratively speaking, “in parallel,” breaking the semantics
you would expect if you had coded the operation with sequential loops.

In my experience, as with Vainaijr’s observation, “the last addition for
each index, and give 2, not 2.4,” the behavior is as if the result for just
one of a set duplicate indices overwrites the others. Furthermore, I
believe that pytorch makes no guarantee as to which index “wins” so
that your result may change if you move your computation to the gpu
or change the size of your tensors, etc.

I would be very interested to hear if you do find a way to perform such
indexed operations in the presence of duplicate indices. (It would be
great if an expert would weigh in on this.)

Best.

K. Frank

5

Thanks for the answer K. Frank!

This is indeed what I mean

It really feels like acting as you describe here, and the more I think about this, the more I feel like there isn’t any clean way of doing this with tensor operations.

I used a workaround specific to my application in order to avoid this issue for now, but I still hope that an expert sees that topic and gives insights as I think that I am not the first to encounter an issue similar to this one.

6

Hello,

I had a similar issue today and here’s my solution (FYI).
(credit : numpy - Add a index selected tensor to another tensor with overlapping indices in pytorch - Stack Overflow)

to_be_filled = torch.zeros((2, 7))
index = torch.tensor([
    [0, 1], # adding 0.4
    [0, 2], # adding -1.4
    [0, 3], # adding -1.13
    [0, 6], # adding -1
    [1, 2], # adding 3
    [0, 1]  # adding 2
])
values = torch.tensor([
    0.4,    # at coordinates (0, 1)
    -1.4,   # at coordinates (0, 2)
    -1.13,  # at coordinates (0, 3)
    -1,     # at coordinates (0, 6)
    3,      # at coordinates (1, 2)
    2       # at coordinates (0, 1)
])

filled = torch.index_put(to_be_filled, (index[:, 0], index[:, 1]), values, accumulate=True)

The result of ‘filled’ is

tensor([[ 0.0000,  2.4000, -1.4000, -1.1300,  0.0000,  0.0000, -1.0000],
        [ 0.0000,  0.0000,  3.0000,  0.0000,  0.0000,  0.0000,  0.0000]])

Best,

1 Like