Different behavior of advanced indexing on CPU and GPU

1

I noticed that advanced indexing works differently on CPU and GPU with PyTorch 0.4, when multi-index is large by length and contains repeated elements.
Minimal working example:

  1. This code works on CPU and saves the last element encountered in a repeated indexed cell.
len_idx = 150
for i in range(10):
    A = torch.zeros(5, 5, 5)
    idx_along_axis = torch.ones(len_idx).long()
    A[idx_along_axis, idx_along_axis, idx_along_axis] = torch.arange(len_idx)
    print('A[1, 1, 1]:', A[1, 1, 1].item())

produces:

A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
  1. The same code on GPU saves a random element from all values corresponding to a repeated indexed cell.
len_idx = 150
for i in range(10):
    A = torch.zeros(5, 5, 5).cuda()
    idx_along_axis = torch.ones(len_idx).long().cuda()
    A[idx_along_axis, idx_along_axis, idx_along_axis] = torch.arange(len_idx).cuda()
    print('A[1, 1, 1]:', A[1, 1, 1].item())

produces different output each time, e.g. this:

A[1, 1, 1]: 149.0
A[1, 1, 1]: 31.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 31.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 149.0
A[1, 1, 1]: 31.0
A[1, 1, 1]: 31.0
A[1, 1, 1]: 31.0
A[1, 1, 1]: 31.0

This only works for large index arrays, for example, the problem cannot be reproduced for len_idx=30 on my machine. If we take larger len_idx, the set of saved values becomes even more random. Is it a bug in the framework?
Thank you.

2

Hi,

The problem is that you assign the exact same entry with different values.
This is undefined behavior: in a single line you say a=1 and a=2. What would you expect the value of a to be after this line?
It’s hard to define a default behavior for that (without large performance cost because conflicting indices would need to be detected).

On cpu it appears that it gets assigned the value which is last in the indexing, on gpu you sometimes get another value.

1 Like
3

Thank you, it seems clear to me now.