# How to achieve tf.scatter\_nd in pytorch? project for copynet **URL:** \ **Category:** autograd\ **Created:** [July 7, 2017, 7:27am UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688 "2017-07-07T07:27:43Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![jason\_wjbian](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/b19c9b/32.png) [@jason\_wjbian](https://discuss.pytorch.org/u/jason_wjbian)\ **Post date:** [July 7, 2017, 7:27am UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/1 "2017-07-07T07:27:44Z")
Hi, I want to implement a CopyNet with pytorch. But I encounter a problem now. In tensorflow, there is scatter\_nd operation > ``` > indices = tf.constant([0,3]) > updates = tf.constant([0.2,0.6]) > scatter = tf.scatter_nd(indices, updates, shape=4) > print scatter > [0.2 , 0 , 0 , 0.6] > > ``` as you can see, the index in indices fill the corresponding value in updates. But I don’t know how to achieve this. I try use torch.cuda.sparse.FloatTensor(indices,updates,size) replace, but it seems can’t backward the gradient. My whole code as follow, can train, but p\_gen become 1 gradually, so I doubt there is no gradient back for new\_atten\_prob > ``` > p_gen = p_gen.unsqueeze(-1) > normal_output_prob = p_gen.expand_as(output_prob) * output_prob > normal_atten_prob = (1 - p_gen).expand_as(atten_prob) * atten_prob > new_atten_list = [] > for input_token, atten_prob_token in zip(input.split(1), normal_atten_prob.split(1)): > atten_prob_token = atten_prob_token.squeeze(0) > new_atten_list.append(torch.cuda.sparse.FloatTensor( > input_token.data, atten_prob_token.data, torch.Size([normal_output_prob.size(1)+30])).to_dense()) > new_atten_prob = torch.stack(new_atten_list, dim=0) > new_normal_prob = torch.cat([normal_output_prob,Variable(torch.zeros(normal_output_prob.size(0),30).cuda())],-1) > prob = (Variable(new_atten_prob,requires_grad=True) + new_normal_prob) > return prob > > ``` ---
**Author:** ![henrye](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c37758/32.png) [@henrye](https://discuss.pytorch.org/u/henrye)\ **Post date:** [July 12, 2017, 10:35am UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/2 "2017-07-12T10:35:41Z")
There is a scatter function in pytorch [http://pytorch.org/docs/master/tensors.html#torch.Tensor.scatter\_](http://pytorch.org/docs/master/tensors.html#torch.Tensor.scatter_) I’m not 100% sure about how the autograd mechanics work during assignment yet. But from my understanding when you create a new tensor using .data it loses the history of the variable. Hence why no backprop would occur. ---
**Author:** ![alexis-jacq](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexis-jacq/32/180_2.png) [@alexis-jacq](https://discuss.pytorch.org/u/alexis-jacq)\ **Post date:** [July 12, 2017, 11:48am UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/3 "2017-07-12T11:48:30Z")
As far as I know, there is no autograd way to do that. If you want to backward through this operation, you must [extend autograd.Function](http://pytorch.org/docs/master/notes/extending.html#extending-torch-autograd) and write the backward method by hand. ---
**Author:** ![henrye](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c37758/32.png) [@henrye](https://discuss.pytorch.org/u/henrye)\ **Post date:** [July 12, 2017, 12:15pm UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/4 "2017-07-12T12:15:54Z")
> As far as I know, there is no autograd way to do that. There’s no autograd through scatter? ---
**Author:** ![alexis-jacq](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexis-jacq/32/180_2.png) [@alexis-jacq](https://discuss.pytorch.org/u/alexis-jacq)\ **Post date:** [July 12, 2017, 12:22pm UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/5 "2017-07-12T12:22:48Z")
There’s only `torch.Tensor.scatter_()`, that is a method of `torch.Tensor`. But no autograd function with backward method. So you can’t backward through it… ---
**Author:** ![henrye](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c37758/32.png) [@henrye](https://discuss.pytorch.org/u/henrye)\ **Post date:** [July 12, 2017, 1:07pm UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/6 "2017-07-12T13:07:12Z")
Damn, thanks for letting me know. I think I’ll be posting my own question about this after all. ---
**Author:** ![alexis-jacq](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexis-jacq/32/180_2.png) [@alexis-jacq](https://discuss.pytorch.org/u/alexis-jacq)\ **Post date:** [July 12, 2017, 1:21pm UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/7 "2017-07-12T13:21:56Z")
I already asked it there: [Edit a subtensor of a 3D tensor](https://discuss.pytorch.org/t/edit-a-subtensor-of-a-3d-tensor/4348/4) But I think I will do this function by hand and create a pull request. It should not be difficult after all. ---
**Author:** ![henrye](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c37758/32.png) [@henrye](https://discuss.pytorch.org/u/henrye)\ **Post date:** [July 12, 2017, 1:55pm UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/8 "2017-07-12T13:55:33Z")
I have the same sort of problem. I’m implementing a copy mechanism and need to get all the probabilities into a larger matrix. ``` attention_scores = Variable(torch.zeros(target_seq_len, batch_size, max_dict_index)) for i in range(batch_size): for j in range(target_seq_len): for k in range(input_seq_len): attention_scores[j, i, input[k, i].data[0]] += attentions[j, i, k] ``` But this naturally doesn’t work because Variables don’t support in-place assignment. Is there another way to construct a complex matrix, as would be done with scatter, and still retain the autograd history? ---
**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\ **Post date:** [July 12, 2017, 1:59pm UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/9 "2017-07-12T13:59:37Z")
@alexis-jacq `Variable` does implement the scatter function, implementation is [here](https://github.com/pytorch/pytorch/blob/10e23943b348e0f3b8f7709beadf3bc1541c763c/torch/autograd/variable.py#L665-L666) with the `Function` definition is [here](https://github.com/pytorch/pytorch/blob/10e23943b348e0f3b8f7709beadf3bc1541c763c/torch/autograd/_functions/tensor.py#L565-L587). ---
**Author:** ![henrye](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c37758/32.png) [@henrye](https://discuss.pytorch.org/u/henrye)\ **Post date:** [July 12, 2017, 2:07pm UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/10 "2017-07-12T14:07:20Z")
Oh, and I see below there’s a ScatterAdd function which would solve my issue of having repeated indices in the input sequence! > [@How to implement scatter add?](https://discuss.pytorch.org/t/how-to-implement-scatter-add/3876): > > Hi, I need the function that does something like torch.Tensor.scatter\_(dim, index, src). x = torch.Tensor([[9,10,11,12]]) id = torch.Tensor([[4,3,4,7]]) out = torch.zeros(1, 8).scatter\_(1, id, x) When I run the code above, I got the output [0, 0, 0, 10, 11, 0, 0, 12]. While I expect the output to be [0, 0, 0, 10, 20, 0, 0, 12]. Note that the value in index 4 is 20 (9+11), not 11. Does anyone know how to implement scatter that add the source value to the tensor, not just cover it? ---
**Author:** ![alexis-jacq](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexis-jacq/32/180_2.png) [@alexis-jacq](https://discuss.pytorch.org/u/alexis-jacq)\ **Post date:** [July 12, 2017, 2:24pm UTC](https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688/11 "2017-07-12T14:24:01Z")
Wow great! I was looking for it in the docs, but I should have first looked in github sources. Thanks @albanD!