How to split the Variables along Batch Dimension to several Variables?

Hi all,

I have got a batch of features (N features). Now I want to pass message among different features.

I found that if I manipulate the features along batch dim, then errors will be raised when backward(). The codes are like below:

import torch
from torch.autograd import Variable
import numpy as np
import torch.nn as nn

a = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
b = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)

for i in range(b.size()[0]):
    for j in range(a.size()[0]):
        b[i] = b[i] + a[j]
        
b.backward(torch.Tensor(5, 3).normal_())

When running, I got ā€œinconsistent tensor size at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485444530918/work/torch/lib/TH/generic/THTensorCopy.c:40ā€.

Is it possible to divid the original Variable into several parts and then process them?

Best,
Yikang

Isnā€™t it the same as if you just summed up a along that dimension and then added it to each element of b?

import torch
from torch.autograd import Variable
import numpy as np
import torch.nn as nn

a = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
b = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)

a_summed = a.sum(0) # sum along first dimension
result = b + a_summed.expand_as(b)
result.backward(torch.randn(5, 3))
1 Like

Yes, thatā€™s the same. However, the ā€˜backward()ā€™ will lead to an error? Could you tell me I to do that?

No, my solution will work. Yours will raise an error, because youā€™re assigning things to b. Assignments are in-place operations, and they will fail if performed on leaf Variables.

2 Likes

Thank you for your timely reply.

I apologize for misleading you with the example.

The code is just an example, What I want to do is passing message along batch dimension. Actually, the link is dynamic during processing.

So what should I do if I want to manipulate the feature along batch dim.

In addition, whatā€™s the meaning of leaf Variables?

Best,
Yikang

Iā€™m sorry but I donā€™t know what passing a message along a dimension is. Iā€™d recommend you to look for out-of-place functions that can do this, but I canā€™t recommend any, as I donā€™t know what are you trying to achieve.

Leaf Variables are Variables that are not results of any computation, so:

x = Variable(torch.randn(5, 5)) # this is a leaf
y = x + 2 # y is not a leaf
1 Like

I have got that! The problem has fixed. Just add an operation will be enough.

import torch
from torch.autograd import Variable
import numpy as np
import torch.nn as nn


x = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
y= Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)

a = x + 0
b = y + 0

for i in range(b.size()[0]):
    for j in range(a.size()[0]):
        b[i] = b[i] + a[j]
        
b.backward(torch.Tensor(3, 5).normal_())

What I want to know is whether pytorchā€™s autograd supports doing operation on part of the variables. Thatā€™s what I need in my future project. Actually, it does support.

Thank you very much for your help and patience.

Best,
Yikang

In most cases (like in your example), you can avoid these assignments, and Iā€™d recommend doing that. It will likely result in better performance. Nevertheless, we do support them, so if thereā€™s no other way, you can still do that.

1 Like

Got it. Thank you very much. I really appreciate your works for PyTorch and the CV community!

Best,
Yikang

1 Like

I have been using PyTorch for a couple of weeks now, working mostly with complex RNNs models. In such situations, the natural way of accumulating state for me till now has been to create a Tensor and keep filling it with new state at each time step. But I havenā€™t been able to write code like that in PyTorch since in-place ops on leaf nodes are not supported. I have had to use lists with manual concatenation and indexing is messy.

I can use the ugly hack shown above where in a dummy operation is performed to make a node non-leaf. But it would be great to have some implicit adjustment that allows for in-place ops on leaves.

If you think this would be a useful feature to have (or conversely if its not something you are intentionally against), and if you can give some pointers as to how to make this happen, I can work on it and submit a PR .

I hope I am not missing something trivial here.

the only reason for us to not support inplace operations on leaves is for correctness.
We shall re-review how complex it would get to support some inplace operations on leaves.

Thanks a lot for the feedback.

You can fill in a leaf Variable. Just donā€™t declare it as requiring grad - you donā€™t need grad w.r.t. the original content that will get overwritten.

1 Like