I have got a batch of features (N features). Now I want to pass message among different features.
I found that if I manipulate the features along batch dim, then errors will be raised when backward(). The codes are like below:
import torch
from torch.autograd import Variable
import numpy as np
import torch.nn as nn
a = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
b = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
for i in range(b.size()[0]):
for j in range(a.size()[0]):
b[i] = b[i] + a[j]
b.backward(torch.Tensor(5, 3).normal_())
When running, I got āinconsistent tensor size at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485444530918/work/torch/lib/TH/generic/THTensorCopy.c:40ā.
Is it possible to divid the original Variable into several parts and then process them?
Isnāt it the same as if you just summed up a along that dimension and then added it to each element of b?
import torch
from torch.autograd import Variable
import numpy as np
import torch.nn as nn
a = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
b = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
a_summed = a.sum(0) # sum along first dimension
result = b + a_summed.expand_as(b)
result.backward(torch.randn(5, 3))
No, my solution will work. Yours will raise an error, because youāre assigning things to b. Assignments are in-place operations, and they will fail if performed on leaf Variables.
Iām sorry but I donāt know what passing a message along a dimension is. Iād recommend you to look for out-of-place functions that can do this, but I canāt recommend any, as I donāt know what are you trying to achieve.
Leaf Variables are Variables that are not results of any computation, so:
x = Variable(torch.randn(5, 5)) # this is a leaf
y = x + 2 # y is not a leaf
I have got that! The problem has fixed. Just add an operation will be enough.
import torch
from torch.autograd import Variable
import numpy as np
import torch.nn as nn
x = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
y= Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
a = x + 0
b = y + 0
for i in range(b.size()[0]):
for j in range(a.size()[0]):
b[i] = b[i] + a[j]
b.backward(torch.Tensor(3, 5).normal_())
What I want to know is whether pytorchās autograd supports doing operation on part of the variables. Thatās what I need in my future project. Actually, it does support.
In most cases (like in your example), you can avoid these assignments, and Iād recommend doing that. It will likely result in better performance. Nevertheless, we do support them, so if thereās no other way, you can still do that.
I have been using PyTorch for a couple of weeks now, working mostly with complex RNNs models. In such situations, the natural way of accumulating state for me till now has been to create a Tensor and keep filling it with new state at each time step. But I havenāt been able to write code like that in PyTorch since in-place ops on leaf nodes are not supported. I have had to use lists with manual concatenation and indexing is messy.
I can use the ugly hack shown above where in a dummy operation is performed to make a node non-leaf. But it would be great to have some implicit adjustment that allows for in-place ops on leaves.
If you think this would be a useful feature to have (or conversely if its not something you are intentionally against), and if you can give some pointers as to how to make this happen, I can work on it and submit a PR .
the only reason for us to not support inplace operations on leaves is for correctness.
We shall re-review how complex it would get to support some inplace operations on leaves.
You can fill in a leaf Variable. Just donāt declare it as requiring grad - you donāt need grad w.r.t. the original content that will get overwritten.