Correct way copy batch of tensors to another single tensor

1

I tried to copy several tensors (a batch of tensors) to one single tensor, but I got the next error:
raise RuntimeError("Only Tensors created explicitly by the user "
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment

Any idea how to copy/transfer a batch of tensors to the single 4D tensor?

I have attached my code:

class GLr(nn.Module):

def __init__(self, shape):
    super(GL, self).__init__()

    assert len(shape) == 4, " Invalid argument: shape must be tuple of fourth"
    self.layers = nn.Parameter(data=torch.Tensor(shape), requires_grad=True)
    self.temp_holder = torch.rand(shape)

def forward(self, images):

    for i, image_ in enumerate(images):
        for j, image in enumerate(image_):
            self.temp_holder[i][j] = self.kernel_fn( .... )
    self.layers.data = copy.deepcopy(self.temp_holder)
    return self.layers
2

You could detach() the tensors to create a new leaf tensor, which would support the deepcopy again.
A few issues besides that:

  • Don’t use torch.Tensor as it would be using uninitialized memory. If that’s your use case, use torch.empy instead.
  • Don’t use the .data attribute as it could yield unwanted side effects. Instead wrap your code into a with torch.no_grad() block and copy_ the new data into the parameter, if needed.
3

Thanks for your answer!

Generally, I try to implement something very similar to the f.conv2d layer, i.e., f.conv2d gets a 4D tensor with several layers, iterate over them one by one, and calculates convolution for each layer.

So, my question is how to iterate over 4D tensor one by one?

As you can see in my code, I got a 4D tensor of layers (images) and iterate over the tensor with ‘double for,’ then I tried to copy results into a new 4D tensor (self.layers).

But I am pretty sure that is the wrong implementation and runs very slow. (The layers suppose to be with trainable parameters for gradients calculation.)

4

The nested loop would be slow, but should generally work as a first experiment.
In your code it seems you are copying the result to an nn.Parameter, which sounds wrong and you should create a new output tensor instead.
A faster approach would be to use unfold and to apply a matrix multiplication to calculate the convolution. The nn.Unfold docs give you an example.