Save Tensors into a list and use it for Multi-connection Network

#1

Hello Everyone,

I am new to Pytorch and I wanted create some novel architecture.

jump_arch
jump_arch.PNG719×268 10 KB

I wanted to create something like the second architecture so I try to save the tensors at every block inside a list like [i, c, p, fc, o] so that I can use the calculated tensors for a new connection. But first, I have a problem when I try to do .backward()

My forward function goes like this

def forward(self, inputs):
        tensor_list.append(inputs)
        for i in range(len(self.nnModList)):
            x = self.NNList[i] (tensor_list[i])
            tensor_list.append(x)

and loss.backward(retain_graph=True)

                or

def forward(self, inputs):
        tensor_list.append(inputs)
        for i in range(len(self.NNList)):
            x = self.NNList[i] (tensor_list[i])
            tensor_list.append(x.detach())

It works but does not seem to converge unlike doing the forward without the list. How can I make the code above work like the normal?

#2

Could you explain more about why you need to save the tensor in a list?
Does the following code do what you want?

def forward(self, inputs):
    x = self.conv(inputs)
    # combine features and output ('concat' is just an example) 
    x = torch.cat([x, inputs])  # supposing their dimensions match
    x = self.pool(x)
    x = self.fc(x)
    return x
#3

Hi @kaixin, I have the a class for concatenation. I wanted to save tensors into a list so that I can use the tensors again in another connection. For example,

input - hidden1 - hidden2 - hidden3 - out
plus hidden1 is connected to hidden3 also

x = []
x.append(operation(input, hidden1))    #list 0
x.append(operation(x[0], hidden2))    #list 1
x.append(operation(x[1], hidden3))    #list 2
x.append(operation(x[0], hidden3))    #list 3
x.append(combine(x[2], x[3]))           #list 4
x.append(operation(x[4], out))           #list 5

return x[5]