Combined model result in Linear Layer followed by RNN for classification

vision
1

hi, i’m new in pytorch and trying to bind the dual model CNN structure with the RNN module for the classification purpose. The RNN code copied from pytorch tutorials. So, facing problem in binding the combined model result by linear layer with RNN block

one of the Error i Got

line 356, in forward
    combined = torch.cat((input, hidden), dim=1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 1 and 8 in dimension 0 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71

class MyRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MyRNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        input = input.view(input.size(0), -1)
        hidden = hidden.view(hidden.size(0), -1)
        combined = torch.cat((input, hidden), dim=1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

class Net(nn.Module):
    def __init__(self, modelA, modelB):
        super(Net, self).__init__()
        self.con = nn.Conv2d(3, 3, 3, 1, 1, 1)
        self.Lrelu = nn.ReLU()
        self.modelA = modelA
        self.modelB = modelB
        self.final_classifier1 = nn.Linear(512, 128)
        self.myRNN = MyRNN(128, 128, 10)


    def forward(self, x):
        x = self.con(x)
        x = self.Lrelu(x)
        x1 = self.modelA(x.clone())
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x.clone())
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        x = self.final_classifier1(F.relu(x))
        x = self.myRNN(x, hidden)
        return x

n_hidden = 128
hidden = torch.zeros(1, n_hidden)
hidden = hidden.to(device)

model = Net(modelA, modelB)

Also tried to initialize the hidden side in the training epoch

...
for epoch in range(1, n_epochs + 1):

    hidden = MyRNN.initHidden()
    model.train()
...
2

The shape mismatch seems to be raise in

combined = torch.cat((input, hidden), dim=1)

which points to a different batch size (dim0).

It seems you are initializing the hidden tensor as torch.zeros(1, self.hidden_size), which would use a fixed batch size of 1.

You could add a batch_size argument to initHidden() and use it to create the right hidden tensor in the forward method via:

x = self.final_classifier1(F.relu(x))
hidden = self.myRNN.initHidden(x.size(0))
x = self.myRNN(x, hidden)

Let me know, if that would work for you.

3

Thanks for identifying the issue. It has been solved by passing the batch size to the initHidden function

    def initHidden(self, batch_size):
        return torch.zeros(batch_size, self.hidden_size).to(device)
4

But i’m getting another issue

AttributeError: 'tuple' object has no attribute 'log_softmax'

and i guess its related to this post AttributeError: 'tuple' object has no attribute 'log_softmax'

5

Based on your code you are calling the nn.LogSoftmax activation on output, which is the output activation of self.i2o (nn.Linear), so it shouldn’t be a tuple.
Did you modify the code somehow and could post the complete stack trace including the layer, which raises this error?

6

Similar error even not using the nn.Logsoftmax activation on out beacuse i tried another nn.Linear layer and also with the direct output.
the piece of code worked for me is;

class MyRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MyRNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        #self.outp = nn.Linear(output_size, 7)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        input = input.view(input.size(0), -1)
        hidden = hidden.view(hidden.size(0), -1)
        combined = torch.cat((input, hidden), dim=1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        #output = self.outp(output)

        output = self.softmax(output)
        return output, hidden

    def initHidden(self, batch_size):
        return torch.zeros(batch_size, self.hidden_size).to(device)

...
...
    def forward(self, x):
       ....
       ....
        hidden = self.myRNN.initHidden(x.size(0))
        x = self.myRNN(x, hidden)
        return x


AttributeError: 'tuple' object has no attribute 'log_softmax'

this error was solved by changing the lines in training code but i wont able to figure out the exact location from where i got the tuple

for epoch in range(1, n_epochs + 1):
    ....
    model.train()
    ....
    output = Net(data)
    loss = criterion(output[0], target)
    ....
    ....
    out, pred = torch.max(F.softmax(output[0], 1),1)
    ....

taking the specific output[0] for the criterion solves the error may be beacuse the MyRNN block is returning 2 output values

Thanks

7

I guess you might have used F.softmax(output, 1) before changing it to F.softmax(output[0], 1)?
Since output is a tuple, this would explain the error. :wink: