'tuple' argument has no attribute 'repeat'

1

Hi!

I am writing a neural network in pytorch and I am running into some issues. I am implementing an attention layer, however I get a runtime error when I want to repeat my hidden layer. My model is as follows:

class Model_P(nn.Module):
def init(self, hidden_size, input_size):
super(Model_P, self).init()
self.hidden_size = hidden_size
self.input_size = input_size
self.lstm = nn.LSTM(input_size, hidden_size, 1, bias=False, batch_first=True, bidirectional=True)
self.dropout = nn.Dropout(0.5)
self.softsign_1 = nn.Softsign()
self.attention_hidden_vector = nn.Linear(hidden_size + 2 * hidden_size, hidden_size)
self.attention_scoring_fn = nn.Linear(hidden_size, 1, bias=False)
self.dense = nn.Linear(hidden_size,1)
self.relu_1 = nn.ReLU()

def forward(self, x):
    #model en data parameters must be the same, thus convert to float
    x.float()
    output, hidden = self.lstm(x.float())
    output = self.softsign_1(output.float())
    src_len = output.shape[0]
    hidden = hidden.repeat(src_len, 1, 1)
    attn_hidden = torch.tanh(self.attn_hidden_vector(torch.cat((hidden, output), dim=2)))
    attn_scoring_vector = self.attn_scoring_fn(attn_hidden).squeeze(2)
    attn_scoring_vector = attn_scoring_vector.permute(1, 0)
    output = F.softmax(attn_scoring_vector, dim=1)
    output = self.dense(output)
    output = self.relu_1(output)

    return output

I get the following error: AttributeError: ‘tuple’ object has no attribute ‘repeat’

I have already tried to convert the tuple ‘hidden’ to a tensor, by converting the tuple to an numpy array, and then creating a tensor, via the following code:

hidden = np.asarray(hidden)
hidden = torch.from_numpy(hidden)

However this raises other erros, which I have not succeeded to solve. Does anybody know of a solution? Many thanks in advance!

2

So what is happening is your hidden output from the lstm is a tuple made up of both the hidden state and the cell state. The lstm has three outputs, the output of the model, the hidden state, and the cell state. When you put only two variables one becomes a tuple that includes the hidden and cell state. If you only want to repeat the hidden state just add another variable for the cell state like this:

    output, hidden, _ = self.lstm(x.float())