Hi all,
I want to transfer a sequence representation as the node representation of size (batch_size, num_nodes, hidden_size) for a graph model. I use scatter_ to assign the sequence embedding of correct indexes to the node representation. And the node_embedding is finally used for graph-level prediction. However, the model seems not work as the accuracy never updates in training.
Here is the code
import torch
batch_size = 2
num_nodes = 9
embedding_dim = 4
hidden_size = 4
id_embedding = torch.nn.Embedding(num_embeddings=num_nodes + 1, embedding_dim=embedding_dim, padding_idx=0) # +1 for padding index 0
# input batch sequence
batch_sequence = torch.LongTensor([
[5, 6, 7, 8, 9], # 5
[1, 2, 3, 0, 0] # 3
])
lengths = torch.LongTensor([5, 3])
seq_model = torch.nn.GRU(embedding_dim, hidden_size, batch_first=True)
x = id_embedding(batch_sequence)
pack = torch.nn.utils.rnn.pack_padded_sequence(x, lengths, batch_first=True)
out, hn = seq_model(pack)
seq_embedding, _ = torch.nn.utils.rnn.pad_packed_sequence(out, batch_first=True)
node_embedding = torch.zeros(batch_size, num_nodes + 1, hidden_size)
node_embedding.scatter_(1, batch_sequence.unsqueeze(-1).expand(-1, -1, seq_embedding.size(-1)), seq_embedding)
the node_embedding will be like
tensor([[[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.2699, 0.3514, 0.2468, 0.0029],
[ 0.4198, 0.2088, 0.2488, 0.1672],
[-0.1704, -0.0910, 0.4044, 0.2269],
[ 0.2420, -0.2462, 0.7400, 0.3019],
[ 0.2985, -0.3831, 0.5072, 0.3748]],
[[ 0.0000, 0.0000, 0.0000, 0.0000],
[-0.2693, 0.2609, -0.1224, 0.1326],
[ 0.0555, 0.2357, -0.1478, 0.2386],
[ 0.4334, 0.4978, -0.2275, 0.1967],
[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, 0.0000],
[ 0.0000, 0.0000, 0.0000, 0.0000]]], grad_fn=<ScatterBackward0>)
Thanks in advance!