# Unexpected predictions of linear regression with embedding layer

Hello everyone!

I have a following issue, which I apparently can’t solve myself.

I’m doing sentiment analysis for two classes with a simple linear regression model and additionally use pretrained Glove embeddings. Here is my model:

``````class LogisticRegressionModel(nn.Module):
def __init__(self, input_size: int, word_input_dim: int,
word_output_dim: int, word_embedding_matrix: np.ndarray,
output_classes: int):
super(LogisticRegressionModel, self).__init__()

self.word_embedding.weight = nn.Parameter(torch.tensor(word_embedding_matrix,
dtype=torch.float32))
self.linear = nn.Linear(input_size * word_output_dim, output_classes)

def forward(self, x):
word_embeddings = self.word_embedding(x)
word_embeddings = word_embeddings.view(x.shape, -1)
outputs = self.linear(word_embeddings)
return outputs
``````

I’m training batch-wise (batch_size=32) this model using SGD as optimiser and CrossEntropy with probabilities as criterion. As labels I use the probabilities of a sample being assigned to a class, e.g.:

``````tensor([[0.0000, 1.0000], [0.6000, 0.4000], [0.3333, 0.6667], ...])
``````

Here is my training snippet:

``````self.model.train()
for curr_epoch in range(num_epochs):
predictions = model(features)
loss = criterion(predictions, labels)
loss.backward()
optimizer.step()
``````

And I get the following predictions:

batch_1:

``````tensor([[-0.0055,  0.0678],
[-0.1317,  0.3271],
[ 0.2585,  0.0894],
...
[ 0.0702, -0.1932],
[ 0.0395,  0.2260],
``````

Such prediction looks ok for me, I would expect something like that.

batch_2:

``````tensor([[-32.1420,  32.2285],
[-28.5901,  28.9668],
[-15.8256,  15.9720],
...
[-31.3301,  31.8487],
[-30.2118,  30.2269],