Learning - Why does .unsqueeze(-1) magically work?

Getting the hang of PyTorch by doing a simple linear regression.

When I use my own “custom” linear regression formula, I get fine results without changing my y_tensor (from a numpy array).

When I use the built in PyTorch linear regression class, I have to pass “y_tensor = y_tensor.unsqueeze(-1)” other wise I get the error:

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py:432: UserWarning: Using a target size (torch.Size([824])) that is different to the input size (torch.Size([824, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)

Code for not having to unsqueeze…

X_tensor = torch.from_numpy(X_train)
X_tensor = X_tensor.float()
y_tensor = torch.tensor(y_train.values,dtype=torch.float)

dep_vars = X_tensor.shape[1]

weights = torch.rand(dep_vars,requires_grad=True,dtype=torch.float)
bias = torch.rand(1,requires_grad=True,dtype=torch.float)

num_iterations = 1000
learning_rate = 0.1

def model(X): # Model function of the form y = w*X + b
    return torch.mv(X_tensor, weights) + bias 

loss_fn = torch.nn.MSELoss()  # Loss function MSE from PyTorch
# Optimizer using my weights and bias Tensor from PyTorch
optimizer = torch.optim.SGD([weights,bias],lr=learning_rate)  

for epoch in range(0,num_iterations): # Loop Through
    predicted = model(X_tensor)  # Compute predicted values
    loss = loss_fn(predicted, y_tensor) # Compute the loss
    if epoch % (num_iterations/10) == 0:
        print('loss at step ', ': ', loss)
    loss.backward() # Gradient of loss through backward propogation
    optimizer.step() # Optimizer update the weights
    optimizer.zero_grad() # Optimizer zero out the gradient for next step

print('weights: ', weights) # Print final weights and biases
print('bias: ', bias)

Code requiring unsqueeze…

X_tensor = torch.from_numpy(X_train)
X_tensor = X_tensor.float()
y_tensor = torch.tensor(y_train.values,dtype=torch.float)
y_tensor = y_tensor.unsqueeze(-1)

dep_vars = X_tensor.shape[1]

learning_rate = .1
num_iterations = 1000

model = torch.nn.Linear(dep_vars, 1, bias=True)
loss_fn = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)

for epoch in range(0,num_iterations):

    pred = model(X_tensor) # Make predictions of Linear Model
    loss = loss_fn(pred, y_tensor) # Calcualte Loss
    loss.backward() # Backward pass
    optimizer.step() # Update weights 
    optimizer.zero_grad() # Zero out gradient

    if epoch % (num_iterations / 10) == 0:
        print(f'{epoch} Loss: {loss}')

for param in model.parameters():

Any ideas?

There’s nothing magic about this. What you got is a warning and not an error. If you look at the documentation for MSELoss it says the target should have the same shape as the input which is what the unsqueeze operation ensures by adding an extra dimension. Hope this helps.

Ok - so thanks for writing that out, it helps.

Basically my model is specified output as [y,1] (pred here) so I have to specify the other input to the loss function also as a [y,1] instead of just [y].

Is that a fair explanation?

Yes. And vice versa too. If your targets are shaped [y], you shape the output from the model as [y].