Pytorch beginner model fit problems

I’m new to pytorch and run into first problem right away and hope to get some help here.
So this is my data generating function:

n_samples = 100
X = np.random.normal(size=(n_samples, 1))
y = np.random.normal(np.cos(5.*X) / (np.abs(X) + 1.), 0.1).ravel()
X_pred = np.atleast_2d(np.linspace(-3., 3., num=100)).T
X = np.hstack((X, X**2, X**3))
X_pred = np.hstack((X_pred, X_pred**2, X_pred**3))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, y_train, X_test, y_test, X_pred = map(torch.tensor, (X_train, y_train, X_test, y_test, X_pred))
train_ds = TensorDataset(X_train, y_train)
train_dl = DataLoader(train_ds, batch_size=6)

My first neural network looks like this:

input_dim = 3    
hidden_dim = 100  
output_dim = 1    

model = torch.nn.Sequential(
        torch.nn.Linear(input_dim, hidden_dim),
        torch.nn.Linear(hidden_dim, hidden_dim),
        torch.nn.Linear(hidden_dim, output_dim),

loss_fn = torch.nn.MSELoss()
learning_rate = 1e-3
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=0.004)

This is m training loop:

for epoch in range(40):  

    for i, data in enumerate(train_dl, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients

        # forward + backward + optimize
        outputs = model(inputs.float())
        loss = loss_fn(outputs, labels.float())

        # print statistics
    if epoch % 10 == 0:
        print(loss_fn(outputs, labels.float()))
print(loss_fn(outputs, labels.float()))

print('Finished Training')

My problem is that the network doesn’t fit at all. When i plot the results it is just a straight line. I tried for hours tune different parameters and look for errors in my code but unfortunatly I did not come to better result.
Hope someone has an idea or hint what do change in the model.

You have a shape mismatch for your output and labels.
Add dim1 to labels and it should work:

loss = loss_fn(outputs, labels.unsqueeze(1).float())

I’m creating a feature request to add a warning, since I’ve seen this particular error a few times in the past already.

It’s already being tracked here.

1 Like

Thank you very much. Should have came here much earlier! it works.I will start tuning the parameters again!

1 Like

Another small question. How can I get rid of the .float() in the traininig loop? Can I somehow cast the variables earlier?

Yes, you could create the numpy arrays in float32 or cast then using .astype(np.float32).
Alternatively, you might create them using PyTorch methods, which will create them in torch.float32 by default.

1 Like

Thanks again. Will do!