Pytorch beginner model fit problems

1

Hello,
I’m new to pytorch and run into first problem right away and hope to get some help here.
So this is my data generating function:

n_samples = 100
X = np.random.normal(size=(n_samples, 1))
y = np.random.normal(np.cos(5.*X) / (np.abs(X) + 1.), 0.1).ravel()
X_pred = np.atleast_2d(np.linspace(-3., 3., num=100)).T
X = np.hstack((X, X**2, X**3))
X_pred = np.hstack((X_pred, X_pred**2, X_pred**3))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, y_train, X_test, y_test, X_pred = map(torch.tensor, (X_train, y_train, X_test, y_test, X_pred))
train_ds = TensorDataset(X_train, y_train)
train_dl = DataLoader(train_ds, batch_size=6)

My first neural network looks like this:

input_dim = 3    
hidden_dim = 100  
output_dim = 1    

model = torch.nn.Sequential(
        torch.nn.Dropout(p=0.1), 
        torch.nn.Linear(input_dim, hidden_dim),
        torch.nn.ReLU(),
        torch.nn.Dropout(p=0.1),
        torch.nn.Linear(hidden_dim, hidden_dim),
        torch.nn.ReLU(),
        torch.nn.Dropout(p=0.1),
        torch.nn.Linear(hidden_dim, output_dim),
)  

loss_fn = torch.nn.MSELoss()
learning_rate = 1e-3
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=0.004)

This is m training loop:

for epoch in range(40):  

    
    for i, data in enumerate(train_dl, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs.float())
        loss = loss_fn(outputs, labels.float())
        loss.backward()
        optimizer.step()

        # print statistics
    if epoch % 10 == 0:
        print(loss_fn(outputs, labels.float()))
print(loss_fn(outputs, labels.float()))

print('Finished Training')

My problem is that the network doesn’t fit at all. When i plot the results it is just a straight line. I tried for hours tune different parameters and look for errors in my code but unfortunatly I did not come to better result.
Hope someone has an idea or hint what do change in the model.

2

You have a shape mismatch for your output and labels.
Add dim1 to labels and it should work:

loss = loss_fn(outputs, labels.unsqueeze(1).float())

I’m creating a feature request to add a warning, since I’ve seen this particular error a few times in the past already.

EDIT:
It’s already being tracked here.

1 Like
3

Thank you very much. Should have came here much earlier! it works.I will start tuning the parameters again!

1 Like
4

Another small question. How can I get rid of the .float() in the traininig loop? Can I somehow cast the variables earlier?

5

Yes, you could create the numpy arrays in float32 or cast then using .astype(np.float32).
Alternatively, you might create them using PyTorch methods, which will create them in torch.float32 by default.

1 Like
6

Thanks again. Will do!