I have implemented the evaluation of the test set as follows:
n_epochs = 1000
batch_size = 32
loss_train=[]
for epoch in range(n_epochs):
permutation1 = torch.randperm(trainX.size()[0])
for i in range(0,trainX.size()[0], batch_size):
optimizer.zero_grad()
indices1 = permutation1[i:i+batch_size]
batch_x_train, batch_y_train = trainX[indices1], trainY[indices1]
model.train()
outputs = model.forward(batch_x_train)
train_loss = criterion(outputs,batch_y_train)
train_loss.backward()
optimizer.step()
loss_train.append(train_loss.item())
model.eval()
y_pred = model(valX)
val_loss = criterion(y_pred, valY)
avg_train_loss = sum(loss_train) / len(loss_train)
print('epoch {}, train loss {}, val loss{}'.format(epoch, avg_train_loss, val_loss))
model.eval()
y_pred = model(testX)
test_loss = criterion(y_pred, testY)
print('test loss is {}'.format(test_loss))
Is this the correct way to evaluate the model on the test set? Also, where and how should I save the model in this case ( torch.save() or model.state_dict() ) if in the future all I would want to do is to load the model and just use it on the test set?
Assuming valX is a tensor with the complete validation data, then this approach would be generally right, but you might of course run out of memory, if this tensor is too large.
The usual approach would be to wrap it in a Dataset and DataLoader and get the predictions for each batch. The data loading tutorial gives you some information how to create a Dataset and DataLoader.
Also, to save memory during evaluation and test, you could wrap the validation and test code into a with torch.no_grad() block.
hi @ptrblck , I have a encoder decoder network where I am adding a softmax layer at the encoder. The training loss is converging nicely but when I run the evaluation on the same train data, it is giving very poor results. Can you check once plz if the test evaluation part is correct?
I’m not familiar with your use case and thus don’t know if the accuracy calculation makes sense or not. Check the raw outputs before rounding and compare some of these outputs manually to their targets to understand what your model is trying to predict.
hi @ptrblck , thanks for your reply. I found out that my issue is with the architecture itself and not inference. I have a simple encoder-decoder model and I am trying to add a softmax classifier layer from the encoder so that I can optimize the classification and reconstruction loss jointly.
class Autoencoder(nn.Module):
def init(self,D_in,H=50,H2=12,latent_dim=7):
# x_recon ist der im forward im Model erstellte recon_batch, x ist der originale x Batch, mu ist mu und logvar ist logvar
#def forward(self, x_recon, x, mu, logvar):
def forward(self, x_recon, x, logits, targets):
loss_MSE = self.mse_loss(x_recon, x)
#loss_KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
classification_loss = self.classification_criterion(logits, torch.max(targets, 1)[1])
return loss_MSE + classification_loss
def train(epoch):
model.train()
train_loss = 0
for batch_idx, data in enumerate(trainloader): data = data.to(device) #print (“data:”, data)
x_train = data[0]
y_train = data[1]
Can you please tell me what am I doing wrong here. Expected o/ps should be probabilities and also why it is predicting the same class always? Thank you so much!
That’s not the case since nn.CrossEntropyLoss expects raw logits (which are unbound) and will apply F.log_softmax internally.
Your model seems to be overfitting to class0 so check if this is the majority class and use a weighted sampling or weighted loss if that’s the case to counter this effect.