I have built a model and overfit the model for a small batch of 16 samples from the training set. When I calculate the accuracy for this same batch, I expect 100% accuracy. I have used two different methods to calculate accuracy and both of them give such different results.
My forward pass function:
def fwd_pass(X, y, train=False):
if not train:
with torch.no_grad():
outputs = model(X)
matches = (torch.argmax (outputs, dim = 1) == y).sum()
acc = matches/len(y)
loss = loss_fn(outputs, y)
return acc, loss
outputs = model(X)
matches = (torch.argmax (outputs, dim = 1) == y).sum()
acc = matches/len(y)
loss = loss_fn(outputs, y)
loss.backward()
opt.step()
model.zero_grad()
return acc, loss
My train function:
def train(net, epochs, batch_size, X, y, val_X=None, val_y=None):
accuracies = []
losses = []
val_accuracies = []
val_losses = []
for ep in tqdm(range(epochs)):
for i in tqdm(range(0, len(X), batch_size)):
batch_X = X[i:i+batch_size].to(device)
batch_y = y[i:i+batch_size].to(device)
acc, loss = fwd_pass(batch_X.float(), batch_y, train=True)
torch.cuda.empty_cache()
if val_X != None and val_y != None:
val_acc, val_loss = fwd_pass(val_X.to(device), val_y.to(device))
print(f'Ep: {ep+1} Acc: {round(float(acc), 5)} Loss: {round(float(loss), 5)}')
if val_X != None and val_y != None:
print(f'Val Acc: {round(float(val_acc), 5)} Val Loss: {round(float(val_loss), 5)}')
accuracies.append(acc)
losses.append(loss)
if val_X != None and val_y != None:
val_accuracies.append(val_acc)
val_losses.append(val_loss)
return accuracies, losses, val_accuracies, val_losses
Now here, I calculate the accuracies:
corr = 0
tot = 0
with torch.no_grad():
for i in range(len(batch_1_X)):
op = model(batch_1_X[i].view(1, 100, 8).transpose(1, 2).to(device).float())
pred = torch.argmax(op)
real = batch_1_y[i]
if pred == real:
corr += 1
tot += 1
print(corr)
print(tot)
The above gives me the following output:
3
16
suggesting that only 3 out of 16 have been predicted correctly, even though the model has been overfit.
Now, this method:
corr = 0
tot = 0
with torch.no_grad():
op = model(batch_1_X.transpose(1, 2).to(device).float())
preds = torch.argmax(op, dim=1)
for p, r in zip(preds, batch_1_y):
if p == r:
corr += 1
tot += 1
print(corr)
print(tot)
gives me the output
16
16
suggesting that all predictions are correct
The most confusing part of all was this:
corr = 0
tot = 0
num = 16
with torch.no_grad():
op = model(batch_1_X[0:num].transpose(1, 2).to(device).float())
preds = torch.argmax(op, dim=1)
for p, r in zip(preds, batch_1_y[0:num]):
if p == r:
corr += 1
tot += 1
print(corr)
print(tot)
When num is equal to 16, the corr and tot values are 16 and 16. When num is equal to something like 1, 3, 4, 5, 6, 7, 8, 9,…, the corr and tot values are equal. But when num is 2, corr is 1 and tot is 2 suggesting that the model got only 1 of the 2 predictions right.
What is the mistake I am making that is giving me this unpredictable behaivior?