I have been dabbling in PyTorch for a while and noticed a weird behaviour, likely because of seeding/instantiating issue.
# Model, cost function and optimizer instancing model = models.CustomNeuralNet().to(device) if is_forward_pass: # Forward Sanity Check _forward_X, _forward_y = models.forward_pass( loader=train_loader, model=model ) my_trainer: trainer.Trainer = trainer.Trainer( params=TRAIN_PARAMS, model=model, device=device, wandb_run=wandb_run, ) curr_fold_best_checkpoint = my_trainer.fit( train_loader, valid_loader, fold )
I won’t be able to paste the whole code pipeline as it is pretty long, but it would be good to know why this is the case.
For example, if I set
is_forward_pass to False, then my trainer will just start training, and say my loss is 0.1 for epoch 1. As a cautious move, I will always check if this particular loss changes when I modified the function (I value reproducibility a lot, though it is a black box to me). So when I switched on
is_forward_pass to True, then my loss changed. Once I turned it off, the loss is the same value so I am sure it is this flag. My hunch is somehow the
DataLoader or the
Model is being called again? Has anyone faced this issue?