I train a model and save the state_dict after training and evaluating it upto a particular accuracy measure. When I reload the state_dict onto a model and re-evaluate the evaluation changes drastically sometimes for better accuracy and sometimes to lower accuracy. I have even removed dropout and tried just to be thorough. It still happens.
The f1score and accuracies on the 5th epoch of First run and 0th epoch of second run should be same. What am I missing here?
First Run
==================================
INFO :trainer.py. do_every_checkpoint >> -- 0 -- loss: min/max/avg/latest: 0.72547/3.50690/1.42059/3.50690, accuracy: min/max/avg/latest: 0.77778/0.94792/0.88651/0.77778
INFO :trainer.py. do_every_checkpoint >> -- 0 -- precision: min/max/avg/latest: 0.10000/1.00000/0.65041/1.00000
INFO :trainer.py. do_every_checkpoint >> -- 0 -- recall: min/max/avg/latest: 0.25000/1.00000/0.71190/1.00000
INFO :trainer.py. do_every_checkpoint >> -- 0 -- f1score: min/max/avg/latest: 0.35184/0.72902/0.58465/0.57798
INFO :trainer.py. do_every_checkpoint >> accuracy of the best model so far 0 and current accuracy: 0.8865086810929435
INFO :trainer.py. do_every_checkpoint >> beat best model
INFO :trainer.py. save_best_model >> saving the last best model with accuracy 0.8865086810929435...
100% 310/310 [00:46<00:00, 6.66it/s]
INFO :trainer.py. train >> -- 0 -- loss: min/max/avg/latest: 0.02586/0.81412/0.13611/0.03917
CRITICAL:trainer.py. train >> memory consumed : 2414202880
100% 310/310 [00:48<00:00, 6.44it/s]
INFO :trainer.py. train >> -- 1 -- loss: min/max/avg/latest: 0.00663/0.81412/0.09141/0.00663
CRITICAL:trainer.py. train >> memory consumed : 2417090560
100% 310/310 [00:43<00:00, 7.10it/s]
INFO :trainer.py. train >> -- 2 -- loss: min/max/avg/latest: 0.00359/0.81412/0.07145/0.00359
CRITICAL:trainer.py. train >> memory consumed : 2420002816
100% 310/310 [00:43<00:00, 7.07it/s]
INFO :trainer.py. train >> -- 3 -- loss: min/max/avg/latest: 0.00237/0.81412/0.06022/0.00237
CRITICAL:trainer.py. train >> memory consumed : 2422865920
100% 310/310 [00:43<00:00, 7.19it/s]
INFO :trainer.py. train >> -- 4 -- loss: min/max/avg/latest: 0.00182/0.81412/0.05297/0.00182
CRITICAL:trainer.py. train >> memory consumed : 2425745408
100% 35/35 [00:06<00:00, 5.75it/s]
INFO :trainer.py. do_every_checkpoint >> -- 5 -- loss: min/max/avg/latest: 0.02749/0.67285/0.20378/0.52243, accuracy: min/max/avg/latest: 0.91647/0.97569/0.95618/0.94444
INFO :trainer.py. do_every_checkpoint >> -- 5 -- precision: min/max/avg/latest: 0.27273/1.00000/0.76438/1.00000
INFO :trainer.py. do_every_checkpoint >> -- 5 -- recall: min/max/avg/latest: 0.60000/1.00000/0.96857/1.00000
INFO :trainer.py. do_every_checkpoint >> -- 5 -- f1score: min/max/avg/latest: 0.64719/0.93600/0.85075/0.87992
INFO :trainer.py. do_every_checkpoint >> accuracy of the best model so far 0.8865086810929435 and current accuracy: 0.9213458350726536
INFO :trainer.py. do_every_checkpoint >> beat best model
INFO :trainer.py. save_best_model >> saving the last best model with accuracy 0.9213458350726536...
39% 120/310 [00:19<00:26, 7.30it/s] C-c C-cERROR :multi_token_entitynet.py. experiment >> ####################
Traceback (most recent call last):
File "multi_token_entitynet.py", line 315, in experiment
trainer.train()
KeyboardInterrupt
Second Run
=====================================
(torch) paarulakan@karunthuLai:~/projects/saama/ade_v3$ python multi_token_entitynet.py train
INFO :trainer.py. do_every_checkpoint >> -- 0 -- loss: min/max/avg/latest: 0.52617/2.88230/1.12987/2.88230, accuracy: min/max/avg/latest: 0.79167/0.95244/0.89640/0.79167
INFO :trainer.py. do_every_checkpoint >> -- 0 -- precision: min/max/avg/latest: 0.18182/1.00000/0.59224/1.00000
INFO :trainer.py. do_every_checkpoint >> -- 0 -- recall: min/max/avg/latest: 0.25000/1.00000/0.69190/0.33333
INFO :trainer.py. do_every_checkpoint >> -- 0 -- f1score: min/max/avg/latest: 0.41175/0.73633/0.61979/0.60060
INFO :trainer.py. do_every_checkpoint >> accuracy of the best model so far 0 and current accuracy: 0.8964035408837455
INFO :trainer.py. do_every_checkpoint >> beat best model
INFO :trainer.py. save_best_model >> saving the last best model with accuracy 0.8964035408837455...