During training, my model output occasionally flips and loss skyrockets.
From the fact that r^2 is maintained and AUC becomes 1-AUC, I can the output have flipped.
Can you guess what is the problem?
Here is my model
Custom_Model(
(submodule_0): Sequential(
(0): Masked_Linear(input_features=17741, output_features=17741, bias=True, avg_mask_on=34.066230764894875)
(1): BatchNorm1d(17741, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ELU(alpha=1.0)
(3): Dropout(p=0.4, inplace=False)
)
(submodule_1): Sequential(
(0): Masked_Linear(input_features=17741, output_features=17741, bias=True, avg_mask_on=21.358266163124966)
(1): BatchNorm1d(17741, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ELU(alpha=1.0)
(3): Dropout(p=0.6, inplace=False)
)
(submodule_2): Sequential(
(0): Masked_Linear(input_features=17741, output_features=17741, bias=True, avg_mask_on=7.067527196888563)
(1): BatchNorm1d(17741, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ELU(alpha=1.0)
(3): Dropout(p=0.6, inplace=False)
)
(submodule_3): Sequential(
(0): Linear(in_features=17741, out_features=1024, bias=True)
(1): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ELU(alpha=1.0)
(3): Dropout(p=0.5, inplace=False)
(4): Linear(in_features=1024, out_features=256, bias=True)
(5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(6): ELU(alpha=1.0)
(7): Dropout(p=0.2, inplace=False)
(8): Linear(in_features=256, out_features=1, bias=True)
)
)
Optimizer
Adagrad (
Parameter Group 0
eps: 1e-10
initial_accumulator_value: 0
lr: 0.01
lr_decay: 0
weight_decay: 3e-07
)
Log
VALID: epoch: 14-852(99.88%), loss: 0.16498, cached: 63.937
VALID: output r2: 0.000241, auc: 0.56510
TRAIN: epoch: 15-2556(99.96%), loss: 0.00577, cached: 63.986
TRAIN: output r2: 0.097297, auc: 0.99790
VALID: epoch: 15-852(99.88%), loss: 0.16588, cached: 63.937
VALID: output r2: 0.000255, auc: 0.56665
TRAIN: epoch: 16-2556(99.96%), loss: 0.00535, cached: 63.986
TRAIN: output r2: 0.092888, auc: 0.99835
VALID: epoch: 16-852(99.88%), loss: 5.11513, cached: 63.937
VALID: output r2: 0.000259, auc: 0.56450
TRAIN: epoch: 17-2556(99.96%), loss: 0.00452, cached: 63.986
TRAIN: output r2: 0.100936, auc: 0.99893
VALID: epoch: 17-852(99.88%), loss: 14.84378, cached: 63.937
VALID: output r2: 0.000285, auc: 0.43133
TRAIN: epoch: 18-2556(99.96%), loss: 0.00371, cached: 63.986
TRAIN: output r2: 0.115433, auc: 0.99923
VALID: epoch: 18-852(99.88%), loss: 0.07149, cached: 63.937
VALID: output r2: 0.000277, auc: 0.56678