Does anyone have experience or theory to suggest whether second order preconditioning optimizers like shampoo should play well with cyclic learning rate schedulers?
Does anyone have experience or theory to suggest whether second order preconditioning optimizers like shampoo should play well with cyclic learning rate schedulers?