I am trying CIFAR10 classification using an Alexnet type network and a multistepLR schedule, which reduces lr at epochs = 100, 150 and wd=1e-2. The training runs smoothly when the initial lr = 1e-3, but when I am using initial lr= 4e-3, I get random accuracy (~10%) with pytorch seed = 0, and around the expected accuracy (~85%) when I set the seed = 2. Is there any possible explanation for this behaviour, or is this solely due to a bug in my code? I am not posting any code for now as it is distributed among many files. Also, I am using kaiming initialization for the weights.
It’s most likely not a bug, but just a “lucky” and “bad” seed. Don’t try to optimize your code for a specific seed, as your training will be overall unstable and might break easily.
Based on your description it seems the learning rate is too high, but apparently some seeds force the initialization methods to generate working parameters for the setup.