It’s hard to tell what might be causing the failure in training you are seeing, but I would think it depends on the hyperparameters you are using as well as the overall training routine.
Can you elaborate on what might be the culprit in the training routine or other hyperparameters, then, if simply changing the optimizer from torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=0.0005) to torch.optim.Adam(params=model.parameters(), lr=0.0001) seems to alleviate my problems (where there was zero improvement in accuracy, there is now much improvement!) in the small, two-class sample run?
Additionally, I found the more or less “canon” implementation. I see they mention using SGD in the wiki
--mini 128 train using SGD with minibatch of 128 examples
but aside from that I cannot untangle where/how it’s being done. I see the learning rate stuff in the code, but nothing related to SGD with “momentum of 0.9” as specified in the paper and mentioned in passing in the wiki.
Here’s where I think they do all the training https://github.com/akrizhevsky/cuda-convnet2/blob/3238bf0367f63eb370e897b9e5714794cb67ddc2/python_util/gpumodel.py#L140 ; perhaps the SGD stuff is obfuscated behind the C++ model they loaded up?? It’s hard for me to decipher what this code is doing, tbh
def import_model(self):
lib_name = "cudaconvnet._ConvNet"
print "========================="
print "Importing %s C++ module" % lib_name
self.libmodel = __import__(lib_name,fromlist=['_ConvNet'])
Anyways, from what else I’ve seen in looking at how people have implemented Alexnet in PyTorch, they consistently seem to use Adam. For instance, this piece of code which I did not see/haven’t looked at until just now, after I ran into this problem in my own implementation https://github.com/dansuh17/alexnet-pytorch/blob/d0c1b1c52296ffcbecfbf5b17e1d1685b4ca6744/model.py#L142.
I think I/we are missing something nuanced about the optimizer, I’m just not sure what… any help would be appreciated (for now, though, I’ll use Adam 
)