CNN training for clustering

I am trying to compare different types of norms in terms of accuracy and using stopping criteria in terms of accuracy as well. For the same hyper-parameters setup one specific norm is working okay but for alternative norms my accuracy fluctuating as in the picture. Anyone can give me some idea why this types of behavior is seen for my model?