I am working on a CNN project on an image dataset. I am applying Early Stopping technique in order to train the model. However, after training the model and obtaining the loss graph, it is heavily fluctuating, as shown in the image below. I have tried training the model more than once and I get a similar loss graph:
This is my model structure:
Therefore, is this a normal loss graph? If not, what may be the underlying issue and how can I solve it? I have read that regularization can solve this problem, but I am already applying Dropout. Is there anything else worth investigating?
Note that this is not really affecting my validation accuracy (accuracy on unseen data). It performs decently on it, with 93.3% accuracy.