I think it could be expected that your model “forgets” the old classes as you are fine-tuning it on the new task only without giving it any information about the older class indices.
Here is a similar topic where the same issue is observed and I don’t know if there are any specific methods to avoid it.