Hi @LeeDoYup,
Seems like I have also faced with similar problem. I was trying to use pretrained ResNet with changing few last layers, but it was not training (loss was increasing, while accuracy was near same).
Here you can see how you can specify which layers in the model should be optimized, by checking whether they are freezed or not.
Hope that will help you.
BR,
Petro