Loss WGAN-GP majority is GP

I’m using the WGAN-GP architecture, but I’m getting a loss that is about 50000, of which 99% is from the gradient penalty and 1% from the normal WGAN loss. I’m not very experienced with this architecture, so I’m wondering whether the gradient penalty should be the major factor in the loss or not. My intuition says it shouldn’t, but I’m not sure what to change in that case.