Looking for methods of fixing image artifacts

I’ve been working on a fully convolutional network but I’m getting some weird behaviors.
My issue is that the model seems to be mixing features, for example, features which should only appear in channel 2 and 3 are appearing in channel 1. I’ve looked through my test data and there doesn’t seem to be any clear cause of this behavior, it seems to be randomly occurring.

I’d love to hear some suggestions on how I could fix this issue.

Thanks in advance,