Frequency Domain Image processing in CNN

Now, I am working on the frequency domain image into the CNN. However, the imaginary part(phase) is discarded in CNN. Can it decrease the accuracy of the neural network? Most of the research papers suggested using the frequency domain loss. They create a lot of frequency domain losses. It doesn’t work for me. Please guide me in this corresponding research field.