Network dimension float16, the output is nan

If I make the network dimension float16, the output is nan (if epoch > 0).

Forward pass is handled correctly (epoch = 0). After I do the backward, on the second epoch nan. Does anyone know why this is?

If I do float32, then everything works correctly. But since my card is RTX, it is better that the network runs on float16.

Adam(eps=1e-04) - I solved the problem