Hi MinkyuChoi,
I just replaced the potential denormals for a certain threshold that worked in my experiments as follows:
def replace_denormals(x: torch.tensor, threshold=1e-10):
y = x.clone()
y[(x < threshold) & (x > -1.0 * threshold)] = threshold
return y