Tensorboard writer add_histogram bug

1

Since I have a problem with vanishing gradients while training my model I added a writer to monitor the models gradients

for name, param in self.model.named_parameters():
    self.writer.add_histogram(f"{name}.grad", param.grad, iteration)

this lead to ValueError("The histogram is empty, please file a bug report.")

even after changing that to

for name, param in self.model.named_parameters():
    if torch.isnan(param.grad).any():
        self.logger.warn(
            f"Gradient {name}.grad is NaN in iteration {iteration}"
        )
    else:
        self.writer.add_histogram(f"{name}.grad", param.grad, iteration)

I still get the same error.

This issue The histogram is empty · Issue #31855 · pytorch/pytorch · GitHub was closed as resolved but multiple user still seem to have the same issue.

Is there something I am missing?

Thanks in advance

2

The warning is raised from here. Could you check if the passed param.grad value might be empty?

3

I added

for name, param in self.model.named_parameters():
    try:
        self.writer.add_histogram(f"{name}.grad", param.grad, iteration)
    except:
        self.logger.warn(
            f"{name} with min:{torch.min(param.grad)} and max:{torch.max(param.grad)} is {param.grad}"
        )

For some batches the gradients are printed, e.g.:
D with min:4.3589534497664784e+26 and max:4.3589534497664784e+26 is tensor([4.3590e+26], device='cuda:0')

This does not look empty to me but the try block still fails.

4

I guess trying to create a histogram from a single value fails (which would be expected) so you might need to check if the gradient contains multiple values.

5 
if param.grad.size(dim=0) >= 2:
    self.writer.add_histogram(f"{name}.grad", param.grad, iteration)

solved it. Thank you!

1 Like