I am trying to use pytorch with tensorboard and I run the tensorboard server with the following command:
Now I am just simulating some fake data as follows:
import numpy as np
from torch.utils.tensorboard import SummaryWriter
train_writer = SummaryWriter(log_dir="./runs/train/")
for i in range(100):
v = np.random.randint(10, 100)
train_writer.add_scalar("loss", v, i)
Now if I do not put the sleep method in there, the script finishes and I can see the graph on the tensorboard front end.
I put the
sleep in there so that I could see the data arrive in a streaming fashion and be able to see the board updating. However, if I put the
sleep in there, no graph ever shows up and all I see is the message that there is no data or it could not find any.
I am not sure if this is a tensorboard issue or if I am doing something wrong in using it from within pytorch.
I am not sure how this works but maybe it waits for enough points to be added before sending it to the server? If you run only for 10 iterations and let it run until the end does it work?
So, if I add
sleep, it does not matter how many iterations I run but it never updates. In fact, even after the program terminates, the chart is not shown. I have to kill tensorboard and restart it for the graph to show.
So it would be tensorboard not updated even though the data has been written to disk?
@smth will know someone more knowledgeable about this that can find the reason for that ?
@lanpa @orionr are the right folks who might know more about this
Hi, can you try if
train_writer.flush() solve you problem?
@Luca_Pamparana are you running TensorBoard without TensorFlow being installed? If so, there was a bug where TB wouldn’t update even if there were updates to an event file. This has been fixed on nightly, so try
pip install tb-nightly and see if that solves the issue.