I’m attempting to generate an embedding visualization using PyTorch 1.1.0. I’m following the example in the documentation but running into an issue. When I try loaded the “Projector” component in Tensorboard, the backend returns a 500 and logs the following error:
I0605 22:51:01.434675 140375100487424 _internal.py:97] ::ffff:172.17.13.114 - - [05/Jun/2019 22:51:01] "GET /data/plugin/projector/runs HTTP/1.1" 500 -
E0605 22:51:01.438719 140375100487424 _internal.py:97] Error on request:
Traceback (most recent call last):
File "/home/tvanas/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 302, in run_wsgi
execute(self.server.app)
File "/home/tvanas/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 290, in execute
application_iter = app(environ, start_response)
File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/backend/application.py", line 309, in __call__
return self.data_applications[clean_path](environ, start_response)
File "/home/tvanas/.local/lib/python3.6/site-packages/werkzeug/wrappers/base_request.py", line 235, in application
resp = f(*args[:-2] + (request,))
File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/plugins/projector/projector_plugin.py", line 471, in _serve_runs
return Respond(request, list(self.configs.keys()), 'application/json')
File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/plugins/projector/projector_plugin.py", line 320, in configs
_latest_checkpoints_changed(self._configs, run_path_pairs)):
File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/plugins/projector/projector_plugin.py", line 181, in _latest_checkpoints_changed
ckpt_path = _find_latest_checkpoint(logdir)
File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/plugins/projector/projector_plugin.py", line 663, in _find_latest_checkpoint
ckpt_path = tf.train.latest_checkpoint(os.path.join(dir_path, os.pardir))
File "/home/tvanas/.local/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_management.py", line 336, in latest_checkpoint
ckpt = get_checkpoint_state(checkpoint_dir, latest_filename)
File "/home/tvanas/.local/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_management.py", line 278, in get_checkpoint_state
+ checkpoint_dir)
ValueError: Invalid checkpoint state loaded from /home/.../results/scannet_poster/logs/..
Anyone got any pointers on how to resolve this? I’m not sure why Tensorboard is looking for a checkpoint.
Output of pip list|grep tensor is:
tensorboard 1.13.1
tensorboardX 1.6
tensorflow 2.0.0a0
tensorflow-estimator-2.0-preview 1.14.0.dev2019060500