In one of my training loops I’m logging quite a bit of debug information after every parameter update. I’ve noticed that when profiling this loop with cPofile, string formatting takes up a significant chunk of the total training runtime (something like 5%).
That doesn’t seem right to me, since the forward and backward pass take around 2-3 seconds on every iteration. Is there some way I can run the logging in parallel with the backward pass? (and is that likely to solve the problem).
My training loop looks something like this:
for img in dataloader:
img = img.to('cuda')
optimizer.zero_grad()
pred, label = network(img)
loss = self.loss(pred, label)
loss.backward()
optimizer.step()
fmt = "loss was {:1.3e}"
msg = fmt.format(loss)
logger.info(msg)
# some more complicated logging stuff