So, fairly typical sequence in many PyTorch training/optimization or evaluation loops:
- Create and initialize a Tensor
- Transfer (maybe) to GPU
- Initialize a Variable with this Tensor
- Variable used from here on, Tensor left dangling
What’s the best practice in this case for optimal memory usage? Especially when this may be done for multiple BxCxHxW tensors. To me it begs for a ‘del <tensor_name>’ after the variable is initialized, but I’m not sure if there is some tricks under the hood to make this unnecessary as I don’t see it done.
It’s often possible to wrap the tensor init within the variable creation but that’s not always conducive to clean code.
if args.init_rand: modifier = torch.normal(input_var.size(), std=0.001).float() else: modifier = torch.zeros(input_var.size()).float() if args.cuda: modifier = modifier.cuda() modifier_var = autograd.Variable(modifier, requires_grad=True) del modifier # best to do this by default if modifier never used again in train/eval loop???