Intermediate results in Checkpoint

1

Can we save intermediate results in checkpoint?
Intermediate results are important to save in checkpoint?

2

Which intermediate results do you mean?
Also, do you mean torch.utils.checkpoint or creating a checkpoint by serializing the model?

3

torch.save()

Does it save only learnable parameters or intermediate results also?

4

Could you explain, what you mean by “intermediate results”?

5

For example, in the transformer model https://pytorch.org/tutorials/beginner/transformer_tutorial.html

outputs are calculated in forward pass, and layer weights and bias are intermediate results.

correct me if I am wrong

6

The weight and bias would be parameters and I wouldn’t call them intermediate results.
I think the intermediate activations could be called intermediate results, but I’m still unsure if that’s really what you are looking for as I haven’t heard this naming so far.
By intermediate activations I mean e.g.:

def forward(self, x):
    intermediate_act1 = self.layer1(x)
    intermediate_act2 = self.layer2(intermediate_act1)
    ...
7

Thanks @ptrblck, now I got clarification