Anyone has a code for FLOP calculation of an epoch?

vision
1

need to calculate flops during training (feedforward+backpropagation).
I would appreciate if anyone has the code to do so.

1 Like
2

there isn’t an official way to do this, but some folks have written ways of approximating it:



2 Likes
3

Thank you, but looks like they only consider the case when model is feed-forwarded or the inference time.
I need something to calculate FLOPs for both backpropagation and feed-forward.
Or am I wrong, these codes also work for backpropagation?

4

More likely the first one. Most papers and code are considered about complexity of the inference process

3 Likes
5

Open AI have a report on training FLOPs of various models:

and they used the following equation to calculate training FLOPs:

(add-multiplies per forward pass) * (2 FLOPs/add-multiply) * (3 for forward and backward pass) * (number of examples in dataset) * (number of epochs)
3 Likes