Question about PyTorch and TensorFlow math

1

Hello,

If y’all don’t mind, I have a weird question.

Let’s say I create the same model in PyTorch and in TensorFlow and then train it on the same dataset, everything the same

Then if I use the API for both to get the intermediate result of lets say layer 5 for the same input data, will the intermediate result matrix be identical in both the cases?

Or, another way to phrase this question would be, the math that would happen in both the models, will it be exactly the same?

Thank you :slight_smile:

2

No, the math could slightly differ (I believe the handling of small eps values e.g. in normalization layers was different between frameworks). Even if the math is equal, the used algorithms might differ so you should at least expect small numerical errors caused by the limited floating point precision and a different order of operations.