Why can't I outperform tensorflow in pytorch?

1

Hi. I did a study on private data with pytorch. but I didn’t get a good result and my training period was too long. I got a very very short training time and a very high accuracy rate when I used tensorflow to work the same, what could be the reason. note: the same model and the same optimization algorithm in both

2

Typically, people with this problem don’t actually have as much “the same model and the same optimization algorithm” as they think. That is because if you had, you would get comparable results.

Best regards

Thomas

3

So how do we know if it’s exactly the same? I wrote the code for both of them myself. everything looks the same

4

What I do is that I check that same input produces same output, then same grads then same optimizer step (with “same” up to numerical accuracy). Many operations have subtle difference between different frameworks, e.g. whether a “mean” is taken or a “sum” or similar.

Best regards

Thomas

1 Like
5

Can you take a look if I send you the codes?

6

There are differences in how some modules are implementing in TensorFlow vs PyTorch.
i.e. batch norm momentum, .1 in PyTorch works same as .9 in TensorFlow.
Paddings are slightly different too.

1 Like
7

I’m fully booked this year, sorry.

8

thank you again for everything

9

I agree with Tom that you cannot just think your code look that same and the results have to be the same.
If you want to make sure everything is the same, layer by layer, you need to have testing for that, from layer to sequential of layers between TensorFlow and Pytorch.

Just for reference: GitHub - jalola/compare-tensorflow-pytorch: Compare outputs between layers written in Tensorflow and layers written in Pytorch

2 Likes