A single conv and same input leads to results different from tensorflow

1

Hi,

I am re-implementing my Tensorflow models with pytorch. The problem comes when I want to test it by loading weights of previously trained Tensorflow models, since I got very different performance. Obviously something goes wrong.

For debugging I start with a single conv layer where I initialise the conv kernel with the same weights and apply to a same input. Surprisingly, the Pytorch implementation and Tensorflow implementation gives different results.
Here’s the code:

# Tensorflow  Python2.7
import numpy as np
import tensorflow as tf

# Different weights for testing
np.save('weg.npy', 0.008*np.random.randint(1,1000,(3,3,3,6)))  # Option1
#np.save('weg.npy', np.random.randint(1,1000,(3,3,3,6)))  # Option2
#np.save('weg.npy', 10.12*np.random.randint(1,1000,(3,3,3,6)))  # Option3

inputs = tf.Variable(1.5*np.ones((1, 10, 10,3), dtype=np.float32))
net = tf.contrib.slim.conv2d(inputs, 6, [3, 3], stride=1, 
                  weights_initializer=tf.constant_initializer(np.load('weg.npy')),
                  biases_initializer=tf.constant_initializer(0),
                  activation_fn=None)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    x = sess.run(net)
    np.save('tf.npy', x)
    print x.shape

Since I am using python3.6 environment for pytorch, so I didn’t put 2 codes together.

# Pytorch Python3.6
import torch.nn as nn
import torch
import numpy as np

# Prepare weights
weights =  torch.from_numpy(np.load('weg.npy')).permute((3, 2, 0, 1)) # swap to NCHW
biases = torch.from_numpy(np.zeros(6))
weight_dict = OrderedDict()
weight_dict.update({'0.weight':weights})
weight_dict.update({'0.bias':biases})

inputs = torch.from_numpy(1.5*np.ones((1, 3, 10, 10), dtype=np.float32))
net = nn.Sequential(nn.Conv2d(3, 6, kernel_size=3, stride=1, padding=1))
net.load_state_dict(weight_dict)

# Compare results
m_ = net(inputs)
m=np.load('tf.npy')
print(np.linalg.norm(m - m_.permute((0, 2, 3,1)).data.numpy()))  # Swap back to NHWC

So I found that the different options to initialise the weights i.e. 'weg.npy’ led to various the errors results(notice the difference between weights options is just different coefficients)

Results:
np.random.randint(1,1000,(3,3,3,6)) => 0.0
0.008*np.random.randint(1,1000,(3,3,3,6)) => 0.00031963884
10.12*np.random.randint(1,1000,(3,3,3,6)) => 0.37016743

Does anyone have any idea why this is happening? I am really confused, hoping someone can help,thanks!

1 Like
2

Hello Zhou,

I have same problems in my work and tried your approach.
In case of yours, changing data type to double might help. There is similar of yours.

I changed your code a bit like below.

# data type
# data_type = np.float32
data_type = np.float64

# Different weights for testing
# weight = np.random.randint(1, 1000, (3, 3, 3, 6)).astype(data_type)
# weight = 0.008 * np.random.randint(1, 1000, (3, 3, 3, 6)).astype(data_type)
weight = 10.12 * np.random.randint(1, 1000, (3, 3, 3, 6)).astype(data_type)

inputs = tf.Variable(1.5 * np.ones((1, 10, 10, 3), dtype=data_type))

# Tensorflow
tf_model = tf.contrib.slim.conv2d(inputs, 6, [3, 3], stride=1,
                                  weights_initializer=tf.constant_initializer(weight),
                                  biases_initializer=tf.constant_initializer(0),
                                  activation_fn=None)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    tf_output = sess.run(tf_model)

# Pytorch
weights = torch.from_numpy(weight).permute((3, 2, 0, 1))
biases = torch.from_numpy(np.zeros(6).astype(data_type))

inputs = torch.from_numpy(1.5 * np.ones((1, 3, 10, 10), dtype=data_type))
torch_model = nn.Conv2d(3, 6, kernel_size=3, stride=1, padding=1)
torch_model.weight = nn.Parameter(weights)
torch_model.bias = nn.Parameter(biases)
torch_output = torch_model(inputs)

# Compare results

print("diff max: ", (tf_output - torch_output.permute((0, 2, 3, 1)).data.numpy()).max())
print("diff norm: ", np.linalg.norm(tf_output - torch_output.permute((0, 2, 3, 1)).data.numpy()))

Here is the result.

Result in case np.float32
diff max:  0.015625
diff norm:  0.286411

Result in case np.float64
diff max:  2.9103830456733704e-11
diff norm:  7.428606633388099e-10
1 Like