My numpy and pytorch codes have totally different results

#1

I wanted to calculate the sum of 1st to K-th power of an array and equally calculate the sum of 1st to k-th power of a tensor. I found out that the following codes and their results are totally different and I don’t know why.

I debugged the code and I know that the results are equal in the first round.

Numpy code:

adj_k_prob = adj_prob
adj_k_pow =  adj_prob
for i in range(K):
    adj_k_pow = np.matmul(adj_prob, adj_k_pow)
    adj_k_prob += adj_k_pow

Pytorch code:

adj_k_prob = adj_prob_tensor
adj_k_pow = adj_prob_tensor
for i in range(K):
    adj_k_pow = torch.matmul(adj_prob_tensor, adj_k_pow)
    adj_k_prob += adj_k_pow
#2

Could you post the values for all inputs.
I just used some random numbers and got the same results.

1 Like
#3

The value of adj_prob_tensor and adj_prob at the start of loop are as follow (when I debugged it):

    tensor([[0.0000, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429],
    [0.2500, 0.0000, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.2500, 0.2500, 0.0000, 0.2500, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5000, 0.0000],
    [0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5000, 0.0000],
    [0.3333, 0.0000, 0.0000, 0.0000, 0.3333, 0.3333, 0.0000, 0.0000],
    [0.2500, 0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000]])

I don’t know what is the problem at all, is there anything that I can check or debug?

#4

Thanks for posting the input values.
Did you make sure to properly copy the values, as you are changing the original array?
This script gives the same results for different values of K (of course you’ll get inf for too large values):

adj_prob = np.array([[0.0000, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429],
    [0.2500, 0.0000, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.2500, 0.2500, 0.0000, 0.2500, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5000, 0.0000],
    [0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5000, 0.0000],
    [0.3333, 0.0000, 0.0000, 0.0000, 0.3333, 0.3333, 0.0000, 0.0000],
    [0.2500, 0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000]])

adj_prob_tensor = torch.from_numpy(adj_prob).clone()

K = 5

adj_k_prob = adj_prob
adj_k_pow =  adj_prob
for i in range(K):
    adj_k_pow = np.matmul(adj_prob, adj_k_pow)
    adj_k_prob += adj_k_pow


adj_k_prob_pt = adj_prob_tensor
adj_k_pow_pt = adj_prob_tensor
for i in range(K):
    adj_k_pow_pt = torch.matmul(adj_prob_tensor, adj_k_pow_pt)
    adj_k_prob_pt += adj_k_pow_pt

print(torch.abs(torch.from_numpy(adj_k_prob) - adj_k_prob_pt).max())
> tensor(0., dtype=torch.float64)
1 Like
#5

Actually I read a file which contains graph and divide every row of tensor/array by its sum. I have absolutely no idea what is going on. When I debug the code, the value of tensor and array are the same (one example is what I’ve written). I just run code for K=5.

#6

Could you post a code snippet to reproduce this issue?
I could create a local file with these values and check, if something goes wrong loading the data etc.

1 Like
#7

Sorry, can you give some hint about what do you mean by properly copying the values? I don’t know what you mean in this code.

#8

As you can see I’ve cloned the values after transforming them to PyTorch.
Otherwise adj_prob_tensor will hold a pointer to adj_prob and will be changed in the first for loop.

1 Like
#9

I’m so sorry for bothering you. The following code has problem. I would be grateful if you help me.

import networkx as nx
import numpy as np
import torch
from scipy import sparse


def sparse_mx_to_torch_sparse_tensor(sparse_mx):
    """Convert a scipy sparse matrix to a torch sparse tensor."""
    sparse_mx = sparse_mx.tocoo().astype(np.float32)
    indices = torch.from_numpy(
        np.vstack((sparse_mx.row, sparse_mx.col)).astype(np.int64))
    values = torch.from_numpy(sparse_mx.data)
    shape = torch.Size(sparse_mx.shape)
    return torch.sparse.FloatTensor(indices, values, shape)


if __name__ == '__main__':
    K = 5
    ### Load Data
    G = nx.karate_club_graph()
    G = nx.adj_matrix(G)
    adj_prob = G.toarray()
    adj_prob = adj_prob / adj_prob.sum(axis=1)[:, None]

    ### Numpy
    adj_k_prob = adj_prob
    adj_k_pow =  adj_prob
    for i in range(K):
        adj_k_pow = np.matmul(adj_prob, adj_k_pow)
        adj_k_prob += adj_k_pow


    ### Pytorch
    adj_prob_sparse = sparse.csr_matrix(adj_prob)
    adj_prob_tensor = sparse_mx_to_torch_sparse_tensor(adj_prob_sparse)

    adj_k_prob_tensor = adj_prob_tensor.to_dense().clone()
    adj_k_pow_tensor = adj_prob_tensor.to_dense().clone()

    for i in range(K):
        adj_k_pow_tensor = torch.matmul(adj_prob_tensor.to_dense(), adj_k_pow_tensor)
        adj_k_prob_tensor += adj_k_pow_tensor



    print(torch.abs(torch.from_numpy(adj_k_prob).double() - adj_k_prob_tensor.double()).max())
#10

The error is regarding the shallow/deep copy.
If you add a copy() after assigning the numpy arrays, your code runs fine with an error of ~2e-7:

    adj_k_prob = adj_prob.copy()
    adj_k_pow =  adj_prob.copy()
1 Like
#11

Many thanks to your. Sorry can you tell me when and why this happens?

#12

A lot of assignments will be just a shallow copy of the data just for performance reasons.
Instead of copying a huge matrix the data pointer will just be passed so that the assignment is basically free.
You can find more information on numpy’s copying here.
Since PyTorch also deals with large data tensors, usually shallow copies will also be used if possible.

a = torch.zeros(2, 2)
b = a
b[0, 0] = 1
print(a)
> tensor([[1., 0.],
          [0., 0.]])
print(b)
> tensor([[1., 0.],
          [0., 0.]])
1 Like