Inconsistency between PyTorch and TensorFlow's variance function's results and how PyTorch implements it using the summation function?

vision
1

I’m working on a vision model and it requires using variance function.

Inconsistency between PyTorch and TensorFlow variance function outputs

For some reasons I am testing the TensorFlow and PyTorch’s variance functions and I find that the results for same input values are different. And it sounds weird because variance is a parameterless function, hence, there are no values to set that might affect the output value.

# TensorFlow Example
import tensorflow as tf
x = tf.constant([[1., 2.], [3., 4.]])

print(tf.math.reduce_variance(x))
print(tf.math.reduce_variance(x, 0))
print(tf.math.reduce_variance(x, 1))

# Outputs
# tf.Tensor(1.25, shape=(), dtype=float32)
# tf.Tensor([1. 1.], shape=(2,), dtype=float32)
# tf.Tensor([0.25 0.25], shape=(2,), dtype=float32)

# PyTorch Example
import torch as t
x = t.tensor([[1., 2.], [3., 4.]])

print(t.var(x))
print(t.var(x, 0))
print(t.var(x, 1))

# Outputs
# tensor(1.6667)
# tensor([2., 2.])
# tensor([0.5000, 0.5000])

Implementation help with PyTorch’s var() using sum() function

Since my model is trained in PyTorch I want to know how to write the var() function using sum() function. The implementation I tried is as follows but the result doesn’t match with that of PyTorch’s official implementation:

import torch as t
_ = t.manual_seed(0)

def var(x, dim):
  x_mean = t.mean(x)
  sq_devs = t.square(x - x_mean)
  return t.mean(sq_devs, dim=dim)

x = t.randn(5, 4, 3)
x_var1 = t.var(x, dim=0)
x_var2 = var(x, dim=0)

print("x:", x)
print("x_var1:", x_var1)
print("x_var2:", x_var2)

Output

x: tensor([[[-1.1258, -1.1524, -0.2506],
         [-0.4339,  0.8487,  0.6920],
         [-0.3160, -2.1152,  0.3223],
         [-1.2633,  0.3500,  0.3081]],

        [[ 0.1198,  1.2377,  1.1168],
         [-0.2473, -1.3527, -1.6959],
         [ 0.5667,  0.7935,  0.5988],
         [-1.5551, -0.3414,  1.8530]],

        [[ 0.7502, -0.5855, -0.1734],
         [ 0.1835,  1.3894,  1.5863],
         [ 0.9463, -0.8437, -0.6136],
         [ 0.0316, -0.4927,  0.2484]],

        [[ 0.4397,  0.1124,  0.6408],
         [ 0.4412, -0.1023,  0.7924],
         [-0.2897,  0.0525,  0.5943],
         [ 1.5419,  0.5073, -0.5910]],

        [[-0.5692,  0.9200,  1.1108],
         [ 1.2899, -1.4959, -0.1938],
         [ 0.4455,  1.3253, -1.6293],
         [-0.5497, -0.4798, -0.4997]]])
x_var1: tensor([[0.5831, 1.0012, 0.4474],
        [0.4593, 1.6567, 1.5645],
        [0.3082, 1.8627, 0.9353],
        [1.5127, 0.2319, 0.9604]])
x_var2: tensor([[0.4835, 0.8038, 0.5476],
        [0.4048, 1.3637, 1.2851],
        [0.2938, 1.5346, 0.7877],
        [1.3801, 0.2065, 0.8126]])

I’d really appreciate anyone willing to help me here. :smiley:

Thanks in advance,
Rahul Bhalley

2

Hi,

I think what you want is to pass unbiased=False to the var function in pytorch: torch.var — PyTorch 1.8.1 documentation
By default, pytorch compute the unbiased variance but I think Tensorflow computes the biased version.

1 Like
4

Thank you! :blush: It helps.

5

Hey @albanD! How can I implement the var() function using PyTorch’s sum() function?

6

I found the solution myself.

Following is an unbiased estimator implementation of variance using summation function, if anyone needs that:

def var(input, dim, keepdim=False):
  input_means = t.mean(input, dim=dim, keepdim=True)
  difference = input - input_means
  squared_deviations = t.square(difference)
  return t.mean(squared_deviations, dim=dim, keepdim=keepdim)