Multiplying with identity matrix doesnt give same input matrix. Can anyone explain why?

I created a single linear layer model with identity matrix as the weights. But by passing the random input values to the model. the output and input values are not same after 3 to 4 decimal points of float type . But incase of integers the values are same.

Output:
image

The input and output have to be same. But you can see that its not the same when i print them.
Code used:

from multiprocessing import reduction

import torch

from torchvision import datasets, models, transforms

from torchsummary import summary

import os

import argparse

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

channel=1

length=1

bredth=2

torch.set_printoptions(threshold=10000,precision=16)

size_=channel*length*bredth

tinymodel = torch.nn.Sequential(torch.nn.Linear(size_, size_, bias=False))

print('The model:----------')

print(tinymodel)

tinymodel.to(device)

weights=torch.eye(size_)

print('Identity matrix:-----------')

print(weights)

weights=weights.int()

print(weights)

print(type(weights))

print(weights.shape)
print('\n\nModel params:-------')
for param in tinymodel.parameters():
    print(param.shape)

with torch.no_grad():
    tinymodel[0].weight.data.copy_(weights)

for param in tinymodel.parameters():
    print(param.shape)
    print(param)
tinymodel.to(device)
input_val=torch.rand([channel, length, bredth], dtype=torch.float, requires_grad=False).cuda()

input_val=torch.reshape(input_val,(-1,))
print("\nInput")
print(input_val.shape)
print(input_val)

# input_val=torch.Tensor([234,244]).to(device)                  // Testing with integer inputs while size_=2
result=tinymodel(input_val)
print("\nOutput")
print(result.shape)
print(result)

loss=torch.nn.MSELoss(reduction='sum')
loss_val=loss(result,input_val)
print("/n",loss_val)

Can anyone explain why is this happening?

Hi Junaid!

This is probably the “TF32 bug.” This post has some details:

Do you get the right answer when running on your cpu? What gpu are
you using? Does torch.backends.cuda.matmul.allow_tf32 = False
fix your issue?

As an aside, please don’t post screenshots of textual information. Doing
so breaks accessibility, searchability, and copy-paste.

Best.

K. Frank

1 Like