Orthogonal Initialization Error

1

Hi,

I am trying to use orthogonal initialization, I have tried many ways but I keep getting an error, the code and stack trace is below.

Code:
torch.nn.init.orthogonal(m.weight) or torch.nn.init.orthogonal(m.weight.data)

Traceback (most recent call last):
File “/home/saby2k13/projects/ctb-ilie/saby2k13/pyDPPI/python/optimise.py”, line 620, in
main()
File “/home/saby2k13/projects/ctb-ilie/saby2k13/pyDPPI/python/optimise.py”, line 465, in main
torch.nn.init.orthogonal(m.weight)
File “/project/6000341/saby2k13/DPPI2/venv/tensorflow/lib/python3.5/site-packages/torch/nn/init.py”, line 445, in deprecated_init
return meth(*args, **kwargs)
File “/project/6000341/saby2k13/DPPI2/venv/tensorflow/lib/python3.5/site-packages/torch/nn/init.py”, line 381, in orthogonal_
raise ValueError(“Only tensors with 2 or more dimensions are supported”)
ValueError: Only tensors with 2 or more dimensions are supported

2

From the docs:

The input tensor must have
at least 2 dimensions, and for tensors with more than 2 dimensions the
trailing dimensions are flattened.

While it seems your tensor only has a single dimension?
If so, could you explain your use case a bit, and why m.weight is used (or which module it comes from)?

3

Thanks for the response!
This is how the code looks like

for m in model.modules():
if isinstance(m, nn.Conv2d):
print(‘m.weight.shape’,m.weight.shape,',dimension = ',m.weight.ndimension())
torch.nn.init.orthogonal(m.weight)

The output of the print statement :

m.weight.shape torch.Size([64, 20, 1, 5]) ,dimension = 4

So definitely the tensor dimension > 2.

4

Thanks for the code!
It should work, as shown with this dummy code snippet:

model = models.resnet50()

for m in model.modules():
    if isinstance(m, nn.Conv2d):
        print('m.weight.shape',m.weight.shape,',dimension = ',m.weight.ndimension())
        torch.nn.init.orthogonal_(m.weight)

Minor issue: you should get a UserWarning:

UserWarning: nn.init.orthogonal is now deprecated in favor of nn.init.orthogonal_.

which shouldn’t be related to the error you are seeing.

5

Thanks !
Yes I had tried that with the underscore already once and then tried again after your suggestion but unfortunately the error is still there.

m.weight.shape torch.Size([64, 20, 1, 5]) ,dimension = 4
m.bias.shape torch.Size([64])
Traceback (most recent call last):
File “/home/saby2k13/projects/ctb-ilie/saby2k13/pyDPPI/python/optimise.py”, line 620, in
main()
File “/home/saby2k13/projects/ctb-ilie/saby2k13/pyDPPI/python/optimise.py”, line 465, in main
torch.nn.init.orthogonal_(m.weight)
File “/project/6000341/saby2k13/DPPI2/venv/tensorflow/lib/python3.5/site-packages/torch/nn/init.py”, line 381, in orthogonal_
raise ValueError(“Only tensors with 2 or more dimensions are supported”)
ValueError: Only tensors with 2 or more dimensions are supported

6

Could you post the model definition, so that we can have a look at this issue?

7

This was a code problem, Sorry about that. Thanks!