What is the proper ways to set parameters?

1

Hello all,

Happy new year,

I hope you all had a grand time. 2022 greeted me with a problem:
this is my neural network.

class NeuralNetwork(nn.Module):
def init(self):
super(NeuralNetwork, self).init()
self.flatten = nn.Flatten()
self.linear_relu_stack = nn.Sequential(
nn.Linear(5, 8),
nn.ReLU(),
nn.Linear(8, 12),
nn.ReLU(),
nn.Linear(12, 2),
nn.ReLU()
)

def forward(self, x):
    x = self.flatten(x)
    logits = self.linear_relu_stack(x)
    return logits

I wrote this function to change its weights.
In a nutshell: it adds up the different parameter tensors, flattens them, modify them a bit and put them back together in the model.

def jiggle(x, y, z):
#E_1, E_2, E_3 are orthogonal vectors in R^3 / 3D
x_coord = (torch.tensor(E_1) * torch.tensor(x))
y_coord = torch.tensor(E_2) * torch.tensor(y)
z_coord = torch.tensor(E_2) * torch.tensor(z)

x_coord = x_coord.to(device)
y_coord = y_coord.to(device
z_coord = z_coord.to(device)

flat_shifted_params = torch.cat(flat_params) * x_coord + torch.cat(flat_params) + torch.cat(flat_params) * z_coord
split_params = torch.split(flat_shifted_params, elem_count)
reshaped_params = [tensor.reshape(shapes[idx]) for idx, tensor in enumerate(split_params)]

def set_weights(i, j, model):
    if i < LEN_MODEL:
        if hasattr(model.linear_relu_stack[i], 'weight'):
            model.linear_relu_stack[i].weight = nn.Parameter(reshaped_params[j])
            j += 1
            i += 1
            set_weights(i, j, model))
        else:
            i += 1
            set_weights(i, j, model)

set_weights(0, 0, model)

When I try to test, I get this traceback:

Traceback (most recent call last):
File “C:/Users/…/Desktop/Raytracer Pytorch parameter space visualization/main.py”, line 314, in
score = test(test_dataloader, model, LOSS_FN)
File “C:/Users/…/Desktop/Raytracer Pytorch parameter space visualization/main.py”, line 101, in test
pred = model(X)
File “D:\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “C:/Users/…/Desktop/Raytracer Pytorch parameter space visualization/main.py”, line 86, in forward
logits = self.linear_relu_stack(x)
File “D:\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “D:\anaconda3\lib\site-packages\torch\nn\modules\container.py”, line 139, in forward
input = module(input)
File “D:\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “D:\anaconda3\lib\site-packages\torch\nn\modules\linear.py”, line 96, in forward
return F.linear(input, self.weight, self.bias)
File “D:\anaconda3\lib\site-packages\torch\nn\functional.py”, line 1847, in linear
return torch._C._nn.linear(input, weight, bias)
RuntimeError: mat2 must be a matrix, got 1-D tensor
when I feed it to the testing function.

Python doesn’t complain if I comment out the jiggle() function I wrote, so clearly I did something wrong here.

I assume mat2 refer to the parameter weights. So I tried debugging and set a breakpoint right after def set_weights(i, j, model):, the console command :

[x for x in model.parameters()] == [nn.Parameter(x) for x in reshaped_params]

returns True.

I’m not sure why is it complaining that it is different?

2

I think the parameter check is performed after you’ve flattened the parameters already, so while it would return True, I guess flattening the parameters in the first place creates the shape mismatch error. Check the shape of each parameter and make sure the linear layer weights are 2 dimensional.

1 Like
3

I inspected the layer one by one after .
It turns out the solution was even simpler than this: I forgot to set the bias, so I inadventantly chucked a 1-D bias tensor into the slot for weight (where it of course expected a matrix).

Rewriting the function set_weights as such did the trick:

def set_weights(i, j):
    if i < LEN_MODEL:
        if hasattr(model.linear_relu_stack[i], 'weight'):
            model.linear_relu_stack[i].weight = nn.Parameter(reshaped_params[j])
            j += 1
            model.linear_relu_stack[i].bias = nn.Parameter(reshaped_params[j])
            i += 1
            j += 1
            set_weights(i, j)
        else:
            i += 1
            set_weights(i, j)

I just lost track of things due to sleep deprivation.