Removing the additional dimensions should work as shown in this code snippet:
conv = nn.Conv2d(3, 64, 3, 1, 1)
bias = conv.bias.clone()
conv.state_dict()['bias'].copy_(bias) # works
conv.state_dict()['bias'].copy_(bias[None, None, None, :])
> RuntimeError: output with shape [64] doesn't match the broadcast shape [1, 1, 1, 64]
The same error message is raised, if I add these additional dimensions to the bias, while it works for the matching shape.
Where did you try to remove the dimensions in your code?