Hello, all.
I am a new-bee to PyTorch.
Here I have a simple idea but I do not know how to implement it with PyTorch.
The question is about how to mix model parameters on the fly, to do something like weighted network parameter sum.
Here I got multiply nn.Module
Instance A,B,C,D
, they share the same network structure but have different parameter values.
Then I got a simple Multiple Layer Perception which is also implemented as an nn.Module
to output a tensor W
of size(4,1).
The tensor W
works as a weight vector to be learned by the networks. And the instances A,B,C,D
are pre-trained, and parameters are frozen to be used as the components to be weighted.
Then I do not know to correctly implement the forward
function of the following class.
class Mixture(nn.Module):
def __init__(self, input_dim, hidden_dim, hidden_depth,base_modules =[]):
super().__init__()
self.weight_network= nn.Softmax(utils.mlp(input_dim, hidden_dim, len(base_modules ),hidden_depth))
self.base_modules = base_modules
for m in self.base_modules :
for c in m.children():
for param in child.parameters():
params.requires_grad = False
self.outputs = dict()
def forward(self, x):
weights= self.weight_network(x)
######################### Here need help ##################
# Here i want to get a network :
# f = w0*param_A+w1*param_B+w2*param_C+w3*param_D
#########################################################
f = ?
y = f(x)
self.outputs['y'] = y
return y
Thanks for any possible instructions and help.