Inference with DCG

chenjus · August 16, 2017, 4:32pm

After running this example dcg code

jcjohnson/pytorch-examples/blob/master/nn/dynamic_net.py

import random
import torch
from torch.autograd import Variable

"""
To showcase the power of PyTorch dynamic graphs, we will implement a very strange
model: a fully-connected ReLU network that on each forward pass randomly chooses
a number between 1 and 4 and has that many hidden layers, reusing the same
weights multiple times to compute the innermost hidden layers.
"""

class DynamicNet(torch.nn.Module):
  def __init__(self, D_in, H, D_out):
    """
    In the constructor we construct three nn.Linear instances that we will use
    in the forward pass.
    """
    super(DynamicNet, self).__init__()
    self.input_linear = torch.nn.Linear(D_in, H)
    self.middle_linear = torch.nn.Linear(H, H)

This file has been truncated. show original

which randomly chooses the number of hidden layers [0, 3] each iteration, I printed the parameters in the network with print [x for x in model.parameters()] and it only shows the original nn.modules.

[Parameter containing:
 0.0682 -0.0155 -0.3447
 0.1916 -0.1639  0.1732
[torch.FloatTensor of size 2x3]
, Parameter containing:
 0.0412
-0.3786
[torch.FloatTensor of size 2]
, Parameter containing:
 0.3711  0.2467
-0.4274  0.6104
[torch.FloatTensor of size 2x2]
, Parameter containing:
 0.5531
 0.0944
[torch.FloatTensor of size 2]
, Parameter containing:
 0.0678 -0.4616
[torch.FloatTensor of size 1x2]
, Parameter containing:
 0.2338
[torch.FloatTensor of size 1]
]

It doesn’t show the dynamic layers used during training. Question: how does pytorch store the dynamic parameters? Are the dynamic parameters used during inference?