in sequence model (such as recurrent neural network)
Sequence model should be a dynamic graph due to the dynamic input length and output length. Right?
In traditional implementation of static graph, we always fix the input length to make it a static graph.
In the example of dynamic layers
def forward(self, x):
h_relu = self.input_linear(x).clamp(min=0)
for _ in range(random.randint(0, 3)):
h_relu = self.middle_linear(h_relu).clamp(min=0)
y_pred = self.output_linear(h_relu)
return y_pred
I think a static graph can also do this. For example, first create a 4-layer h_relu
, then random.randint(0, 3)
as input instead of graph structure, which dynamically disable the h_relu
layer.