Can we replace Relu6 with hardtanh(0,6)

bigtree · November 18, 2021, 11:04pm

Can we replace Relu6 with Hardtah(0,6) since both clamp the value in the range (0~6)

Eta_C · November 19, 2021, 2:54am

YES, we can see that ReLU6 is a subclass of Hardtanh.

pytorch/pytorch/blob/11ca64149120b68cc612ebbdb8e0a24c908709f1/torch/nn/modules/activation.py#L242-L269

    
      
          class ReLU6(Hardtanh):
              r"""Applies the element-wise function:
          
          
    .. math::
                  \text{ReLU6}(x) = \min(\max(0,x), 6)
          
          
    Args:
                  inplace: can optionally do the operation in-place. Default: ``False``
          
          
    Shape:
                  - Input: :math:`(*)`, where :math:`*` means any number of dimensions.
                  - Output: :math:`(*)`, same shape as the input.
          
          
    .. image:: ../scripts/activation_images/ReLU6.png
          
          
    Examples::
          
          
        >>> m = nn.ReLU6()
                  >>> input = torch.randn(2)
                  >>> output = m(input)

This file has been truncated. show original